feat(verifiers): read coding-agent CLI trajectories in the judge action log

[kaihackney-scale]

Why

The direct (non-agent) judge's action log (compaction_type: screenshot, trajectory_text: action_log) only parsed OTel spans. Trajectories from coding-agent CLIs (stream-json rows: system / assistant / user / result) and from OpenCode (part events) produced an empty transcript. A direct judge on those runs therefore graded without seeing a single tool call. The fixed per-call cuts (110 chars of arguments, 160 of result) are also too small for MCP tool traffic.

What

• transcript_from_raw detects the format:
  • OTel spans: existing behaviour, moved unchanged into _transcript_from_otel.
  • stream-json rows: tool_use blocks are paired with their tool_result by id; is_error results become {"error": ...}; assistant text blocks become messages; the closing result row is skipped (the judge already gets the final response).
  • OpenCode events: tool parts (input, then output or error) and text parts; a part re-emitted as its state advances keeps its first position and its latest state.
  • Stream-json is recognised only by rows Claude Code writes (a system row, or a turn whose message is an object). Gemini CLI also closes on {"type": "result"} but must not be read as an empty Claude run; Gemini and Codex trajectories are unrecognised (see the guard below).
  • Tool names are spelled one way in both formats: <server>_<tool>, with a random per-deploy server alias (env + 4 digits from random_mcp_server_name, or the older mcp_<hex>) dropped. Claude Code's mcp__hg__slack_post and OpenCode's hg_slack_post both render as hg_slack_post; a declared server name is kept because OpenCode's names give no way to tell it from the tool name.
  • In the two new parsers, JSON-in-text tool results are decoded (up to 3 levels), because a result like {"result": "{\"id\": ...}"} otherwise renders as mostly escape characters.
• action_log takes max_args_chars / max_result_chars. TrajectoryFilter gains action_log_max_args_chars / action_log_max_result_chars (validated like action_log_max_lines, serialized in to_dict / from_dict). Defaults stay 110 / 160, so existing tasks render exactly as before. rubrics_verifier passes them through.
• Criterion matching and the regex knobs keep using rendered_action at the default cut, as before.
• Transcript records the format it was read from (source). When trajectory_text=action_log and the trajectory is JSON in no format the action log reads, the verifier raises instead of handing the judge an empty log. Text that is not JSON at all stays a failed run, as before.

Testing

• New tst/unit/task_step/verifiers/judge_utils/test_frame_selection_cli_formats.py covers stream-json pairing (errors, images, orphan results), OpenCode parts (status updates, error, prefix), nested-JSON decoding, the new cuts, and that unrecognised lists still give an empty transcript. Its 4 behaviour tests fail on main and pass here.
• test_rubrics_verifier_per_criterion.py: the new filter fields round-trip and are validated, and a stream-json trajectory reaches the direct judge's prompt as an action log at the configured cuts.
• The guard: an unrecognised format raises before any judge call, and an empty trajectory ([]) still reaches the judge. With the guard disabled, its test fails.
• An OpenCode error whose reason is in output keeps the reason, and a [] stream-json result stays apart from no result. Each test fails with its fix reverted.
• Gemini- and Codex-shaped lists are unrecognised; a Claude run that stopped before its first turn is still read as one. Tool names come out the same from both formats.
• Checked on 13 real MCP-agent trajectories (10 Claude Code, 3 OpenCode): no random alias left, and the same tool from a declared server has the same name in both.
• tst/unit/task_step/verifiers/: 362 passed locally. The full suite is left to CI: the local environment lacks some newer dependencies.

Why it matters

On a sample of real MCP-agent runs, a direct judge reading this action log agreed with the agent judge on about 92% of criteria, at a fraction of the latency.

No task's configuration changes in this PR.

🤖 Generated with Claude Code

Confidence Score: 5/5

The PR appears safe to merge; no new actionable issues remain.

Summary

Adds Claude Code and OpenCode tool calls to the judge’s action log, configurable text limits, and a guard against unsupported JSON formats.

• Direct judges can read tool calls from CLI and OpenCode logs.
• The action log can use separate limits for arguments and results.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Raw trajectory JSON] --> B{Recognized format?}
  B -->|OTel| C[Read spans]
  B -->|Claude Code| D[Pair calls and results]
  B -->|OpenCode| E[Read latest tool parts]
  C --> F[Transcript]
  D --> F
  E --> F
  F --> G[Apply configured text limits]
  G --> H[Judge action log]
  B -->|Unsupported nonempty JSON| I[Raise before judging]

Loading

_{Reviews (3) · Last reviewed commit: "fix(verifiers): spell MCP tool names one..." · Reviewed by Greptile}

[earakely-scale]

🤖 Automated Claude routine — Env Pod PR Context Review. This is a bot, not Edgar Arakelyan; not a human review.

Context from outside the diff, on the OpenCode prefix strip.

frame_selection.py:36 drops ^mcp_[0-9a-f]+_ (used at :224), described as "the random mcp_<hex>_ tool prefix". That isn't the alias shape this repo mints today: random_mcp_server_name() (src/agent_env/env/gateway/constants.py:144-146) returns env + 4 random digits, and that's what a deploy draws whenever no MultiEnv name is declared (src/agent_env/providers/env_providers/env_gateway_provider.py:655). env0123 doesn't match [0-9a-f]+ (the n), so for an env deployed by current agent-env the OpenCode action log keeps the prefix — env0123_gmail_list_messages. Only images still minting the legacy hex alias get stripped.

A related naming note on the stream-json path: :192 takes block["name"] verbatim, and cli/env/multi.py:29 documents the agent-visible spelling as mcp__<name>__<tool>. So the same tool can reach the judge as mcp__env0123__gmail_list_messages from one harness and env0123_gmail_list_messages from another, which also reaches rendered_action and the criterion regex knobs.

I haven't checked a real OpenCode export to confirm which spelling it carries, so this is a question about intended coverage rather than a claim the parsing is wrong.

---

Generated by Claude Code

[earakely-scale]

🤖 Automated Claude routine — Env Pod PR Context Review. This is a bot, not Edgar Arakelyan; not a human review.

Context from elsewhere in this repo rather than from the diff: the explorer already parses five CLI trajectory formats, and two of them land awkwardly on the new dispatch.

src/agent_env/explorer/ui/src/lib/parse-trajectory.ts:300-334 dispatches to parse-trajectory-claude-cli.ts, -gemini.ts, -codex.ts, -openclaw.ts and -opencode.ts. This PR's transcript_from_raw (frame_selection.py:113-133) recognises three: OTel, Claude-shaped stream-json (_STREAM_JSON_TYPES, :34) and OpenCode part events.

Gemini slips past the new guard rather than being caught by it. parse-trajectory-gemini.ts:1-28 documents its rows as flat init / message / tool_use / tool_result / result, notes "message.content is plain text; Claude's is an array of typed content blocks", and carries tool calls as top-level rows (tool_name / parameters), not content blocks. Its closing {"type": "result", …} row is in _STREAM_JSON_TYPES, so :129 picks the stream-json branch — but _transcript_from_stream_json requires row["message"] to be a dict with a list content (:179-181), and a Gemini row has no message key at all, so every row is skipped. Running this PR's frame_selection.py over a Gemini- and a Codex-shaped list:

gemini   source='stream-json'  actions=0 msgs=0 log=''
codex    source=None           actions=0 msgs=0 log=''

Since source is 'stream-json' rather than None, the guard at rubrics_verifier.py:867 doesn't fire, and the judge gets exactly the empty action log that guard exists to prevent.

Codex goes the other way. parse-trajectory-codex.ts:1-32 has only thread.started and item.completed rows and states "No per-turn aggregate result record", so no branch matches and trajectory_text=action_log now raises instead of grading. Loud over silent is the intent here — just worth knowing it's a new hard failure for that harness, as it would be for OpenClaw (parse-trajectory-openclaw.ts:14: "Distinct from the CLI stream-json parsers (claude/gemini/codex) and OTEL").

If Gemini and Codex runs aren't in scope for the direct judge, only the first looks worth a line: a format that reports a source but yields nothing is indistinguishable from a run that did nothing. Raising it only because it isn't visible from the diff alone.

---

Generated by Claude Code

[kaihackney-scale]

Replying to the two context-review comments above (Oct 17 and Oct 18). Both are fixed as of a8ebd3e.

Gemini read as an empty stream-json run (255e19f): stream-json is now recognised only by rows Claude Code writes, a system row or a turn whose message is an object. A Gemini list (init / flat message / tool_use / result) and a Codex list (thread.started / item.completed) now both come out with source=None, so the guard raises instead of grading an empty log. Gemini and Codex runs aren't in scope for the direct judge yet, so they fail loudly rather than getting parsers here. A Claude run that stopped before its first turn (system + result only) is still read as an empty Claude run, so it stays a failed run, not an error.

Tool-name spelling (a8ebd3e): I checked real OpenCode exports before changing this. They carry neither the hex alias nor env####; they use the declared server name, hg_<tool>, with nothing in the file that names the server. So stripping the prefix from OpenCode names isn't safe. Instead both formats now render as <server>_<tool>:

• Claude Code's mcp__<server>__<tool> becomes <server>_<tool>, so mcp__hg__slack_post and OpenCode's hg_slack_post match.
• A random per-deploy alias (env + 4 digits from random_mcp_server_name, or the older mcp_<hex>) is dropped in both, since it changes every run. Plain env (the declared default) is kept.

On 13 real trajectories (10 Claude Code, 3 OpenCode), no random alias is left and the same tool from a declared server has the same name in both. A tool served under a random alias still renders bare while one under a declared name keeps it; that follows the deploy, not the harness. The criterion regex knobs match with an unanchored search on the rendered action, so a pattern on the bare tool name matches either.