fix(agent): append system_prompt to the Claude Code preset instead of replacing

docs/AB_EXPERIMENTS.md

@@ -136,6 +136,11 @@ fair game: `model`, `permission_mode`, `allowed_tools`, `disallowed_tools`,
 `plugins`, `system_prompt` / `system_prompt_file`, `setting_sources`,
 `claude_settings`, `sdk_options`.
 
+> **`system_prompt_mode` is a poor A/B lever.** A `replace` arm sends no default
+> Claude Code prompt *at all*, so the delta measures the missing behavioral guidance
+> (tool-call batching, conciseness), not your prompt text. To A/B prompt *content*,
+> vary `system_prompt` and leave both arms on `append`.
+
 > **Path-resolution gotcha.** Relative file paths in variant config resolve
 > against _different_ base directories depending on the field:
 >

docs/REPORT_SCHEMA.md

@@ -137,6 +137,20 @@ The authoritative per-replicate record.
 `ClaudeAgentOptions` dump), `sandbox_path`, `task_config`
 (`{resolved, source_yaml, source_file, lineage}` — `lineage` maps each field to
 `{value, source, source_detail}` so you can trace which config layer set it).
+`environment_info.system_prompt_semantics` (`"append"` / `"replace"` /
+`"unknown"`) records the system-prompt regime the agent ran with. Every agent
+emits it — the base `Agent` supplies `"unknown"` for an agent that has not
+declared its regime (including out-of-tree plugin agents), so an absent key
+means one thing only: a run predating the marker. Those runs used
+replace-on-set / empty-on-unset semantics on Claude Code and are not
+score-comparable, so consumers should segment on it (absent ⇒ pre-append
+regime; `"unknown"` ⇒ current run, undeclared agent). Codex runs before the
+marker dropped `system_prompt` entirely and Antigravity always appended, so for
+those two the boundary is a reporting change, not a behavioral one.
+`sdk_options.system_prompt` is a `SystemPromptPreset` dict
+(`{type: "preset", preset: "claude_code", exclude_dynamic_sections: true, append?: str}`)
+on append-mode Claude Code runs and a plain string only in replace mode — it is
+no longer `str | null`, so consumers must not string-handle it unconditionally.
 
 **Telemetry/totals:** `total_token_usage` ([TokenUsage](#tokenusage)),
 `command_stats` (`CommandStatistics`), `total_assistant_turns`, `expected_commands` /

docs/agents/ANTIGRAVITY.md

@@ -109,6 +109,14 @@ Antigravity exposes a `thinking_level` field (`minimal` / `low` / `medium` /
 Antigravity-specific — Claude Code and Codex don't take this field. Thinking tokens
 are billed as **output** tokens (see [Telemetry](#telemetry)).
 
+### `system_prompt`
+
+`agent.system_prompt` is passed to the SDK as `system_instructions`, whose string
+shorthand maps to `TemplatedSystemInstructions` — a named section **appended** to
+the harness's default system instructions, never a replacement. This matches the
+append-only semantics of the shared config field across agents (Claude Code appends
+via the `claude_code` preset; Codex via `developer_instructions`).
+
 ### Skills (SKILL.md)
 
 Antigravity supports [Agent Skills](https://agentskills.io/specification)

docs/agents/CLAUDE_CODE.md

@@ -99,8 +99,9 @@ agent:
 | `allowed_tools` | `list[str] \| null` | Tool allowlist. Unset ⇒ all tools allowed. |
 | `disallowed_tools` | `list[str] \| null` | Tool denylist. (`ToolSearch` is always appended for Bedrock parity.) |
 | `plugins` | `list[{type: local, path}]` | Local plugin/skill directories; `$VAR` in `path` is expanded and resolved to an absolute path. |
-| `system_prompt` | `str \| null` | **Replaces** the default system prompt (there is no *append* seam). Mutually exclusive with `system_prompt_file`. |
-| `system_prompt_file` | `str \| null` | Path (relative to the task YAML) loaded into `system_prompt` at resolution. |
+| `system_prompt` | `str \| null` | **Appended** to the default Claude Code system prompt (via the SDK's `claude_code` preset) — the default's behavioral guidance is kept unless `system_prompt_mode: replace` opts out. Mutually exclusive with `system_prompt_file`. An empty or whitespace-only value is treated as unset. |
+| `system_prompt_mode` | `"append"` (default) / `"replace"` | `replace` sends `system_prompt` as the **entire** system prompt (no preset) and requires a non-blank `system_prompt` / `system_prompt_file` (validated at load). Used by judge sub-agents and the user simulator, which must not carry the coding-agent persona; rarely needed in tasks — see [the migration note](#migrating-tasks-that-set-system_prompt). |
+| `system_prompt_file` | `str \| null` | Path (relative to the task YAML) loaded into `system_prompt` at resolution. Works with either `system_prompt_mode`. |
 | `setting_sources` | `list["user"\|"project"\|"local"] \| null` | Which host setting sources the SDK reads. Default resolves to `["project"]`. See [Sandbox isolation](#sandbox-isolation). |
 | `claude_settings` | `str \| dict \| null` | Passed to the SDK `--settings`. A dict is JSON-serialized; a str is a settings file path. Use `permissions.deny` to block tools/paths. |
 | `sdk_options` | `dict` (default `{}`) | Pass-through for `ClaudeAgentOptions` fields Coder Eval doesn't own (e.g. `effort`). Validated at load — an unknown or framework-owned key is a hard error. |
@@ -111,6 +112,47 @@ agent:
 > `setting_sources`, `include_partial_messages`, …) are rejected there — set those
 > through their typed fields or `-D run_limits.*`. MCP servers are not a YAML field.
 
+> **System-prompt reproducibility.** In `append` mode the preset's *dynamic
+> sections* (working directory, git status, auto-memory) are excluded so the system
+> prompt stays identical across runs — the per-run sandbox tempdir path would
+> otherwise be baked into it, breaking prompt caching and run comparability. The
+> SDK re-injects the stripped content into the first user message, so the agent
+> loses nothing. Note the default-prompt baseline tracks the installed Claude Code
+> CLI version; `environment_info.claude_code_cli` in `run.json` records which
+> version a run used, and `environment_info.system_prompt_semantics`
+> (`append` / `replace`) records the prompt regime — runs predating that marker
+> used replace-on-set / empty-on-unset semantics and are not score-comparable.
+
+### Migrating tasks that set `system_prompt`
+
+`system_prompt` used to **replace** Claude Code's default system prompt. It now
+appends to it. If your task or experiment sets `system_prompt`, the agent gains back
+every default behavioral instruction it was previously running without — parallel
+tool-call batching, conciseness rules, the `Read`/`Grep`/`Glob` tool preferences, and
+the default security guardrails.
+
+That is a genuine behavior change, so **scores are not comparable across this
+boundary**. Pick one:
+
+- **Keep the repair (recommended).** Do nothing. Re-baseline any threshold or
+  reference score the task gates on, and expect turn counts to drop on tasks that
+  depend on batched tool calls.
+- **Preserve the old behavior.** Add `system_prompt_mode: replace` to the `agent:`
+  block. The configured prompt again becomes the entire system prompt. Only do this
+  if the task *intends* to run without the default guidance — a prompt that merely
+  adds sandbox policy or a persona does not.
+
+Segment dashboards on `environment_info.system_prompt_semantics` to keep the two
+regimes in separate cohorts. Note the append-mode baseline also tracks the installed
+CLI version, so a `CLAUDE_CODE_VERSION` bump becomes a score-affecting change
+(attributable via `environment_info.claude_code_cli`).
+
+> **Consumers reading `sdk_options.system_prompt`.** The persisted value changed
+> shape: a plain `str` in the old regime, a `SystemPromptPreset` dict
+> (`{type, preset, exclude_dynamic_sections, append}`) in append mode. Code that
+> string-handles that field needs to branch on the type — see
+> [Report schema](../REPORT_SCHEMA.md).
+
 ### Setting fields from the CLI
 
 Any of these merge-resolve through `-D` / `--set` (see

docs/agents/CODEX.md

@@ -213,6 +213,7 @@ The Codex SDK is synchronous. The agent uses `_run_async()` helper to detect and
 | **SDK Type** | Subprocess (CLI via JSON generator) | Sync client (app-server subprocess) |
 | **Command Tracking** | Full telemetry (tool name, params, duration) | Streamed telemetry: shell → `Bash`, apply_patch → `Write` |
 | **Model Selection** | Direct via `--model` or config | `agent.model` pinned into `thread_start` |
+| **System prompt** | `system_prompt` appended to the default prompt (SDK `claude_code` preset) | `system_prompt` passed as `developer_instructions` on top of the Codex base prompt |
 | **Session Resume** | `--resume {session_id}` | Via thread ID |
 | **Permissions** | `permission_mode` + `allowed_tools` | `permission_mode` → sandbox/approval + `allowed_tools`/`disallowed_tools` → thread config |
 | **Tool Enforcement** | Not enforced by Coder Eval wrapper | `enabled_tools` honored; `disabled_tools` NOT enforced by the SDK |
@@ -229,6 +230,20 @@ Run-limit semantics per harness: [Run-Limit Parity](HARNESS_PARITY.md).
 4. **Authentication** - Requires `CODEX_API_KEY` in the environment (point it at whichever endpoint's key you use — OpenAI, gateway, or Azure); the agent calls `login_api_key` when a key is present. `OPENAI_API_KEY`/`AZURE_OPENAI_API_KEY` are NOT read.
 5. **Model field** - `TurnRecord.model_used` reflects the pinned `agent.model`; the Codex `Turn` payload itself doesn't carry the resolved model.
 6. **Skills with Windows paths** - Symlink creation may fail on Windows; agent falls back to copying (slower).
+7. **No `system_prompt_mode`** - `replace` semantics are Claude-Code-only. `system_prompt` is always appended as `developer_instructions`; setting `system_prompt_mode` on a Codex `agent:` block is a validation error (unknown field).
+
+## Migrating tasks that set `system_prompt`
+
+`system_prompt` was previously **ignored** on Codex tasks — silently dropped, so the
+task ran on Codex's base prompt alone. It is now forwarded as
+`developer_instructions`, layered on top of that base prompt. Any Codex task setting
+the field now actually receives those instructions, so **scores are not comparable
+across this boundary**. Two things to check:
+
+- A prompt written for Claude (naming `Read`/`Grep`/`Glob`, or Claude tool etiquette)
+  is now live on Codex, where those tool names don't exist.
+- There is **no opt-out** (see Known Limitations #7). To restore the old behavior,
+  remove `system_prompt` from the Codex variant — otherwise re-baseline.
 
 ## Future Enhancements
 

src/coder_eval/agent.py

@@ -11,7 +11,7 @@
 from .errors import AgentCrashError, TurnTimeoutError
 from .errors.agent import format_timeout_reason, truncate_crash_message
 from .models import AgentState as AgentState
-from .models import BaseAgentConfig, TurnRecord
+from .models import BaseAgentConfig, SystemPromptSemantics, TurnRecord
 from .streaming.callbacks import StreamCallback
 from .streaming.collector import EventCollector
 from .streaming.events import AgentEndStatus
@@ -86,6 +86,16 @@ def __init__(self, config: ClaudeCodeAgentConfig, ...):
     # crash every agent (NoOp/Codex/Antigravity/plugins) whose ``__init__`` lacks it.
     supports_cost_log_tags: ClassVar[bool] = False
 
+    # How this agent combines a configured ``system_prompt`` with its own default
+    # prompt, recorded per run as ``environment_info.system_prompt_semantics``.
+    # Declared on the base (not only on the agents that implement a regime) so the
+    # marker is present on EVERY run: dashboards can then read "absent" as one thing
+    # only — a run from before the marker existed — instead of conflating it with a
+    # plugin agent that never declared. Agents whose regime is fixed set this
+    # ClassVar; an agent whose regime depends on its config (Claude Code) overrides
+    # ``get_environment_info`` and emits the resolved value instead.
+    system_prompt_semantics: ClassVar[SystemPromptSemantics] = "unknown"
+
     def _begin_turn(self) -> None:
         """Mark the start of a ``communicate()`` turn: reset the pending slot and
         bump the iteration counter so a mid-turn failure can be rolled back.
@@ -330,9 +340,14 @@ def get_environment_info(self) -> dict[str, Any]:
         Lets an agent surface non-default endpoint/model routing (e.g. a custom
         base URL or wire protocol) so runs are auditable and comparable across
         operators. The orchestrator merges this into ``environment_info`` after
-        the agent starts. Default: nothing to add.
+        the agent starts.
+
+        The base emits ``system_prompt_semantics`` (from the ClassVar of the same
+        name) so every agent — including out-of-tree SPI agents — records the
+        regime. Overrides should spread ``super().get_environment_info()`` rather
+        than returning a bare dict, or that guarantee is lost for that agent.
 
         Returns:
-            A flat dict of JSON-serializable keys to merge; empty by default.
+            A flat dict of JSON-serializable keys to merge.
         """
-        return {}
+        return {"system_prompt_semantics": self.system_prompt_semantics}

src/coder_eval/agents/antigravity_agent.py

@@ -47,6 +47,7 @@
     CommandTelemetry,
     ContentBlock,
     DirectRoute,
+    SystemPromptSemantics,
     TokenUsage,
     TranscriptMessage,
     TurnRecord,
@@ -232,6 +233,11 @@ class AntigravityAgent(Agent[AntigravityAgentConfig]):
     # ``should_stop`` check runs, so this agent supports early-stop-on-criterion.
     supports_cooperative_stop: ClassVar[bool] = True
 
+    # Antigravity has always appended (TemplatedSystemInstructions wraps
+    # system_instructions around its own harness prompt), so its runs are
+    # comparable across the marker boundary.
+    system_prompt_semantics: ClassVar[SystemPromptSemantics] = "append"
+
     def __init__(
         self,
         config: AntigravityAgentConfig,
@@ -747,6 +753,7 @@ def kill_sync(self) -> None:
     def get_environment_info(self) -> dict[str, Any]:
         """Record the resolved Gemini model + thinking level for auditability."""
         return {
+            **super().get_environment_info(),
             "antigravity_model": self._effective_model(),
             "antigravity_thinking_level": self.config.thinking_level,
         }

src/coder_eval/agents/claude_code_agent.py

@@ -28,6 +28,10 @@
 # its kill target and timeouts will no longer be enforced at the agent layer.
 from claude_agent_sdk._internal.transport.subprocess_cli import SubprocessCLITransport
 
+# SystemPromptPreset is not re-exported from the SDK root, so claude_agent_sdk.types
+# is the only import route (same treatment as evaluation/verdict_tool.py).
+from claude_agent_sdk.types import SystemPromptPreset
+
 from coder_eval.agent import Agent, AgentState
 from coder_eval.agents._logging import PrefixedAdapter, log_raw_sdk_event
 from coder_eval.agents.registry import AgentRegistry
@@ -47,6 +51,7 @@
     DirectRoute,
     LiteLLMRoute,
     ResultSummary,
+    SystemPromptSemantics,
     TokenUsage,
     TranscriptMessage,
     TurnRecord,
@@ -660,6 +665,11 @@ class ClaudeCodeAgent(Agent[ClaudeCodeAgentConfig]):
     # ANTHROPIC_CUSTOM_HEADERS for the proxy-side actual-cost join (LiteLLM backend).
     supports_cost_log_tags: ClassVar[bool] = True
 
+    # One warning per agent for a replace-mode config with no prompt to replace
+    # with: _resolve_system_prompt() runs on every query and once per env-info
+    # snapshot, and a per-turn repeat would bury the rest of task.log.
+    _warned_prompt_mode_downgrade: bool = False
+
     def __init__(
         self,
         config: ClaudeCodeAgentConfig,
@@ -1173,6 +1183,19 @@ def _build_claude_query(
         if "ToolSearch" not in disallowed_tools:
             disallowed_tools.append("ToolSearch")
 
+        # The SDK maps system_prompt=None to `--system-prompt ""` (an explicit
+        # EMPTY custom prompt) and a plain string to a full replacement — either
+        # way Claude Code's default behavioral guidance (parallel tool-call
+        # batching, conciseness) is lost. So ALWAYS send the claude_code preset:
+        # without `append` the CLI runs its default prompt; with it the configured
+        # prompt is appended. exclude_dynamic_sections keeps the prompt static
+        # across runs (the per-run tempdir path would otherwise be baked into the
+        # system prompt, breaking prompt caching and run comparability); the SDK
+        # re-injects the stripped sections into the first user message.
+        # system_prompt_mode="replace" (judge sub-agents) opts out of the preset:
+        # the configured prompt IS the entire system prompt.
+        system_prompt = self._resolve_system_prompt()
+
         # as_posix(), not str(): bash on Windows strips backslashes from unquoted
         # paths, so a redirect like `> D:\foo\bar` ends up writing to "Dfoobar".
         options = ClaudeAgentOptions(
@@ -1192,7 +1215,7 @@ def _build_claude_query(
             # summing per-message values undercounts by 10x+. Without this flag
             # StreamEvents are suppressed by the SDK.
             include_partial_messages=True,
-            system_prompt=self.config.system_prompt,
+            system_prompt=system_prompt,
             setting_sources=self.config.setting_sources if self.config.setting_sources is not None else ["project"],
             resume=self._session_id,
             settings=json.dumps(self.config.claude_settings)
@@ -1216,6 +1239,49 @@ def _build_claude_query(
 
         return options, transport, effective_model
 
+    def _resolve_system_prompt(self) -> str | SystemPromptPreset:
+        """The system-prompt VALUE that actually goes on the wire.
+
+        Single source of truth for both the options builder and the
+        ``system_prompt_semantics`` run-record marker (derived from this value,
+        never re-computed), so the persisted regime can never disagree with what
+        was sent. Returning the value rather than a mode string is what lets the
+        caller skip a type-narrowing re-check of the invariant resolved here.
+
+        ``replace`` requires a configured prompt (the config validator rejects
+        the pair at load, but a mutated or hand-built config falls back to the
+        preset here — fail open to append).
+        """
+        if self.config.system_prompt_mode == "replace":
+            if self.config.system_prompt is not None:
+                return self.config.system_prompt
+            if not self._warned_prompt_mode_downgrade:
+                # Warn once per agent so the downgrade is visible in task.log
+                # rather than only inferable from run.json's marker.
+                self._warned_prompt_mode_downgrade = True
+                logger.warning(
+                    "system_prompt_mode='replace' with no system_prompt — falling back to the claude_code "
+                    + "preset (append regime). run.json records the regime actually used."
+                )
+        preset = SystemPromptPreset(type="preset", preset="claude_code", exclude_dynamic_sections=True)
+        if self.config.system_prompt is not None:
+            preset["append"] = self.config.system_prompt
+        return preset
+
+    def get_environment_info(self) -> dict[str, Any]:
+        """Record which system-prompt regime built this run's prompts.
+
+        ``append`` = the claude_code preset (dynamic sections excluded) with the
+        configured system_prompt, if any, appended; ``replace`` = the configured
+        prompt is the ENTIRE system prompt (judge sub-agents). Unlike the other
+        agents this is per-config, not fixed, so it overrides the base ClassVar
+        with the resolved value. Runs from before this marker existed used
+        replace-on-set / empty-on-unset semantics — trend dashboards must not
+        pool scores across that boundary.
+        """
+        semantics: SystemPromptSemantics = "replace" if isinstance(self._resolve_system_prompt(), str) else "append"
+        return {**super().get_environment_info(), "system_prompt_semantics": semantics}
+
     async def stop(self) -> None:
         """Stop the agent and clean up resources."""
         self.client = None