Repository navigation
Conversation
Agent setup installs the agent in every trial's container: about two minutes for deepseek-harness on a Terminal-Bench image (apt, nvm, Node, npm), and up to the 360-second timeout when apt or npm is slow. With --cache-agent-setup (environment.cache_agent_setup), the first trial commits its main container, as setup left it and before the agent runs, to hbench-setup-cache:<hash>, and later trials with the same main image, agent, model, options, MCP servers, agent env names, setup user and hbench version start from that image. Setup still runs on it, so an agent that writes per-task config writes it again; an agent whose pinned version is already installed skips its install. deepseek-harness now checks dsh --version against its pin before installing, as claude-code already did. Nothing of the trial reaches the image's config: the cache is used only when the main container sets no env (task env, --ee, egress proxy), has no mounts and no task or extra compose file, so docker commit keeps the base image's env. The container's com.docker.compose.* labels are blanked, or compose's down --rmi local would take the image for the trial project's and remove it. It is not used for oracle and nop, an agent from import_path, skills, a task with MCP servers, a user agent, or version=latest. hbench cache clean removes the images. On adaptive-rejection-sampler (terminal-bench@2.0.1) with deepseek-harness --install-only: agent setup 116-155 s without the image, 1 s from it.
|
@greptileai review |
 [Medium impact] The change affects opted-in Docker trials that meet the setup-cache conditions. Fix task-file carryover and stale image reuse before merging. FindingsSummaryAdds opt-in agent setup images through
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| let plain = egress.is_none() | ||
| && ec.env.is_empty() | ||
| && spec.persistent_env.is_empty() | ||
| && spec.mounts.is_empty() | ||
| && spec.extra_compose.is_empty() | ||
| && !task_compose.exists(); | ||
| let mut build_image = build_image; | ||
| let setup_cache = match (&spec.setup_cache_key, main.get("image").and_then(Value::as_str)) { | ||
| (Some(key), Some(base)) if plain => { | ||
| let image = setup_cache_image(key, base); |
There was a problem hiding this comment.
Tasks sharing a prebuilt image and agent can reuse a cache containing another task’s files. start() uploads env_dir before the container is committed, but the cache key includes neither those files nor their destination. The next task’s upload overwrites matching paths without removing unrelated files, so those files remain and can change the benchmark result.
Disable caching for this upload path, or include the uploaded files and destination in the key.
| let image = setup_cache_image(key, base); | ||
| let hit = docker_output(&["image", "inspect", &image], QUICK_COMMAND_TIMEOUT).await?.status.success(); | ||
| if hit { | ||
| log.line(&format!("agent setup cache: starting from {image}")); | ||
| main.insert("image".into(), json!(image)); | ||
| main.remove("build"); | ||
| build_image = None; |
There was a problem hiding this comment.
A cache hit replaces the main image without checking its current contents and clears build_image even when force_build is set. If a user updates a prebuilt image under the same tag, trials still use the old setup image. Likewise, --force-build cannot rebuild a Dockerfile-backed image when its setup cache exists.
Bypass the setup cache for forced builds and tie reuse to the resolved base image, not just its name.
…r host files devin writes its API key into credentials.toml and gemini-cli copies OAuth credentials during setup; several agents upload host files whose contents are not in the cache key. A committed image would keep those. Agents now opt in with Agent::setup_is_cacheable: claude-code, codex, deepseek-harness and opencode, whose setups only install pinned packages.
…r and the scrubbed files The key held the main image's name, so a pulled tag that now points at another image would still start from the old setup. A pulled image is now keyed by its local id: before start when it is there, else the id the container ran on, read at commit time. An image hbench builds keeps its name, which hashes the build context; the trial removes it, so its id changes with every build. Tasks that share an image also share a cache only when their environment dirs (uploaded into the workdir for prebuilt images) and the verifier-only files removed before setup match.
…agged ones Concurrent trials that all miss the cache each commit the same tag, and each commit untags the one before it. A trial now skips the commit when the image is already there, and hbench cache clean also removes the untagged setup cache images a race can still leave.
A cache hit drops the main image's build, so --force-build would not rebuild a Dockerfile-backed image once its setup cache exists.
Hacktron Security Check - SkippedReason: Code Review is not active for this organization
|
Opt-in (
--cache-agent-setup): the first trial commits its main container after agent setup tohbench-setup-cache:<hash>; later trials start from it when the main image (a built image's name, else the pulled image's id), the environment dir, agent, model, options and hbench version match. Only for agents whose setup writes no credentials or host files (claude-code, codex, deepseek-harness, opencode), only when the container sets no env and has no mounts, egress control or own compose, and never with--force-build. Nothing deletes the images automatically;hbench cache cleanremoves them. deepseek-harness skips its install when its pin is installed. Measured with--install-only: opencode on hello-world, agent setup 185–245 s → 10–12 s; deepseek-harness on adaptive-rejection-sampler, 116–155 s → 1 s. An experiment for review.