Issues

• [BUG] The agent's tool can't write its working folder when it exists owned by root (OpenCode on LabBench's image)

  benchflow-ai/benchflow#1183 · Filter by author xdotli opened on Oct 13, 2026

• [BUG] An ACP agent's final reply never reaches /logs/agent, so verifiers that grade the reply read it as empty

  benchflow-ai/benchflow#1182 · Filter by author xdotli opened on Oct 13, 2026

• [BUG] On Daytona, compose tasks that mount their task folder find nothing there

  benchflow-ai/benchflow#1181 · Filter by author xdotli opened on Oct 13, 2026

• [BUG] A verifier that sets PYTHONPATH loses the plugin guard, and pytest refuses to start

  benchflow-ai/benchflow#1180 · Filter by author xdotli opened on Oct 13, 2026

• [BUG] The verifier's environment has no TEST_DIR

  benchflow-ai/benchflow#1179 · Filter by author xdotli opened on Oct 13, 2026

• [BUG] A reward file with extra fields, or a reward above 1, is thrown away

  benchflow-ai/benchflow#1178 · Filter by author xdotli opened on Oct 13, 2026

• [BUG] Verifier hardening deletes every conftest.py in the image, the repository's own included

  benchflow-ai/benchflow#1177 · Filter by author xdotli opened on Oct 13, 2026

• [BUG] The plugin guard distrusts Debian's /usr/local/lib/pythonX.Y, so images' own plugins are never loaded

  benchflow-ai/benchflow#1176 · Filter by author xdotli opened on Oct 13, 2026

• [BUG] Plugin flags BenchFlow adds apply to every pytest the verifier runs

  benchflow-ai/benchflow#1175 · Filter by author xdotli opened on Oct 13, 2026

• [BUG] Images without python3 lose every tool directory on their PATH at verification

  benchflow-ai/benchflow#1174 · Filter by author xdotli opened on Oct 13, 2026

• [BUG] Verifier hardening hides a repository's own conftest.py, so correct patches fail on about a fifth of SWE-bench Verified

  benchflow-ai/benchflow#1171 · Filter by author xdotli opened on Oct 12, 2026

• [BUG] The plugin guard refuses a pytest plugin the verifier installs itself (USACO: every trial fails, oracle included)

  benchflow-ai/benchflow#1170 · Filter by author xdotli opened on Oct 12, 2026

• [BUG] The in-sandbox LiteLLM proxy install fills small task disks, so the agent never starts

  benchflow-ai/benchflow#1169 · Filter by author xdotli opened on Oct 12, 2026

• [BUG] Verifier hardening can't chown matplotlib's build tree, so matplotlib's SWE-bench Verified tasks end in a verifier failure

  benchflow-ai/benchflow#1168 · Filter by author xdotli opened on Oct 12, 2026

• Evidence capture rejects an absolute in-workspace symlink with '..' parts ("Unsafe evidence path"): a verified cell never gets a review

  benchflow-ai/benchflow#1157 · Filter by author Benjamin-eecs opened on Oct 10, 2026

• skill-eval judge can score with a different provider than judge_model (content[0].text on thinking replies, key-based provider fallthrough)

  benchflow-ai/benchflow#1154 · Filter by author snazzybunny opened on Oct 8, 2026

• docker sandbox: a runner killed by a signal leaves its container running (no SIGTERM/SIGHUP handler; prune only removes stopped containers)

  benchflow-ai/benchflow#1153 · Filter by author Benjamin-eecs opened on Oct 8, 2026

• LiteLLM drop_params silently strips reasoning_effort on azure-foundry-openai routes LiteLLM does not list it for (azure/grok-4.7): run says xhigh, model runs at default

  benchflow-ai/benchflow#1152 · Filter by author Benjamin-eecs opened on Oct 8, 2026

• Egress firewall: iptables install and rules share one fixed 120 s exec, so a slow apt mirror or docker exec kills the launch before the first prompt

  benchflow-ai/benchflow#1151 · Filter by author Benjamin-eecs opened on Oct 8, 2026

• Docker sandbox image tag depends only on the task directory name, so concurrent rollouts collide (No such image, wrong image)

  benchflow-ai/benchflow#1149 · Filter by author EYH0602 opened on Oct 2, 2026

• Reviewer stage is not retried after a Daytona PTY transport reset (close_code=1006): rollout ends without a verdict

  benchflow-ai/benchflow#1144 · Filter by author Benjamin-eecs opened on Sep 25, 2026

• bench eval score: Daytona PTY readline timeout stays at the 900 s default regardless of --reviewer-timeout-sec

  benchflow-ai/benchflow#1143 · Filter by author Benjamin-eecs opened on Sep 25, 2026

• Stale .scoring.lock after a reviewer death makes every resume crash in resume_pending_reviews (Event loop stopped before Future completed)

  benchflow-ai/benchflow#1142 · Filter by author Benjamin-eecs opened on Sep 25, 2026

• ACP idle watchdog kills an active session over one pending tool call whose terminal update was lost (per-call grace ignores session activity)

  benchflow-ai/benchflow#1141 · Filter by author Benjamin-eecs opened on Sep 25, 2026

• claude-agent-acp cannot use an Anthropic-compatible third-party endpoint directly: the mandatory LiteLLM bridge rewrites /v1/messages to chat/completions and loses prompt caching (Kimi K3 on Baseten, GLM on Z.ai)

  benchflow-ai/benchflow#1140 · Filter by author Benjamin-eecs opened on Sep 23, 2026