feat(reference)!: directory-only references + anti-cheat permission window

.claude/commands/coder-eval-task-create.md

@@ -167,7 +167,7 @@ Quick reference of available types:
 | `json_check` | Fractional | `path`, `schema`, `assertions` (JMESPath) |
 | `run_command` | Binary / Continuous | `command`, `expected_exit_code`, `expected_stdout`, `stdout_match` |
 | `file_matches_regex` | Binary | `path`, `pattern` |
-| `reference_comparison` | Continuous | `agent_file` (requires `reference` block) |
+| `reference_comparison` | Continuous | `agent_file`, `reference_file` (requires `reference` block) |
 | `command_executed` | Fractional | `tool_name`, `command_pattern`, `min_count`, `require_success` |
 | `commands_efficiency` | Continuous | `expected_commands` |
 | `classification_match` | Binary | `path`, `expected_label`, `allowed_labels`, `case_sensitive` |

.github/workflows/pr-checks.yml

@@ -377,7 +377,10 @@ jobs:
   e2e-smoke:
     name: E2E Smoke Tests (Real API)
     runs-on: uipath-ubuntu-latest
-    timeout-minutes: 10
+    # 15 (was 10): the bucket now includes anti_cheat_reference, a driver: docker
+    # task that spins its own container on top of the two image builds this job
+    # already does. Headroom, not an expected duration.
+    timeout-minutes: 15
     # Skip on fork PRs where secrets aren't available
     if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
 
@@ -391,12 +394,16 @@ jobs:
       AWS_BEARER_TOKEN_BEDROCK: ${{ secrets.AWS_BEARER_TOKEN_BEDROCK }}
       AWS_REGION: ${{ secrets.AWS_REGION }}
       BEDROCK_MODEL: ${{ secrets.BEDROCK_MODEL }}
-      # tasks_run for --tags smoke-pass. 6 task files (hello_date, dataset_example,
-      # smoke_llm_judge, smoke_agent_judge, byod_smoke_test, agentless_smoke_test);
-      # dataset_example fans out to 2 inline rows, so 7 sub-tasks. If you add/remove a
-      # smoke-pass task or change the dataset row count, bump these.
-      EXPECTED_SMOKE_PASS_RUN: "7"
-      EXPECTED_SMOKE_PASS_SUCCEEDED: "7"
+      # tasks_run for --tags smoke-pass. 7 task files (hello_date, dataset_example,
+      # smoke_llm_judge, smoke_agent_judge, byod_smoke_test, agentless_smoke_test,
+      # anti_cheat_reference); dataset_example fans out to 2 inline rows, so 8
+      # sub-tasks. If you add/remove a smoke-pass task or change the dataset row
+      # count, bump these.
+      #
+      # anti_cheat_reference lives in a SUBDIRECTORY, which `tasks/*.yaml` does not
+      # match — the smoke-pass step names its path explicitly. Keep that in sync.
+      EXPECTED_SMOKE_PASS_RUN: "8"
+      EXPECTED_SMOKE_PASS_SUCCEEDED: "8"
       # smoke-fail bucket: three tasks expected to fail.
       #   1. smoke_negative_path: file_contains criterion is unsatisfiable
       #      (sentinel-string regression detection for success-checker).
@@ -463,9 +470,13 @@ jobs:
       # that Bedrock rejects with 400 (no such cross-region profile). Falling
       # back to BEDROCK_MODEL — a valid pre-formatted Bedrock profile id — is
       # the same pattern live-tests uses (see test_claude_settings_enforcement_live._model_for_env).
+      # `tasks/*.yaml` is NOT recursive, so subdirectory tasks are listed
+      # explicitly. anti_cheat_reference is the adversarial probe that the agent
+      # cannot read the reference solution during its turn; it needs the
+      # coder-eval-agent image built above (it is a driver: docker task).
       - name: Run smoke-pass bucket (expect all to succeed)
         run: |
-          .venv/bin/coder-eval run tasks/*.yaml \
+          .venv/bin/coder-eval run tasks/*.yaml tasks/anti_cheat_reference/*.yaml \
             --tags smoke-pass \
             --run-dir runs/ci-smoke-pass
 

Makefile

@@ -83,13 +83,21 @@ clean:  ## Clean build artifacts and cache
 	rm -rf runs/2025-* runs/latest
 	find . -type d -name __pycache__ -exec rm -rf {} + 2>/dev/null || true
 
+# Task globs. `tasks/*.yaml` does NOT recurse, so every subdirectory holding a
+# task must be listed. Kept identical to the CI e2e-smoke job's globs
+# (.github/workflows/pr-checks.yml) and pinned there by
+# tests/test_tags.py::TestCiSmokePassContract -- when the two drifted, `make
+# test-smoke` silently skipped tasks CI was gating on.
+TASK_GLOBS := tasks/*.yaml tasks/agents/*.yaml tasks/anti_cheat_reference/*.yaml
+SMOKE_GLOBS := tasks/*.yaml tasks/anti_cheat_reference/*.yaml
+
 run:	## Run coder-eval on all tasks with 8 parallel jobs
-	uv run coder-eval run tasks/*.yaml tasks/agents/*.yaml -j 8
+	uv run coder-eval run $(TASK_GLOBS) -j 8
 
 test-smoke:  ## Run e2e smoke tests with real API (mirrors CI "E2E Smoke Tests" job)
-	uv run coder-eval run tasks/*.yaml --tags smoke-pass --model claude-haiku-4-5-20251001
+	uv run coder-eval run $(SMOKE_GLOBS) --tags smoke-pass --model claude-haiku-4-5-20251001
 	@echo "--- now running smoke-fail bucket (expected to exit non-zero) ---"
-	! uv run coder-eval run tasks/*.yaml --tags smoke-fail --model claude-haiku-4-5-20251001
+	! uv run coder-eval run $(SMOKE_GLOBS) --tags smoke-fail --model claude-haiku-4-5-20251001
 
 docker-image:  ## Build the coder-eval-agent image (core + both agents baked in; no creds needed)
 	@VERSION=$$($(VERSION_CMD)); \

docs/DOCKER_ISOLATION.md

@@ -304,6 +304,39 @@ The host's run dir is bind-mounted **read-write** into the container at the same
 
 The host's `DockerRunner` (`coder_eval/isolation/docker_runner.py`) renders the `docker run` argv, bind-mounts task inputs at `/work/input`, allocates an output dir at `/work/output`, and tails container stdout into `docker.log` in the task's run dir.
 
+If the task declares a `reference:` block, a throwaway **copy** of its directory is bind-mounted **read-write** at `/work/references`, and an empty tmpfs is layered over its original location inside the task-directory mount so the agent cannot reach the solution through `$TASK_DIR`. The mount is read-write on purpose — a `:ro` bind mount cannot be chmod'd (EROFS), and the orchestrator holds this path at mode `000` for the duration of each agent turn. The container also runs with `--cap-drop DAC_OVERRIDE --cap-drop DAC_READ_SEARCH` (see the caveat below for why `FOWNER`/`CHOWN` are deliberately kept).
+
+> **What this does and does not guarantee.** The mode-000 window stops an agent
+> that *passively reads* the reference — `ls`, `cat`, `grep` all fail. Two
+> things it does **not** stop:
+>
+> 1. **A deliberate re-chmod.** `chmod(2)` is gated on *owner-or-`CAP_FOWNER`*,
+>    and the container runs as root, which owns the bind-mounted copy. Verified:
+>    `chmod 755 /work/references` succeeds from inside the container and the
+>    solution is then readable. Dropping `FOWNER`/`CHOWN` does **not** fix this
+>    and is deliberately not done — the in-container orchestrator that *applies*
+>    the window is the same root process with the same capabilities, so dropping
+>    `FOWNER` breaks the harness's own `chmod` wherever the bind mount preserves
+>    a non-root owner (native Linux). Verified: container root, uid-1000-owned
+>    directory, `FOWNER` dropped → `chmod: Operation not permitted`. The drop
+>    only bites on the hosts where it also disables the control.
+> 2. **Waiting the window out.** The window spans `agent.communicate`, so
+>    between turns and after the final one the reference is back at its normal
+>    mode, and nothing reaps the agent's child processes at turn end — a
+>    backgrounded read loop started during a turn succeeds once the window
+>    closes.
+>
+> The *write* half of (2) IS closed: the reference tree is hashed at staging
+> time and re-verified before grading, and a mismatch fails the run with
+> `ReferenceTamperedError` rather than scoring `reference_comparison` against a
+> file the agent wrote. **Closing the read half fully requires running the agent
+> as a non-root uid that does not own the reference, and holding the window for
+> the agent's whole lifetime** — tracked as follow-up. Treat the current control
+> as raising the cost of accidental leakage, not as a boundary against an
+> adversarial agent.
+
+See [Reference Solutions](TASK_DEFINITION_GUIDE.md#reference-solutions).
+
 Inside the container, the entrypoint invokes `coder-eval _run-task-internal` (hidden subcommand), which loads the staged YAML + context, runs the standard in-process Orchestrator (driver auto-coerced back to `tempdir`), and writes `task.json` to the output mount. Host reads it and feeds the existing aggregation pipeline.
 
 A `result_kind` discriminator on `CriterionResult` ensures `ClassificationCriterionResult` subclasses survive the JSON round-trip — without it, host-side aggregation would silently lose `observed_label`/`expected_label`.

docs/EXTENDING.md

@@ -161,8 +161,11 @@ from coder_eval.models import CriterionResult
 class MyChecker(BaseCriterion[MyCriterion]):
     criterion_type = "my_criterion"          # must match the model discriminator
 
-    def _check_impl(self, criterion, sandbox, reference_code=None, *,
+    def _check_impl(self, criterion, sandbox, *,
                     turn_records=None, context=None) -> CriterionResult:
+        # `context` carries the live run state: `context.route` (for criteria
+        # that call a model) and `context.reference_dir` (the staged reference
+        # copy, for criteria that grade against a reference solution).
         ok = ...  # your logic
         return CriterionResult(
             criterion_type=self.criterion_type,