Repository navigation
Generate swebench-verified@1.0 and swebench-lite@1.0 - #10
Merged
Merged
Conversation
A built version names one image today. environment.task_images is the additive alternative: a glob over task names for each image, every task matching exactly one. Each task names its image in docker_image, by tag until images.lock.json pins it, and its adapter writes environment/, the files hbench copies into the image when the task starts; a per-task build context would make hbench build instead of using the image. The Build images workflow locks one image per run, so lock-image now pins the tasks on its tag and keeps the other images' entries. A lock may pin some images before the others, always all of an image's tasks; such a version validates with a warning, and publishing waits until every image is pinned. The single-image form is unchanged.
210 tasks, 35 practice exercises from each of Exercism's C++, Go, Java, JavaScript, Python and Rust tracks, written by adapters/exercism_polyglot from the problem specifications and the six tracks at pinned commits, each archive pinned by sha256. All seven are MIT; NOTICE carries both of Exercism's notices and Catch2's BSL-1.0 licence. Each task starts with the exercise in /app as the track ships it, tests included, since most C++ stubs are empty. The verifier restores the tests and every other non-solution file from tests/, enables the skipped tests and runs them offline; the oracle copies the track's example solution. Java compiles with javac and runs on the JUnit console launcher instead of Gradle, whose wrapper is left out, and its image carries the public-domain org.json 20260814 rather than the JSON-licensed 20190722. food-chain is left out for its lyrics' copyright. One image per language, built from official Docker Hub bases copied to GHCR. The Exercism polyglot workflow builds each language's image on a GitHub-hosted runner and requires, with hbench, that every oracle passes and every untouched stub fails.
The first CI run found three problems. Exercism's C++ CMakeLists.txt names the build after its directory, so a C++ exercise now sits in /app/<slug>. The canary comment went above a #! line in JavaScript grep's example; it now goes below it. And four refactoring exercises ship stubs that already pass every test, ledger in Go and JavaScript, markdown in Go and tree-building in Java, so they are left out and the ranking fills their places: affine-cipher in JavaScript, go-counting and rectangles in Go, state-of-tic-tac-toe in Java. The Java job also times Gradle against javac on five exercises, once.
#![feature(test)] starts parallel-letter-frequency's benchmark, and the canary comment went below it as if it were a #! line. A shebang is now #! not followed by [, and the adapter's own check finds the canary comment on the first or second line.
adapters/swebench writes one task per SWE-bench instance from SWE-bench's own parquet files at pinned revisions, read with a standard-library Parquet reader (adapters/common/parquet.py). Tasks are named by their instance id. Licences: only instances from repositories under a permissive licence are kept (adapters/swebench/licences.py; MPL counts as copyleft and is dropped). Each repository's licence is what GitHub's licence API reports, cached with its licence file and copyright lines in repo-licences.json; where GitHub asserts none (sympy, sphinx), the licence file was read and the review recorded. Dropped: matplotlib (the Matplotlib License, not OSI-approved; GitHub detects none) and pylint (GPL-2.0). Kept: Verified 456 of 500, Lite 271 of 300. Environments are not redistributed. A new environment policy, built-locally, lets each task's environment/Dockerfile start FROM its own upstream image pinned by digest in images.lock.json (the pinned-upstream shape), which pin_images.py now resolves from the adapter's base_images(). The tasks start from Epoch AI's GHCR images; all 727 resolved. remove_answer_leaks.sh deletes later tags, every other reference, the reflogs and unreachable objects, and fails the build unless only HEAD's history is left (SWE-bench issue #465). Grading is SWE-bench 5.0.2's: a collect hook in the agent's container runs the instance's eval script with the exit-code recording the harness adds and keeps the log as an artifact; a separate python:3.12-slim verifier grades it with SWE-bench's grading code and log parsers, vendored byte-identical (MIT). Reward 1 only when every FAIL_TO_PASS and PASS_TO_PASS test passes. CI: the Sample tasks workflow runs the oracle (must score 1) and nop (0) on 10 tasks per built-locally version on GitHub-hosted runners, and checks the built image keeps no history after the base commit; Sweep tasks, run by hand, does every task in shards.
On the GitHub-hosted runner, in the Java image, an offline Gradle run with warm caches and no daemon took 5.2 to 5.3 s for bowling, forth and react, and javac with the JUnit console launcher 1.3 to 1.4 s. Gradle's first run, which downloads it, took 11.2 s. The verifier keeps javac, which also spares the image Gradle and its caches.
The sample check found that the empty patch resolves psf__requests-863: its FAIL_TO_PASS tests (test_POSTBIN_GET_POST_FILES_WITH_HEADERS and three more) are network tests against httpbin that pass without the fix, and the test the pull request adds is not among them. A task that rewards doing nothing cannot grade an agent, so datasets.py records it as invalid, with why; the adapter and the pin step leave it out and the NOTICE lists it. Lite is 270 tasks.
The plan job of Sample tasks and of Sweep tasks generates the built-locally versions anyway, so sample_tasks.py --copy-to now copies the chosen task directories, and the jobs that run them download that one artifact instead of generating the version again. A version whose upstream files are large is then downloaded once per run rather than once per job. The first sample run took 100 to 250 s and 1.9 to 4.4 GB per task, except psf__requests-2317, whose tests wait on httpbin.org: 38 minutes.
This was referenced Oct 21, 2026
…lyglot # Conflicts: # README.md # docs/licensing.md # images/bases.yaml # images/images.yaml
Built-locally versions and versions on several images both go through GenerationContext: base_images for the first, task_docker_images for the second. versions_built_on skips built-locally versions, which have no built image, and --list skips versions on several images and takes --policy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
SWE-bench Verified (456 of 500 instances) and Lite (271 of 300), generated by
adapters/swebenchfrom SWE-bench's own parquet files at pinned revisions. Tasks are named by instance id.adapters/swebench/licences.py; MPL excluded). Licences come from GitHub's licence API, cached inrepo-licences.json; sympy and sphinx, which GitHub cannot identify, were read by hand. Dropped: matplotlib (34/23) and pylint (10/6).built-locally. Each task'senvironment/Dockerfilestarts FROM its Epoch AI GHCR image pinned by digest (images.lock.json, frompin_images.py) and removes the git history after the base commit. No image is redistributed.Nothing is pushed to a registry.