Repository navigation
Generate swebench-multilingual@1.0 and multi-swe-bench@1.0 - #11
Merged
Merged
Conversation
A built version names one image today. environment.task_images is the additive alternative: a glob over task names for each image, every task matching exactly one. Each task names its image in docker_image, by tag until images.lock.json pins it, and its adapter writes environment/, the files hbench copies into the image when the task starts; a per-task build context would make hbench build instead of using the image. The Build images workflow locks one image per run, so lock-image now pins the tasks on its tag and keeps the other images' entries. A lock may pin some images before the others, always all of an image's tasks; such a version validates with a warning, and publishing waits until every image is pinned. The single-image form is unchanged.
210 tasks, 35 practice exercises from each of Exercism's C++, Go, Java, JavaScript, Python and Rust tracks, written by adapters/exercism_polyglot from the problem specifications and the six tracks at pinned commits, each archive pinned by sha256. All seven are MIT; NOTICE carries both of Exercism's notices and Catch2's BSL-1.0 licence. Each task starts with the exercise in /app as the track ships it, tests included, since most C++ stubs are empty. The verifier restores the tests and every other non-solution file from tests/, enables the skipped tests and runs them offline; the oracle copies the track's example solution. Java compiles with javac and runs on the JUnit console launcher instead of Gradle, whose wrapper is left out, and its image carries the public-domain org.json 20260814 rather than the JSON-licensed 20190722. food-chain is left out for its lyrics' copyright. One image per language, built from official Docker Hub bases copied to GHCR. The Exercism polyglot workflow builds each language's image on a GitHub-hosted runner and requires, with hbench, that every oracle passes and every untouched stub fails.
The first CI run found three problems. Exercism's C++ CMakeLists.txt names the build after its directory, so a C++ exercise now sits in /app/<slug>. The canary comment went above a #! line in JavaScript grep's example; it now goes below it. And four refactoring exercises ship stubs that already pass every test, ledger in Go and JavaScript, markdown in Go and tree-building in Java, so they are left out and the ranking fills their places: affine-cipher in JavaScript, go-counting and rectangles in Go, state-of-tic-tac-toe in Java. The Java job also times Gradle against javac on five exercises, once.
#![feature(test)] starts parallel-letter-frequency's benchmark, and the canary comment went below it as if it were a #! line. A shebang is now #! not followed by [, and the adapter's own check finds the canary comment on the first or second line.
adapters/swebench writes one task per SWE-bench instance from SWE-bench's own parquet files at pinned revisions, read with a standard-library Parquet reader (adapters/common/parquet.py). Tasks are named by their instance id. Licences: only instances from repositories under a permissive licence are kept (adapters/swebench/licences.py; MPL counts as copyleft and is dropped). Each repository's licence is what GitHub's licence API reports, cached with its licence file and copyright lines in repo-licences.json; where GitHub asserts none (sympy, sphinx), the licence file was read and the review recorded. Dropped: matplotlib (the Matplotlib License, not OSI-approved; GitHub detects none) and pylint (GPL-2.0). Kept: Verified 456 of 500, Lite 271 of 300. Environments are not redistributed. A new environment policy, built-locally, lets each task's environment/Dockerfile start FROM its own upstream image pinned by digest in images.lock.json (the pinned-upstream shape), which pin_images.py now resolves from the adapter's base_images(). The tasks start from Epoch AI's GHCR images; all 727 resolved. remove_answer_leaks.sh deletes later tags, every other reference, the reflogs and unreachable objects, and fails the build unless only HEAD's history is left (SWE-bench issue #465). Grading is SWE-bench 5.0.2's: a collect hook in the agent's container runs the instance's eval script with the exit-code recording the harness adds and keeps the log as an artifact; a separate python:3.12-slim verifier grades it with SWE-bench's grading code and log parsers, vendored byte-identical (MIT). Reward 1 only when every FAIL_TO_PASS and PASS_TO_PASS test passes. CI: the Sample tasks workflow runs the oracle (must score 1) and nop (0) on 10 tasks per built-locally version on GitHub-hosted runners, and checks the built image keeps no history after the base commit; Sweep tasks, run by hand, does every task in shards.
On the GitHub-hosted runner, in the Java image, an offline Gradle run with warm caches and no daemon took 5.2 to 5.3 s for bowling, forth and react, and javac with the JUnit console launcher 1.3 to 1.4 s. Gradle's first run, which downloads it, took 11.2 s. The verifier keeps javac, which also spares the image Gradle and its caches.
The sample check found that the empty patch resolves psf__requests-863: its FAIL_TO_PASS tests (test_POSTBIN_GET_POST_FILES_WITH_HEADERS and three more) are network tests against httpbin that pass without the fix, and the test the pull request adds is not among them. A task that rewards doing nothing cannot grade an agent, so datasets.py records it as invalid, with why; the adapter and the pin step leave it out and the NOTICE lists it. Lite is 270 tasks.
The plan job of Sample tasks and of Sweep tasks generates the built-locally versions anyway, so sample_tasks.py --copy-to now copies the chosen task directories, and the jobs that run them download that one artifact instead of generating the version again. A version whose upstream files are large is then downloaded once per run rather than once per job. The first sample run took 100 to 250 s and 1.9 to 4.4 GB per task, except psf__requests-2317, whose tests wait on httpbin.org: 38 minutes.
swebench-multilingual@1.0 is the third dataset of adapters/swebench: 283 of SWE-bench Multilingual's 300 instances, from SWE-bench's own Docker Hub images pinned by digest, since Epoch AI publishes none for it. terraform (BUSL-1.1) and redis (RSALv2, SSPLv1 or AGPLv3) are left out; javaparser is taken under the Apache-2.0 of its LGPL-or-Apache offer; GitHub cannot identify the licences of jq, micropython, lombok and fpm, whose licence files are MIT. multi-swe-bench@1.0 is generated by adapters/multi_swe_bench from the 39 per-repository JSONL files of ByteDance-Seed/Multi-SWE-bench at 56ff018c (data CC0, harness Apache-2.0), with the same task layout (adapters/swebench/tasks.py). 1,594 of 1,632 instances: logstash, whose x-pack code is under the Elastic License, is left out; zstd is taken under BSD-3-Clause. Each environment starts from mswebench/<org>_m_<repo>:pr-<n> pinned by digest (all 1,594 resolved with HEAD requests) and deletes /home/fix.patch and /home/test.patch as well as the later history. The collect hook resets the test patch's files, restores the test patch and runs the image's own test-run.sh; the verifier grades the log with the repository's parse_log and the harness's Report and the category checks of its evaluation, vendored byte-identical from multi-swe-bench at 24f493f8, in a python:3.12-slim image that installs dataclasses-json and unidiff from hash-pinned requirements. Its upstream files are 1.8 GB, too large to fetch here, so generated.lock.json is left for CI to compute: generated_source.py now writes the lock it computes to its output, and the generated job uploads it when it differs.
…or CI to fill The generated job runs only once the descriptors validate, so the lock names the 1,594 tasks with zero digests; that job then refuses them and uploads the lock it computes, which replaces this one.
From the lock the generated job computed (run 39686163254). The 489 tasks of its small upstream files, generated here, have the same digests.
The sample check ran out of time on ponylang__ponyc-3650 in both trials: its test-run.sh builds LLVM before ponyc and was at 74% after the 1800 s that SWE-bench's harness allows. Multi-SWE-bench's harness sets no limit, so its tasks now allow three hours (RepositoryIssue.evaluation_timeout_seconds; SWE-bench's stay at 1800 s), and a sample job may take six hours. The lock of digests is recomputed by CI.
…uation From the lock the generated job computed (run 39694657247); the 489 tasks of the small upstream files, generated here, have the same digests.
…lyglot # Conflicts: # README.md # docs/licensing.md # images/bases.yaml # images/images.yaml
Built-locally versions and versions on several images both go through GenerationContext: base_images for the first, task_docker_images for the second. versions_built_on skips built-locally versions, which have no built image, and --list skips versions on several images and takes --policy.
…ed-swebench-multi # Conflicts: # README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Built on #10 (its commits show here until it merges).
adapters/swebench. Environments start from SWE-bench's Docker Hub images pinned by digest. Dropped: terraform (BUSL-1.1), redis (RSALv2/SSPLv1/AGPLv3).adapters/multi_swe_benchwith the same task layout. Environments start frommswebench/*:pr-<n>pinned by digest, with/home/fix.patch,/home/test.patchand the later history removed. The collect hook runs the image'stest-run.sh; grading uses the repository'sparse_logandReport, vendored from the harness at 24f493f8. Dropped: logstash (Elastic License in x-pack).generated.lock.jsonis computed in CI: the generated job now uploads the lock it computes when it differs.Nothing is pushed to a registry.