Repository navigation
Generate bigcodebench@1.0, with Hard as a tagged subset - #7
Merged
Merged
Conversation
…gged subset adapters/bigcodebench writes the 1,140 tasks of BigCodeBench's v0.1.4 split (bigcode/bigcodebench at b74c0d0b) as bigcodebench-<n>: the complete prompt as the instruction, the unittest TestCases and a port of upstream's evaluator (untrusted_check at 09dd993f: calibration with the code prompt, reliability guard, kill guards, swallowed I/O, 240 + 1 s) in tests/, and complete_prompt plus canonical_solution as the oracle. The parquet files are read with a small standard-library reader, so adapters stay stdlib-only. BigCodeBench-Hard is a subset, not a second dataset: its 148 rows equal the full split's, so a second dataset would publish the same tasks twice under two names and split one task's results; hbench selects tasks by name and the descriptor needs no subset field. The Hard file is read for its task IDs only (its card states no licence) and those tasks get the tag "hard". images/bigcodebench builds on python:3.10-slim-bookworm (copied to GHCR as base-python-3.10-slim-bookworm) with upstream's requirements-eval.txt resolved for cp310 as of 2025-02-23, when upstream built its v0.2.4 image, without the generation tooling. python-Levenshtein-wheels is left out: it is unpinned, has no 3.10 wheel and installs the same Levenshtein module as Levenshtein==0.25.0. scikit-image 0.18.0 and four pure-Python packages are built from source with pinned build requirements. The five NLTK data packages the tasks need are baked in from nltk_data at 56512418, pinned by sha256. No task imports a library the image leaves out, so none is excluded. The checker runs the tests as uid 65534 with PYTHONHASHSEED=0; upstream's seed changes per run. The function checker's sandbox is not reused because it calls python -P, which 3.10 lacks, and its verify module imports tomllib. The BigCodeBench workflow builds the image once (or pulls the locked one), hands it to shards as an artifact, and runs each task's oracle (retried once, reported flaky if the retry passes) and a wrong solution: the Hard subset in 4 shards on pull requests, every task on dispatch. Validate's base-image loop skips this dataset.
…e-name task_toml Main's function checker now hands each concurrent checker its own unprivileged id, since each kills every process of its user when its solution ends. The BigCodeBench checker and verify do the same, and the adapter writes task.toml with task_toml(name, description, tables).
…s tests Its tests fetch https://en.wikibooks.org/wiki/Main_Page with urllib's default User-Agent, which Wikimedia now answers with HTTP 403 (T400119), so the canonical solution failed on both runs in CI and can pass nowhere. 1,139 tasks remain, 147 of them in the Hard subset. The image job no longer frees disk: the runner had 86 GB free and the build used 3 GB. Sizes also go to the job log.
A one-off measurement of the full set's checker times and of the tasks whose oracle cannot pass; the next commit restores the Hard subset.
…ard again on pull requests The one-off run of all 1,139 tasks (8 shards, 3 checkers each) found: - 220: two tests open a real turtle Screen, which needs an X display; - 363: the canonical solution's Pool deadlocks on shutdown, since the kill guard, as upstream's, blocks Pool.terminate's os.kill; it timed out on both runs in CI and on 2 of 3 runs locally; - 1005: getsamplefiles.com refuses urllib's default User-Agent (403); - 272: its tests never call task_func, so a wrong solution passes. Every other oracle passed on its first run, and every other wrong solution failed. 1,135 tasks remain, 147 tagged hard. Measured on ubuntu-24.04 runners: the build takes 9 min (8 of them compiling scikit-image 0.18.0), the saved image is 1.0 GB with zstd -3, the checker spends 60 runner-minutes on the full set (2,486 s on oracles, 1,111 s on wrong solutions; median oracle 1 s, slowest 156 s, largest 626 MiB) and 13 on Hard. Pull requests check Hard in 2 shards; a dispatch defaults to 4.
# Conflicts: # README.md # docs/licensing.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
bigcodebench@1.0: 1,135 of the 1,140 tasks of BigCodeBench's v0.1.4 split (Apache-2.0), generated byadapters/bigcodebenchfrom the pinned parquet file, with thecompleteprompt and a port of upstream's unittest harness (calibration, reliability and kill guards, 240 + 1 s).hard(147). Only the Hard file's task IDs are used; its card states no licence.EXCLUDED_TASKS): 220 (turtle needs a display), 363 (Pool deadlocks under the kill guard), 590 and 1005 (sites refuse urllib's User-Agent), 272 (tests never calltask_func). No task needs a library the image drops.bigcodebench-1.0onpython:3.10-slim-bookworm(new base): upstream'srequirements-eval.txt, hash-locked for cp310 as of upstream's 2025-02-23 image, without generation tooling orpython-Levenshtein-wheels, plus five pinned NLTK data packages. 1.0 GB saved with zstd.Nothing is pushed to a registry by this PR.