diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 379b205..1d184b4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -90,8 +90,9 @@ version is a generated one; see [`datasets/humaneval-plus/1.0`](datasets/humanev it, so a change to the adapter, its shared helpers or the image's build context is committed with the lock it produces. 5. **After the pull request is merged**, a maintainer runs the Build images workflow and commits the - `images.lock.json` and `generated.lock.json` it writes, as the README describes. Only then can the - version be published. + `images.lock.json` and `generated.lock.json` it writes, as + [docs/PUBLISHING.md](docs/PUBLISHING.md#building-images) describes. Only then can the version be + published. ## Versions are immutable diff --git a/README.md b/README.md index c402df5..6072b21 100644 --- a/README.md +++ b/README.md @@ -1,67 +1,160 @@ +```text +██╗ ██╗██████╗ ███████╗███╗ ██╗ ██████╗██╗ ██╗ +██║ ██║██╔══██╗██╔════╝████╗ ██║██╔════╝██║ ██║ +███████║██████╔╝█████╗ ██╔██╗ ██║██║ ███████║ +██╔══██║██╔══██╗██╔══╝ ██║╚██╗██║██║ ██╔══██║ +██║ ██║██████╔╝███████╗██║ ╚████║╚██████╗██║ ██║ +╚═╝ ╚═╝╚═════╝ ╚══════╝╚═╝ ╚═══╝ ╚═════╝╚═╝ ╚═╝ + +██████╗ █████╗ ████████╗ █████╗ ███████╗███████╗████████╗███████╗ +██╔══██╗██╔══██╗╚══██╔══╝██╔══██╗██╔════╝██╔════╝╚══██╔══╝██╔════╝ +██║ ██║███████║ ██║ ███████║███████╗█████╗ ██║ ███████╗ +██║ ██║██╔══██║ ██║ ██╔══██║╚════██║██╔══╝ ██║ ╚════██║ +██████╔╝██║ ██║ ██║ ██║ ██║███████║███████╗ ██║ ███████║ +╚═════╝ ╚═╝ ╚═╝ ╚═╝ ╚═╝ ╚═╝╚══════╝╚══════╝ ╚═╝ ╚══════╝ +``` + # hbench-datasets -Benchmark datasets for [hbench](https://github.com/node-q/hbench) (`node-q/hbench`), the CLI that -runs AI agents on benchmark tasks in Docker, published as signed OCI artifacts. +Benchmark datasets for [hbench](https://github.com/node-q/hbench), published as signed OCI artifacts +at `ghcr.io/node-q/hbench-datasets`. -**Nothing is published yet.** The publish workflow is ready, but it refuses to run until a -node-q/hbench release has `hbench datasets push`. +Each dataset version is one OCI image manifest, tagged with the version, with one reproducible layer +per task. hbench pulls it with `--registry oci://ghcr.io/node-q/hbench-datasets` and checks every layer +against the manifest's digest before it unpacks a task. Every version is signed with cosign keyless, +carries SLSA build provenance and Croissant 1.1 metadata, and has a GitHub release with its digest. -## What's here +- [Installation](#installation) +- [Usage](#usage) +- [Examples](#examples) +- [Datasets](#datasets) +- [Environments](#environments) · [Verifying a dataset](#verifying-a-dataset) · [Offline use and mirrors](#offline-use-and-mirrors) · [Licensing](#licensing) +- [Development](#development) +- [License](#license) +- [Contributors and contact](#contributors-and-contact) -| Path | What it is | -| --- | --- | -| `datasets///source.yaml` | Where a dataset version's task directories come from (git URL, full commit and path, or the adapter that generates them and the upstream files it reads) and what the published dataset declares: licence, upstream and citation, canary string, environment policy, attribution. | -| `datasets///LICENSE`, `NOTICE` | The dataset's own licence and attribution, published with it. | -| `datasets///images.lock.json` | For a version that pins its environment images by digest: each task's upstream or built image, its digest, and what its `docker_image` is pinned to. | -| `datasets///generated.lock.json` | For a generated version: each task's hbench package digest. | -| `adapters/` | The adapters that generate task directories, and `adapters/common`, what they share. | -| `images/` | The environment images this repository builds (`images.yaml`) and the base images they start from (`bases.yaml`). | -| [`schema/source.v1.json`](schema/source.v1.json) | The JSON Schema for `source.yaml`. | -| `scripts/` | Descriptor validation, fetching a source at its pinned commit, the pushed-manifest check, the Croissant generator, a verified hbench install, and the image mirror and pin scripts. | -| [`docs/licensing.md`](docs/licensing.md) | Which datasets can be republished, and why. | +## Installation -No task files are committed. CI fetches each git source at its pinned commit, or generates a -generated source's tasks from its pinned upstream files, and packs them. +Requirements: hbench 0.3.0 or later, and a Docker daemon. Nothing from this repository needs to be +installed: hbench pulls a dataset when a run names it. -Each dataset version is one OCI image manifest at -`ghcr.io/node-q/hbench-datasets/:`: +Install the latest hbench release: -- `artifactType: application/vnd.hbench.dataset.v1`; -- a config blob, `application/vnd.hbench.dataset.config.v1+json`; -- one reproducible `tar+gzip` layer per task, which unpacks byte-identical to the upstream task - directory. +```bash +curl -fsSL https://raw.githubusercontent.com/node-q/hbench/master/scripts/install.sh | sh +``` -It is signed with cosign keyless (GitHub OIDC), carries SLSA build provenance from GitHub artifact -attestations, and has a Croissant 1.1 JSON-LD file attached for discovery. Tags are never moved: a -new version is a new tag. Each dataset is also mirrored to a second registry, not chosen yet. +`HBENCH_VERSION=vX.Y.Z` pins a release and `HBENCH_INSTALL_DIR` overrides `~/.local/bin` (see +[hbench's README](https://github.com/node-q/hbench)). To check signatures and +provenance, or to copy a dataset without hbench, you also need [cosign](https://github.com/sigstore/cosign), +the [GitHub CLI](https://cli.github.com) or [oras](https://oras.land). -| Dataset | Tasks | Licence | Status | -| --- | ---: | --- | --- | -| `terminal-bench@2.0` | 89 | Apache-2.0 | Configured, not published yet | -| `terminal-bench@2.0.1` | 89 | Apache-2.0 | Configured | -| `humaneval-plus@1.0` | 164 | Apache-2.0 | Generated; waits for its image to be built | -| `humanevalfix@1.0` | 164 | Apache-2.0 | Generated; waits for its image lock | -| `mbpp-plus@1.0` | 378 | Apache-2.0 | Generated; waits for its image lock | -| `cruxeval-input@1.0` | 800 | Apache-2.0 | Generated; waits for its image lock | +## Usage -## Running a dataset +```bash +# Run +hbench run -d terminal-bench@2.0.1 --registry oci://ghcr.io/node-q/hbench-datasets -a oracle +hbench run -d humaneval-plus@1.0 --registry oci://ghcr.io/node-q/hbench-datasets --lock-write # record the digest in ./hbench.lock +hbench run -d humaneval-plus@1.0 --registry oci://ghcr.io/node-q/hbench-datasets --locked # refuse any other digest -This needs an hbench release with the OCI read path (`--registry oci://…`), which is being built. +# Download and copy +hbench datasets download humaneval-plus@1.0 --registry oci://ghcr.io/node-q/hbench-datasets -o ./cache +hbench datasets export -d terminal-bench@2.0.1 --registry oci://ghcr.io/node-q/hbench-datasets --oci-layout ./hbench-layout +hbench datasets list --registry oci-layout:./hbench-layout +hbench run -d terminal-bench@2.0.1 --registry oci-layout:./hbench-layout -```sh -hbench run -d terminal-bench@2.0 --registry oci://ghcr.io/node-q/hbench-datasets ``` -hbench resolves `terminal-bench:2.0` to its manifest digest and checks every layer against it -before unpacking a task. In `terminal-bench@2.0` the tasks pull their prebuilt environment images -from Docker Hub by tag (`alexgshaw/:20251031`). `terminal-bench@2.0.1` is the same tasks with -each image pinned by digest to its mirror (see [Mirrored environments](#mirrored-environments)). +`--registry oci://ghcr.io/node-q/hbench-datasets` names the namespace, and `-d @` +picks the repository `` in it and the tag ``. A registry can't be listed, so a run +names its dataset; an OCI layout on disk can be listed. In a job config, the dataset sets +`registry: "oci://ghcr.io/node-q/hbench-datasets"` with its `name` and `version`, and may set +`digest: "sha256:..."` to run that manifest whatever the tag says. + +`--lock-write` records each dataset's manifest digest in `./hbench.lock`, keyed by `name@version`. A +dataset in `hbench.lock` runs at its locked digest, and `--locked` refuses one that is not in it or +whose tag now resolves to another digest. To check a dataset's signature and provenance, see +[Verifying a dataset](#verifying-a-dataset). + +## Examples + +Check the setup with the oracle agent, which runs each task's reference solution, on five tasks: + +```bash +hbench run -d humaneval-plus@1.0 --registry oci://ghcr.io/node-q/hbench-datasets -a oracle -l 5 +``` + +Run Claude Code on Terminal-Bench 2.0, with every task's image pinned by digest: + +```bash +ANTHROPIC_API_KEY=... hbench run -d terminal-bench@2.0.1 --registry oci://ghcr.io/node-q/hbench-datasets \ + -a claude-code -m anthropic/claude-opus-5-5 -n 4 +``` + +Keep a series of runs on the same bytes: record the digests once, commit `hbench.lock`, and run with +`--locked` from then on: + +```bash +hbench run -d mbpp-plus@1.0 --registry oci://ghcr.io/node-q/hbench-datasets -a oracle -l 1 --lock-write +hbench run -d mbpp-plus@1.0 --registry oci://ghcr.io/node-q/hbench-datasets -a codex -m openai/gpt-5 --locked +``` + +Browse the results in hbench's web viewer: + +```bash +hbench view jobs +``` + +## Datasets + +| Dataset | Tasks | What it tests | Licence | +|---|---:|---|---| +| `terminal-bench@2.0` | 89 | Work in a terminal: build, debug, configure, train. Images from Docker Hub, by the tags upstream declares | Apache-2.0 | +| `terminal-bench@2.0.1` | 89 | The same tasks, with each image pinned by digest to node-q's mirror on GHCR | Apache-2.0 | +| `humaneval-plus@1.0` | 164 | Write a Python function from its signature and docstring: HumanEval with EvalPlus's extended tests | Apache-2.0 | +| `humanevalfix@1.0` | 164 | Fix a bug written by hand into a HumanEval function (BigCode's HumanEvalPack) | Apache-2.0 | +| `mbpp-plus@1.0` | 378 | Write a Python function from a description and an example: MBPP with EvalPlus's extended tests | Apache-2.0 | +| `cruxeval-input@1.0` | 800 | Given a Python function and an output, find an input that gives it (CRUXEval-I) | Apache-2.0 | + +Each version has a [release](https://github.com/node-q/hbench-datasets/releases) named +`-`, such as +[`terminal-bench-2.0.1`](https://github.com/node-q/hbench-datasets/releases/tag/terminal-bench-2.0.1): +its reference and manifest digest, task count, licence and upstream commit, how to run and check it, +and its `croissant.json`, `source.yaml`, `LICENSE` and `NOTICE`. A published tag never moves; a fix is +a new version. Each dataset will also be mirrored to a second registry, not chosen yet. + +A version is one OCI image manifest at `ghcr.io/node-q/hbench-datasets/:`: + +- `artifactType: application/vnd.hbench.dataset.v1`; +- a config blob, `application/vnd.hbench.dataset.config.v1+json`, with the dataset's licence, + upstream, canary and each task's digests; +- one reproducible `tar+gzip` layer per task, which unpacks byte-identical to the task directory + it was packed from. + +hbench's README describes the [format](https://github.com/node-q/hbench#dataset-format) in full. + +## Environments + +A task runs in the Docker image its `task.toml` names as `docker_image`. With none, or with +`--force-build`, hbench builds the task's `environment/Dockerfile` instead. A version's environment +policy says where its images come from: + +| Policy | Images | Versions | +|---|---|---| +| `as-declared` | Upstream's images, by the tags upstream declares. Upstream can move a tag | `terminal-bench@2.0` | +| `pinned-mirror` | Upstream's images, copied unmodified with their digests to `ghcr.io/node-q/hbench-envs/`, and named by digest | `terminal-bench@2.0.1` | +| `pinned-upstream` | Upstream's images, named by digest and not copied: for images that are not ours to redistribute or are too large to mirror | none yet | +| `built` | Images this repository builds, tags of `ghcr.io/node-q/hbench-envs`, named by digest. Each starts `FROM` a base image copied to GHCR, so neither a build nor a run needs Docker Hub | `humaneval-plus@1.0`, `humanevalfix@1.0`, `mbpp-plus@1.0` and `cruxeval-input@1.0`, which share `humaneval-plus-1.0` | + +A version that pins its images has an `images.lock.json` next to its `source.yaml`, listing each +task's image and digest. [docs/PUBLISHING.md](docs/PUBLISHING.md) describes how the images are +mirrored, pinned and built. ## Verifying a dataset The signature, made by this repository's publish workflow: -```sh +```bash cosign verify ghcr.io/node-q/hbench-datasets/terminal-bench:2.0 \ --certificate-identity-regexp '^https://github.com/node-q/hbench-datasets/.github/workflows/publish.yml@' \ --certificate-oidc-issuer https://token.actions.githubusercontent.com @@ -69,87 +162,105 @@ cosign verify ghcr.io/node-q/hbench-datasets/terminal-bench:2.0 \ The build provenance: -```sh +```bash gh attestation verify oci://ghcr.io/node-q/hbench-datasets/terminal-bench:2.0 --repo node-q/hbench-datasets ``` -The Croissant metadata is a referrer of the dataset's manifest: +The Croissant metadata, a referrer of the dataset's manifest: -```sh +```bash oras discover --artifact-type application/ld+json ghcr.io/node-q/hbench-datasets/terminal-bench:2.0 ``` -## Mirroring +A mirrored image, with its digest from the version's `images.lock.json`, is signed by the mirror +workflow. Its digest is upstream's own: `crane digest alexgshaw/:20251031` prints the same one +for as long as upstream keeps the tag where it is. + +```bash +cosign verify ghcr.io/node-q/hbench-envs/terminal-bench@sha256: \ + --certificate-identity-regexp '^https://github.com/node-q/hbench-datasets/.github/workflows/mirror-images.yml@' \ + --certificate-oidc-issuer https://token.actions.githubusercontent.com +``` + +A built image is signed by the build workflow: -```sh +```bash +cosign verify ghcr.io/node-q/hbench-envs@sha256: \ + --certificate-identity-regexp '^https://github.com/node-q/hbench-datasets/.github/workflows/build-images.yml@' \ + --certificate-oidc-issuer https://token.actions.githubusercontent.com +``` + +## Offline use and mirrors + +`hbench datasets export` copies a dataset into an OCI image layout, `/`, and hbench reads +it with `--registry oci-layout:`: + +```bash +hbench datasets export -d terminal-bench@2.0 --registry oci://ghcr.io/node-q/hbench-datasets --oci-layout ./hbench-layout +hbench run -d terminal-bench@2.0 --registry oci-layout:./hbench-layout +``` + +`oras cp ghcr.io/node-q/hbench-datasets/terminal-bench:2.0 --to-oci-layout ./hbench-layout/terminal-bench:2.0` +writes the same layout without hbench. The tasks' environment images are not part of the layout: +pull or load them before going offline. + +To mirror a dataset to another registry: + +```bash oras copy -r ghcr.io/node-q/hbench-datasets/terminal-bench:2.0 //terminal-bench:2.0 ``` `-r` copies the referrers too: the signature, the attestation and the Croissant metadata. Digests do not change, so the mirror verifies with the same commands, and hbench reads it with -`--registry oci:///`. +`--registry oci:///`. `hbench.lock` is keyed by `name@version`, not by registry, +so a locked run accepts the mirror only because it serves the same digest. -## Mirrored environments +## Licensing -A dataset version whose environment policy is `pinned-mirror` runs its tasks' prebuilt images from -node-q's mirror. The **Mirror images** workflow copies each image, registry to registry, unmodified -and with its digest unchanged, to one GHCR package per dataset, `ghcr.io/node-q/hbench-envs/`, -tagged `-` (for example -`ghcr.io/node-q/hbench-envs/terminal-bench:adaptive-rejection-sampler-20251031`). It signs each digest -with cosign keyless and writes `images.lock.json`. Each task's `task.toml` then names its image by -digest, `ghcr.io/node-q/hbench-envs/terminal-bench@sha256:…`, and nothing else in it changes. +The tooling in this repository is Apache-2.0 ([`LICENSE`](LICENSE)). Each dataset keeps its own +licence, in its directory and published with it. -To verify a mirrored image, take its digest from the version's `images.lock.json`: +- **terminal-bench@2.0** is Apache-2.0. +- **terminal-bench@2.0.1** is the same tasks, Apache-2.0. Its environment images are redistributed + unmodified from upstream's published images; the software in them carries its own licences. +- **humaneval-plus@1.0** is generated by node-q from HumanEval+ (EvalPlus, Apache-2.0), which + extends OpenAI's HumanEval (MIT). Its task directories are Apache-2.0, with both notices in its + NOTICE. +- **humanevalfix@1.0** (from BigCode's HumanEvalPack, MIT), **mbpp-plus@1.0** (from EvalPlus's + MBPP+, Apache-2.0, derived from Google's MBPP, CC BY 4.0) and **cruxeval-input@1.0** (from Meta's + CRUXEval, MIT) are generated by node-q. Their task directories are Apache-2.0, with every upstream + notice in their NOTICE. +- **swebench-verified@1.0 and aider-polyglot@1.0** have MIT data, but their task directories come + from a third-party shared task repository with no LICENSE file. They need that repository's + permission, or task directories of our own, which would be new versions. +- **livecodebench@6.0** has an unspecified "cc" licence and problems scraped from contest sites. It + is not republished. +- **SWE-bench's prebuilt images** embed GPL-2.0 code among other licences. They are pinned upstream + by digest, not mirrored. -```sh -cosign verify ghcr.io/node-q/hbench-envs/terminal-bench@sha256: \ - --certificate-identity-regexp '^https://github.com/node-q/hbench-datasets/.github/workflows/mirror-images.yml@' \ - --certificate-oidc-issuer https://token.actions.githubusercontent.com -``` +The details are in [`docs/licensing.md`](docs/licensing.md). -The digest is upstream's own: `crane digest alexgshaw/:20251031` prints the same one for as -long as upstream keeps the tag where it is. - -A version whose policy is `pinned-upstream` pins upstream's images by digest without copying them, -for images that are not ours to redistribute or are too large to mirror. The **Pin images** workflow -resolves each tag with a registry HEAD request, which Docker Hub does not count as a pull, and writes -`images.lock.json`. - -`images.lock.json` maps each task, sorted, to the image upstream declares (`source`), its digest -(`source_digest`) and what `task.toml`'s `docker_image` becomes (`pinned`); a mirrored image also has -its tag in the mirror (`mirror_tag`): - -```json -{ - "adaptive-rejection-sampler": { - "source": "alexgshaw/adaptive-rejection-sampler:20251031", - "source_digest": "sha256:…", - "pinned": "ghcr.io/node-q/hbench-envs/terminal-bench@sha256:…", - "mirror_tag": "adaptive-rejection-sampler-20251031" - } -} -``` +## Development -For `pinned-upstream`, `pinned` is `alexgshaw/adaptive-rejection-sampler@sha256:…` and there is no -`mirror_tag`. A version is committed with `{}` as its lock; it validates with a warning, and the -publish workflow refuses it until the lock is filled. +### Repository layout -To mirror a dataset's images, a maintainer: +| Path | What it is | +|---|---| +| `datasets///source.yaml` | Where a dataset version's task directories come from (git URL, full commit and path, or the adapter that generates them and the upstream files it reads) and what the published dataset declares: licence, upstream and citation, canary string, environment policy, attribution. | +| `datasets///LICENSE`, `NOTICE` | The dataset's own licence and attribution, published with it. | +| `datasets///images.lock.json` | For a version that pins its environment images by digest: each task's upstream or built image, its digest, and what its `docker_image` is pinned to. | +| `datasets///generated.lock.json` | For a generated version: each task's hbench package digest. | +| `adapters/` | The adapters that generate task directories, and `adapters/common`, what they share. | +| `images/` | The environment images this repository builds (`images.yaml`) and the base images they start from (`bases.yaml`). | +| [`schema/source.v1.json`](schema/source.v1.json) | The JSON Schema for `source.yaml`. | +| `scripts/` | Descriptor validation, fetching a source at its pinned commit, the pushed-manifest check, the Croissant generator, a verified hbench install, and the image mirror and pin scripts. | +| [`docs/licensing.md`](docs/licensing.md) | Which datasets can be republished, and why. | +| [`docs/PUBLISHING.md`](docs/PUBLISHING.md) | For maintainers: mirroring, pinning and building images, and publishing a version. | -1. optionally adds the repository secrets `DOCKERHUB_USERNAME` and `DOCKERHUB_TOKEN`. Without them, - the copy pulls from Docker Hub anonymously, in shards on separate runners, and waits and retries - when it is rate limited; -2. runs **Mirror images** from `main` with the dataset and the version whose tasks declare the images - (`terminal-bench`, `2.0`). A mirror tag that already holds the source's digest is skipped and one - that holds another digest is never overwritten, so a failed run is finished by re-running it; -3. after the first run, makes the package public: a new GHCR package starts private, and only its - [settings page](https://github.com/orgs/node-q/packages/container/hbench-envs%2Fterminal-bench/settings) - changes that. Publishing a `pinned-mirror` version refuses to run until every mirrored image can - be pulled anonymously; -4. commits the run's `images.lock.json` artifact to the `pinned-mirror` version's directory - (`datasets/terminal-bench/2.0.1/`) in a pull request. CI checks it against every task. +No task files are committed. CI fetches each git source at its pinned commit, or generates a +generated source's tasks from its pinned upstream files, and packs them. -## Generated datasets +### Generated datasets A dataset version whose source kind is `generated` has task directories this repository writes itself. Its descriptor names an adapter, `adapters/`, and the upstream files the adapter @@ -177,97 +288,31 @@ package digest, as `hbench tasks digest` prints it. CI generates every generated fails if a digest differs from the lock, then runs each task's checker on its oracle and on a wrong solution. `fetch_source.py` generates instead of fetching, so publishing works as for any source. -A generated version's environment policy is `built`: every task runs in an image this repository -builds, a tag of the one GHCR package `ghcr.io/node-q/hbench-envs`. `images/bases.yaml` lists base -images, copied from Docker Hub once with their digests unchanged and tagged `base--` -(`base-python-3.12-slim`); `images/images.yaml` lists the built images, tagged `-` -(`humaneval-plus-1.0`), each with its build context and base. Every built image's Dockerfile starts -`FROM` its base's GHCR copy pinned by digest, so neither a build nor a run needs Docker Hub, and the -task's `environment/Dockerfile`, the same file, is what `--force-build` builds. Versions whose tasks -need the same environment share an image: `humanevalfix@1.0`, `mbpp-plus@1.0` and -`cruxeval-input@1.0` run in `humaneval-plus-1.0`, and `generated_source.py lock-image` writes the -locks of every version built on a tag at once. - -To build an image, a maintainer: - -1. runs **Build images** from `main` with its tag (`humaneval-plus-1.0`). The workflow copies any base - not on GHCR yet, builds the image for its platforms, pushes it, refusing an existing tag, and - signs it with cosign keyless; -2. the first time, makes [`hbench-envs`](https://github.com/orgs/node-q/packages/container/hbench-envs/settings) - public: a new GHCR package starts private, and it is one package for every built image and base; -3. commits the run's `locks-` artifact, the `images.lock.json` and `generated.lock.json` of every - version built on the image, in a pull request. - -```sh -cosign verify ghcr.io/node-q/hbench-envs@sha256: \ - --certificate-identity-regexp '^https://github.com/node-q/hbench-datasets/.github/workflows/build-images.yml@' \ - --certificate-oidc-issuer https://token.actions.githubusercontent.com -``` - -Until then `images.lock.json` is `{}`: the version validates with a warning, its tasks name no -`docker_image`, and the publish workflow refuses it. - -## Offline use +### Verification -hbench's `export` command, which is not in a release yet, writes an OCI image layout to a -directory, and hbench reads it with `--registry oci-layout:`: - -```sh -hbench datasets export --oci-layout ./hbench-layout terminal-bench@2.0 --registry oci://ghcr.io/node-q/hbench-datasets -hbench run -d terminal-bench@2.0 --registry oci-layout:./hbench-layout +```bash +python -m pip install --require-hashes -r requirements.txt +python scripts/validate_sources.py # every source.yaml, its LICENSE, NOTICE and SPDX id +python scripts/environment_images.py check # the environment images and their bases +python -m unittest discover --start-directory tests --verbose ``` -`oras copy -r --to-oci-layout ghcr.io/node-q/hbench-datasets/terminal-bench:2.0 ./hbench-layout:2.0` -writes the same layout today. The tasks' environment images are not part of the layout: pull or -load them before going offline. +CI (`.github/workflows/validate.yml`) runs these on GitHub-hosted runners with Python 3.12, fetches +every dataset version's task directories at their pinned commits, and generates each generated +version twice and checks every task. -## Licensing +## License -The tooling in this repository is Apache-2.0 ([`LICENSE`](LICENSE)). Each dataset keeps its own -licence, in its directory and published with it. +The tooling in this repository is Apache-2.0. See [LICENSE](LICENSE). Each dataset keeps its own +licence (see [Licensing](#licensing)). -- **terminal-bench@2.0** is Apache-2.0 and configured. -- **terminal-bench@2.0.1** is the same tasks, Apache-2.0. Its environment images are redistributed - unmodified from upstream's published images; the software in them carries its own licences. -- **humaneval-plus@1.0** is generated by node-q from HumanEval+ (EvalPlus, Apache-2.0), which - extends OpenAI's HumanEval (MIT). Its task directories are Apache-2.0, with both notices in its - NOTICE. -- **humanevalfix@1.0** (from BigCode's HumanEvalPack, MIT), **mbpp-plus@1.0** (from EvalPlus's - MBPP+, Apache-2.0, derived from Google's MBPP, CC BY 4.0) and **cruxeval-input@1.0** (from Meta's - CRUXEval, MIT) are generated by node-q. Their task directories are Apache-2.0, with every upstream - notice in their NOTICE. -- **swebench-verified@1.0 and aider-polyglot@1.0** have MIT data, but their task directories come - from a third-party shared task repository with no LICENSE file. They need that repository's - permission, or task directories of our own, which would be new versions. -- **livecodebench@6.0** has an unspecified "cc" licence and problems scraped from contest sites. It - is not republished. -- **SWE-bench's prebuilt images** embed GPL-2.0 code among other licences. They are pinned upstream - by digest, not mirrored. - -The details are in [`docs/licensing.md`](docs/licensing.md). +## Contributors and contact -## Publishing - -A maintainer runs the **Publish** workflow from `main` with the dataset, the version and the -node-q/hbench release to pack with. It runs on GitHub-hosted runners and: - -1. refuses to run unless that hbench release has `hbench datasets push`, installing it only after - checking `SHA256SUMS` and its build provenance; -2. refuses to overwrite a tag that already exists; -3. fetches the source at its pinned commit, or generates it, and checks its task count and images - against the descriptor; for a version that pins its images, refuses an `images.lock.json` that is - not filled yet, or mirrored or built images that cannot be pulled anonymously, and pins each - task's `docker_image` from the lock; -4. packs and pushes with `hbench datasets push`, then checks the pushed manifest; -5. signs the manifest digest with cosign, attests it with `actions/attest`, and attaches the - Croissant file; -6. writes the digest to the job summary; -7. once an anonymous pull of the new digest works, creates the GitHub release - `-` (for example `terminal-bench-2.0`): the reference and digest, how to run - and verify it, and its `croissant.json`, `source.yaml`, `LICENSE` and `NOTICE` attached. - -A new GHCR package starts private. After the first publish of a dataset, make its package public -in the package's settings, then re-run the release job. +hbench-datasets is maintained by [node-q](https://github.com/node-q). See the +[contributors graph](https://github.com/node-q/hbench-datasets/graphs/contributors) for everyone who +has contributed. +Report bugs and request datasets in [GitHub issues](https://github.com/node-q/hbench-datasets/issues). To propose a dataset, see [CONTRIBUTING.md](CONTRIBUTING.md). To report a problem with a published -dataset, see [SECURITY.md](SECURITY.md). +dataset, such as a signature that does not verify or a tag that moved, use private reporting as +[SECURITY.md](SECURITY.md) describes. diff --git a/docs/PUBLISHING.md b/docs/PUBLISHING.md new file mode 100644 index 0000000..978270e --- /dev/null +++ b/docs/PUBLISHING.md @@ -0,0 +1,116 @@ +# Publishing + +For maintainers: how a dataset version's environment images are mirrored, pinned or built, and how +the version is published. Every step runs in this repository's workflows, from `main`, on +GitHub-hosted runners. Contributors propose versions as [CONTRIBUTING.md](../CONTRIBUTING.md) +describes. + +- [Mirroring images](#mirroring-images) +- [Pinning upstream images](#pinning-upstream-images) +- [Building images](#building-images) +- [images.lock.json](#imageslockjson) +- [Publishing a version](#publishing-a-version) + +## Mirroring images + +A version whose environment policy is `pinned-mirror` runs its tasks' prebuilt images from node-q's +mirror. The **Mirror images** workflow copies each image, registry to registry, unmodified and with +its digest unchanged, to one GHCR package per dataset, `ghcr.io/node-q/hbench-envs/`, tagged +`-` (for example +`ghcr.io/node-q/hbench-envs/terminal-bench:adaptive-rejection-sampler-20251031`). It signs each digest +with cosign keyless and writes `images.lock.json`. Each task's `task.toml` then names its image by +digest, `ghcr.io/node-q/hbench-envs/terminal-bench@sha256:…`, and nothing else in it changes. + +To mirror a dataset's images, a maintainer: + +1. optionally adds the repository secrets `DOCKERHUB_USERNAME` and `DOCKERHUB_TOKEN`. Without them, + the copy pulls from Docker Hub anonymously, in shards on separate runners, and waits and retries + when it is rate limited; +2. runs **Mirror images** from `main` with the dataset and the version whose tasks declare the images + (`terminal-bench`, `2.0`). A mirror tag that already holds the source's digest is skipped and one + that holds another digest is never overwritten, so a failed run is finished by re-running it; +3. after the first run, makes the package public: a new GHCR package starts private, and only its + [settings page](https://github.com/orgs/node-q/packages/container/hbench-envs%2Fterminal-bench/settings) + changes that. Publishing a `pinned-mirror` version refuses to run until every mirrored image can + be pulled anonymously; +4. commits the run's `images.lock.json` artifact to the `pinned-mirror` version's directory + (`datasets/terminal-bench/2.0.1/`) in a pull request. CI checks it against every task. + +## Pinning upstream images + +A version whose policy is `pinned-upstream` pins upstream's images by digest without copying them, +for images that are not ours to redistribute or are too large to mirror. The **Pin images** workflow +resolves each tag with a registry HEAD request, which Docker Hub does not count as a pull, and writes +`images.lock.json`. Commit it to the version's directory in a pull request, as for a mirror. + +## Building images + +A generated version's environment policy is `built`: every task runs in an image this repository +builds, a tag of the one GHCR package `ghcr.io/node-q/hbench-envs`. `images/bases.yaml` lists base +images, copied from Docker Hub once with their digests unchanged and tagged `base--` +(`base-python-3.12-slim`); `images/images.yaml` lists the built images, tagged `-` +(`humaneval-plus-1.0`), each with its build context and base. Every built image's Dockerfile starts +`FROM` its base's GHCR copy pinned by digest, so neither a build nor a run needs Docker Hub, and the +task's `environment/Dockerfile`, the same file, is what `--force-build` builds. Versions whose tasks +need the same environment share an image: `humanevalfix@1.0`, `mbpp-plus@1.0` and +`cruxeval-input@1.0` run in `humaneval-plus-1.0`, and `generated_source.py lock-image` writes the +locks of every version built on a tag at once. + +To build an image, a maintainer: + +1. runs **Build images** from `main` with its tag (`humaneval-plus-1.0`). The workflow copies any base + not on GHCR yet, builds the image for its platforms, pushes it, refusing an existing tag, and + signs it with cosign keyless; +2. the first time, makes [`hbench-envs`](https://github.com/orgs/node-q/packages/container/hbench-envs/settings) + public: a new GHCR package starts private, and it is one package for every built image and base; +3. commits the run's `locks-` artifact, the `images.lock.json` and `generated.lock.json` of every + version built on the image, in a pull request. + +Until then `images.lock.json` is `{}`: the version validates with a warning, its tasks name no +`docker_image`, and the publish workflow refuses it. + +## images.lock.json + +`images.lock.json` maps each task, sorted, to the image upstream declares or the tag this repository +built (`source`), its digest (`source_digest`) and what `task.toml`'s `docker_image` becomes +(`pinned`); a mirrored image also has its tag in the mirror (`mirror_tag`): + +```json +{ + "adaptive-rejection-sampler": { + "source": "alexgshaw/adaptive-rejection-sampler:20251031", + "source_digest": "sha256:…", + "pinned": "ghcr.io/node-q/hbench-envs/terminal-bench@sha256:…", + "mirror_tag": "adaptive-rejection-sampler-20251031" + } +} +``` + +For `pinned-upstream`, `pinned` is `alexgshaw/adaptive-rejection-sampler@sha256:…` and there is no +`mirror_tag`. A version is committed with `{}` as its lock; it validates with a warning, and the +publish workflow refuses it until the lock is filled. + +## Publishing a version + +A maintainer runs the **Publish** workflow from `main` with the dataset, the version and the +node-q/hbench release to pack with. It: + +1. refuses to run unless that hbench release has `hbench datasets push`, installing it only after + checking `SHA256SUMS` and its build provenance; +2. refuses to overwrite a tag that already exists; +3. fetches the source at its pinned commit, or generates it, and checks its task count and images + against the descriptor; for a version that pins its images, refuses an `images.lock.json` that is + not filled yet, or mirrored or built images that cannot be pulled anonymously, and pins each + task's `docker_image` from the lock; +4. packs and pushes with `hbench datasets push`, then checks the pushed manifest; +5. signs the manifest digest with cosign, attests it with `actions/attest`, and attaches the + Croissant file; +6. writes the digest to the job summary; +7. once an anonymous pull of the new digest works, creates the GitHub release + `-` (for example `terminal-bench-2.0`): the reference and digest, how to run + and verify it, and its `croissant.json`, `source.yaml`, `LICENSE` and `NOTICE` attached. + +A new GHCR package starts private. After the first publish of a dataset, make its package public in +the package's settings +(`https://github.com/orgs/node-q/packages/container/hbench-datasets%2F/settings`), then +re-run the release job.