From fff59a7b9e467e83ace11204b15f95da51c967b5 Mon Sep 17 00:00:00 2001 From: rzonyx <77541804+rzonyx@users.noreply.github.com> Date: Thu, 22 Oct 2026 11:15:33 +0700 Subject: [PATCH] README in hbench's sections; maintainer steps move to docs/PUBLISHING.md The README follows hbench's layout: a logo, a table of contents, then Installation, Usage, Examples and Datasets, the reference sections (environments, verifying, offline use and mirrors, licensing), Development, License and Contributors and contact. It now says the six versions are published, needs hbench 0.3.0, and shows export as 0.3.0 takes it. Mirroring, pinning and building images, images.lock.json and the publish workflow are maintainer steps, so they move to docs/PUBLISHING.md, as hbench moved its release steps to docs/RELEASING.md. CONTRIBUTING points there. --- CONTRIBUTING.md | 5 +- README.md | 421 +++++++++++++++++++++++++-------------------- docs/PUBLISHING.md | 116 +++++++++++++ 3 files changed, 352 insertions(+), 190 deletions(-) create mode 100644 docs/PUBLISHING.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 379b205..1d184b4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -90,8 +90,9 @@ version is a generated one; see [`datasets/humaneval-plus/1.0`](datasets/humanev it, so a change to the adapter, its shared helpers or the image's build context is committed with the lock it produces. 5. **After the pull request is merged**, a maintainer runs the Build images workflow and commits the - `images.lock.json` and `generated.lock.json` it writes, as the README describes. Only then can the - version be published. + `images.lock.json` and `generated.lock.json` it writes, as + [docs/PUBLISHING.md](docs/PUBLISHING.md#building-images) describes. Only then can the version be + published. ## Versions are immutable diff --git a/README.md b/README.md index c402df5..6072b21 100644 --- a/README.md +++ b/README.md @@ -1,67 +1,160 @@ +```text +██╗ ██╗██████╗ ███████╗███╗ ██╗ ██████╗██╗ ██╗ +██║ ██║██╔══██╗██╔════╝████╗ ██║██╔════╝██║ ██║ +███████║██████╔╝█████╗ ██╔██╗ ██║██║ ███████║ +██╔══██║██╔══██╗██╔══╝ ██║╚██╗██║██║ ██╔══██║ +██║ ██║██████╔╝███████╗██║ ╚████║╚██████╗██║ ██║ +╚═╝ ╚═╝╚═════╝ ╚══════╝╚═╝ ╚═══╝ ╚═════╝╚═╝ ╚═╝ + +██████╗ █████╗ ████████╗ █████╗ ███████╗███████╗████████╗███████╗ +██╔══██╗██╔══██╗╚══██╔══╝██╔══██╗██╔════╝██╔════╝╚══██╔══╝██╔════╝ +██║ ██║███████║ ██║ ███████║███████╗█████╗ ██║ ███████╗ +██║ ██║██╔══██║ ██║ ██╔══██║╚════██║██╔══╝ ██║ ╚════██║ +██████╔╝██║ ██║ ██║ ██║ ██║███████║███████╗ ██║ ███████║ +╚═════╝ ╚═╝ ╚═╝ ╚═╝ ╚═╝ ╚═╝╚══════╝╚══════╝ ╚═╝ ╚══════╝ +``` + # hbench-datasets -Benchmark datasets for [hbench](https://github.com/node-q/hbench) (`node-q/hbench`), the CLI that -runs AI agents on benchmark tasks in Docker, published as signed OCI artifacts. +Benchmark datasets for [hbench](https://github.com/node-q/hbench), published as signed OCI artifacts +at `ghcr.io/node-q/hbench-datasets`. -**Nothing is published yet.** The publish workflow is ready, but it refuses to run until a -node-q/hbench release has `hbench datasets push`. +Each dataset version is one OCI image manifest, tagged with the version, with one reproducible layer +per task. hbench pulls it with `--registry oci://ghcr.io/node-q/hbench-datasets` and checks every layer +against the manifest's digest before it unpacks a task. Every version is signed with cosign keyless, +carries SLSA build provenance and Croissant 1.1 metadata, and has a GitHub release with its digest. -## What's here +- [Installation](#installation) +- [Usage](#usage) +- [Examples](#examples) +- [Datasets](#datasets) +- [Environments](#environments) · [Verifying a dataset](#verifying-a-dataset) · [Offline use and mirrors](#offline-use-and-mirrors) · [Licensing](#licensing) +- [Development](#development) +- [License](#license) +- [Contributors and contact](#contributors-and-contact) -| Path | What it is | -| --- | --- | -| `datasets///source.yaml` | Where a dataset version's task directories come from (git URL, full commit and path, or the adapter that generates them and the upstream files it reads) and what the published dataset declares: licence, upstream and citation, canary string, environment policy, attribution. | -| `datasets///LICENSE`, `NOTICE` | The dataset's own licence and attribution, published with it. | -| `datasets///images.lock.json` | For a version that pins its environment images by digest: each task's upstream or built image, its digest, and what its `docker_image` is pinned to. | -| `datasets///generated.lock.json` | For a generated version: each task's hbench package digest. | -| `adapters/` | The adapters that generate task directories, and `adapters/common`, what they share. | -| `images/` | The environment images this repository builds (`images.yaml`) and the base images they start from (`bases.yaml`). | -| [`schema/source.v1.json`](schema/source.v1.json) | The JSON Schema for `source.yaml`. | -| `scripts/` | Descriptor validation, fetching a source at its pinned commit, the pushed-manifest check, the Croissant generator, a verified hbench install, and the image mirror and pin scripts. | -| [`docs/licensing.md`](docs/licensing.md) | Which datasets can be republished, and why. | +## Installation -No task files are committed. CI fetches each git source at its pinned commit, or generates a -generated source's tasks from its pinned upstream files, and packs them. +Requirements: hbench 0.3.0 or later, and a Docker daemon. Nothing from this repository needs to be +installed: hbench pulls a dataset when a run names it. -Each dataset version is one OCI image manifest at -`ghcr.io/node-q/hbench-datasets/:`: +Install the latest hbench release: -- `artifactType: application/vnd.hbench.dataset.v1`; -- a config blob, `application/vnd.hbench.dataset.config.v1+json`; -- one reproducible `tar+gzip` layer per task, which unpacks byte-identical to the upstream task - directory. +```bash +curl -fsSL https://raw.githubusercontent.com/node-q/hbench/master/scripts/install.sh | sh +``` -It is signed with cosign keyless (GitHub OIDC), carries SLSA build provenance from GitHub artifact -attestations, and has a Croissant 1.1 JSON-LD file attached for discovery. Tags are never moved: a -new version is a new tag. Each dataset is also mirrored to a second registry, not chosen yet. +`HBENCH_VERSION=vX.Y.Z` pins a release and `HBENCH_INSTALL_DIR` overrides `~/.local/bin` (see +[hbench's README](https://github.com/node-q/hbench)). To check signatures and +provenance, or to copy a dataset without hbench, you also need [cosign](https://github.com/sigstore/cosign), +the [GitHub CLI](https://cli.github.com) or [oras](https://oras.land). -| Dataset | Tasks | Licence | Status | -| --- | ---: | --- | --- | -| `terminal-bench@2.0` | 89 | Apache-2.0 | Configured, not published yet | -| `terminal-bench@2.0.1` | 89 | Apache-2.0 | Configured | -| `humaneval-plus@1.0` | 164 | Apache-2.0 | Generated; waits for its image to be built | -| `humanevalfix@1.0` | 164 | Apache-2.0 | Generated; waits for its image lock | -| `mbpp-plus@1.0` | 378 | Apache-2.0 | Generated; waits for its image lock | -| `cruxeval-input@1.0` | 800 | Apache-2.0 | Generated; waits for its image lock | +## Usage -## Running a dataset +```bash +# Run +hbench run -d terminal-bench@2.0.1 --registry oci://ghcr.io/node-q/hbench-datasets -a oracle +hbench run -d humaneval-plus@1.0 --registry oci://ghcr.io/node-q/hbench-datasets --lock-write # record the digest in ./hbench.lock +hbench run -d humaneval-plus@1.0 --registry oci://ghcr.io/node-q/hbench-datasets --locked # refuse any other digest -This needs an hbench release with the OCI read path (`--registry oci://…`), which is being built. +# Download and copy +hbench datasets download humaneval-plus@1.0 --registry oci://ghcr.io/node-q/hbench-datasets -o ./cache +hbench datasets export -d terminal-bench@2.0.1 --registry oci://ghcr.io/node-q/hbench-datasets --oci-layout ./hbench-layout +hbench datasets list --registry oci-layout:./hbench-layout +hbench run -d terminal-bench@2.0.1 --registry oci-layout:./hbench-layout -```sh -hbench run -d terminal-bench@2.0 --registry oci://ghcr.io/node-q/hbench-datasets ``` -hbench resolves `terminal-bench:2.0` to its manifest digest and checks every layer against it -before unpacking a task. In `terminal-bench@2.0` the tasks pull their prebuilt environment images -from Docker Hub by tag (`alexgshaw/:20251031`). `terminal-bench@2.0.1` is the same tasks with -each image pinned by digest to its mirror (see [Mirrored environments](#mirrored-environments)). +`--registry oci://ghcr.io/node-q/hbench-datasets` names the namespace, and `-d @` +picks the repository `` in it and the tag ``. A registry can't be listed, so a run +names its dataset; an OCI layout on disk can be listed. In a job config, the dataset sets +`registry: "oci://ghcr.io/node-q/hbench-datasets"` with its `name` and `version`, and may set +`digest: "sha256:..."` to run that manifest whatever the tag says. + +`--lock-write` records each dataset's manifest digest in `./hbench.lock`, keyed by `name@version`. A +dataset in `hbench.lock` runs at its locked digest, and `--locked` refuses one that is not in it or +whose tag now resolves to another digest. To check a dataset's signature and provenance, see +[Verifying a dataset](#verifying-a-dataset). + +## Examples + +Check the setup with the oracle agent, which runs each task's reference solution, on five tasks: + +```bash +hbench run -d humaneval-plus@1.0 --registry oci://ghcr.io/node-q/hbench-datasets -a oracle -l 5 +``` + +Run Claude Code on Terminal-Bench 2.0, with every task's image pinned by digest: + +```bash +ANTHROPIC_API_KEY=... hbench run -d terminal-bench@2.0.1 --registry oci://ghcr.io/node-q/hbench-datasets \ + -a claude-code -m anthropic/claude-opus-5-5 -n 4 +``` + +Keep a series of runs on the same bytes: record the digests once, commit `hbench.lock`, and run with +`--locked` from then on: + +```bash +hbench run -d mbpp-plus@1.0 --registry oci://ghcr.io/node-q/hbench-datasets -a oracle -l 1 --lock-write +hbench run -d mbpp-plus@1.0 --registry oci://ghcr.io/node-q/hbench-datasets -a codex -m openai/gpt-5 --locked +``` + +Browse the results in hbench's web viewer: + +```bash +hbench view jobs +``` + +## Datasets + +| Dataset | Tasks | What it tests | Licence | +|---|---:|---|---| +| `terminal-bench@2.0` | 89 | Work in a terminal: build, debug, configure, train. Images from Docker Hub, by the tags upstream declares | Apache-2.0 | +| `terminal-bench@2.0.1` | 89 | The same tasks, with each image pinned by digest to node-q's mirror on GHCR | Apache-2.0 | +| `humaneval-plus@1.0` | 164 | Write a Python function from its signature and docstring: HumanEval with EvalPlus's extended tests | Apache-2.0 | +| `humanevalfix@1.0` | 164 | Fix a bug written by hand into a HumanEval function (BigCode's HumanEvalPack) | Apache-2.0 | +| `mbpp-plus@1.0` | 378 | Write a Python function from a description and an example: MBPP with EvalPlus's extended tests | Apache-2.0 | +| `cruxeval-input@1.0` | 800 | Given a Python function and an output, find an input that gives it (CRUXEval-I) | Apache-2.0 | + +Each version has a [release](https://github.com/node-q/hbench-datasets/releases) named +`-`, such as +[`terminal-bench-2.0.1`](https://github.com/node-q/hbench-datasets/releases/tag/terminal-bench-2.0.1): +its reference and manifest digest, task count, licence and upstream commit, how to run and check it, +and its `croissant.json`, `source.yaml`, `LICENSE` and `NOTICE`. A published tag never moves; a fix is +a new version. Each dataset will also be mirrored to a second registry, not chosen yet. + +A version is one OCI image manifest at `ghcr.io/node-q/hbench-datasets/:`: + +- `artifactType: application/vnd.hbench.dataset.v1`; +- a config blob, `application/vnd.hbench.dataset.config.v1+json`, with the dataset's licence, + upstream, canary and each task's digests; +- one reproducible `tar+gzip` layer per task, which unpacks byte-identical to the task directory + it was packed from. + +hbench's README describes the [format](https://github.com/node-q/hbench#dataset-format) in full. + +## Environments + +A task runs in the Docker image its `task.toml` names as `docker_image`. With none, or with +`--force-build`, hbench builds the task's `environment/Dockerfile` instead. A version's environment +policy says where its images come from: + +| Policy | Images | Versions | +|---|---|---| +| `as-declared` | Upstream's images, by the tags upstream declares. Upstream can move a tag | `terminal-bench@2.0` | +| `pinned-mirror` | Upstream's images, copied unmodified with their digests to `ghcr.io/node-q/hbench-envs/`, and named by digest | `terminal-bench@2.0.1` | +| `pinned-upstream` | Upstream's images, named by digest and not copied: for images that are not ours to redistribute or are too large to mirror | none yet | +| `built` | Images this repository builds, tags of `ghcr.io/node-q/hbench-envs`, named by digest. Each starts `FROM` a base image copied to GHCR, so neither a build nor a run needs Docker Hub | `humaneval-plus@1.0`, `humanevalfix@1.0`, `mbpp-plus@1.0` and `cruxeval-input@1.0`, which share `humaneval-plus-1.0` | + +A version that pins its images has an `images.lock.json` next to its `source.yaml`, listing each +task's image and digest. [docs/PUBLISHING.md](docs/PUBLISHING.md) describes how the images are +mirrored, pinned and built. ## Verifying a dataset The signature, made by this repository's publish workflow: -```sh +```bash cosign verify ghcr.io/node-q/hbench-datasets/terminal-bench:2.0 \ --certificate-identity-regexp '^https://github.com/node-q/hbench-datasets/.github/workflows/publish.yml@' \ --certificate-oidc-issuer https://token.actions.githubusercontent.com @@ -69,87 +162,105 @@ cosign verify ghcr.io/node-q/hbench-datasets/terminal-bench:2.0 \ The build provenance: -```sh +```bash gh attestation verify oci://ghcr.io/node-q/hbench-datasets/terminal-bench:2.0 --repo node-q/hbench-datasets ``` -The Croissant metadata is a referrer of the dataset's manifest: +The Croissant metadata, a referrer of the dataset's manifest: -```sh +```bash oras discover --artifact-type application/ld+json ghcr.io/node-q/hbench-datasets/terminal-bench:2.0 ``` -## Mirroring +A mirrored image, with its digest from the version's `images.lock.json`, is signed by the mirror +workflow. Its digest is upstream's own: `crane digest alexgshaw/:20251031` prints the same one +for as long as upstream keeps the tag where it is. + +```bash +cosign verify ghcr.io/node-q/hbench-envs/terminal-bench@sha256: \ + --certificate-identity-regexp '^https://github.com/node-q/hbench-datasets/.github/workflows/mirror-images.yml@' \ + --certificate-oidc-issuer https://token.actions.githubusercontent.com +``` + +A built image is signed by the build workflow: -```sh +```bash +cosign verify ghcr.io/node-q/hbench-envs@sha256: \ + --certificate-identity-regexp '^https://github.com/node-q/hbench-datasets/.github/workflows/build-images.yml@' \ + --certificate-oidc-issuer https://token.actions.githubusercontent.com +``` + +## Offline use and mirrors + +`hbench datasets export` copies a dataset into an OCI image layout, `/`, and hbench reads +it with `--registry oci-layout:`: + +```bash +hbench datasets export -d terminal-bench@2.0 --registry oci://ghcr.io/node-q/hbench-datasets --oci-layout ./hbench-layout +hbench run -d terminal-bench@2.0 --registry oci-layout:./hbench-layout +``` + +`oras cp ghcr.io/node-q/hbench-datasets/terminal-bench:2.0 --to-oci-layout ./hbench-layout/terminal-bench:2.0` +writes the same layout without hbench. The tasks' environment images are not part of the layout: +pull or load them before going offline. + +To mirror a dataset to another registry: + +```bash oras copy -r ghcr.io/node-q/hbench-datasets/terminal-bench:2.0 //terminal-bench:2.0 ``` `-r` copies the referrers too: the signature, the attestation and the Croissant metadata. Digests do not change, so the mirror verifies with the same commands, and hbench reads it with -`--registry oci:///`. +`--registry oci:///`. `hbench.lock` is keyed by `name@version`, not by registry, +so a locked run accepts the mirror only because it serves the same digest. -## Mirrored environments +## Licensing -A dataset version whose environment policy is `pinned-mirror` runs its tasks' prebuilt images from -node-q's mirror. The **Mirror images** workflow copies each image, registry to registry, unmodified -and with its digest unchanged, to one GHCR package per dataset, `ghcr.io/node-q/hbench-envs/`, -tagged `-` (for example -`ghcr.io/node-q/hbench-envs/terminal-bench:adaptive-rejection-sampler-20251031`). It signs each digest -with cosign keyless and writes `images.lock.json`. Each task's `task.toml` then names its image by -digest, `ghcr.io/node-q/hbench-envs/terminal-bench@sha256:…`, and nothing else in it changes. +The tooling in this repository is Apache-2.0 ([`LICENSE`](LICENSE)). Each dataset keeps its own +licence, in its directory and published with it. -To verify a mirrored image, take its digest from the version's `images.lock.json`: +- **terminal-bench@2.0** is Apache-2.0. +- **terminal-bench@2.0.1** is the same tasks, Apache-2.0. Its environment images are redistributed + unmodified from upstream's published images; the software in them carries its own licences. +- **humaneval-plus@1.0** is generated by node-q from HumanEval+ (EvalPlus, Apache-2.0), which + extends OpenAI's HumanEval (MIT). Its task directories are Apache-2.0, with both notices in its + NOTICE. +- **humanevalfix@1.0** (from BigCode's HumanEvalPack, MIT), **mbpp-plus@1.0** (from EvalPlus's + MBPP+, Apache-2.0, derived from Google's MBPP, CC BY 4.0) and **cruxeval-input@1.0** (from Meta's + CRUXEval, MIT) are generated by node-q. Their task directories are Apache-2.0, with every upstream + notice in their NOTICE. +- **swebench-verified@1.0 and aider-polyglot@1.0** have MIT data, but their task directories come + from a third-party shared task repository with no LICENSE file. They need that repository's + permission, or task directories of our own, which would be new versions. +- **livecodebench@6.0** has an unspecified "cc" licence and problems scraped from contest sites. It + is not republished. +- **SWE-bench's prebuilt images** embed GPL-2.0 code among other licences. They are pinned upstream + by digest, not mirrored. -```sh -cosign verify ghcr.io/node-q/hbench-envs/terminal-bench@sha256: \ - --certificate-identity-regexp '^https://github.com/node-q/hbench-datasets/.github/workflows/mirror-images.yml@' \ - --certificate-oidc-issuer https://token.actions.githubusercontent.com -``` +The details are in [`docs/licensing.md`](docs/licensing.md). -The digest is upstream's own: `crane digest alexgshaw/:20251031` prints the same one for as -long as upstream keeps the tag where it is. - -A version whose policy is `pinned-upstream` pins upstream's images by digest without copying them, -for images that are not ours to redistribute or are too large to mirror. The **Pin images** workflow -resolves each tag with a registry HEAD request, which Docker Hub does not count as a pull, and writes -`images.lock.json`. - -`images.lock.json` maps each task, sorted, to the image upstream declares (`source`), its digest -(`source_digest`) and what `task.toml`'s `docker_image` becomes (`pinned`); a mirrored image also has -its tag in the mirror (`mirror_tag`): - -```json -{ - "adaptive-rejection-sampler": { - "source": "alexgshaw/adaptive-rejection-sampler:20251031", - "source_digest": "sha256:…", - "pinned": "ghcr.io/node-q/hbench-envs/terminal-bench@sha256:…", - "mirror_tag": "adaptive-rejection-sampler-20251031" - } -} -``` +## Development -For `pinned-upstream`, `pinned` is `alexgshaw/adaptive-rejection-sampler@sha256:…` and there is no -`mirror_tag`. A version is committed with `{}` as its lock; it validates with a warning, and the -publish workflow refuses it until the lock is filled. +### Repository layout -To mirror a dataset's images, a maintainer: +| Path | What it is | +|---|---| +| `datasets///source.yaml` | Where a dataset version's task directories come from (git URL, full commit and path, or the adapter that generates them and the upstream files it reads) and what the published dataset declares: licence, upstream and citation, canary string, environment policy, attribution. | +| `datasets///LICENSE`, `NOTICE` | The dataset's own licence and attribution, published with it. | +| `datasets///images.lock.json` | For a version that pins its environment images by digest: each task's upstream or built image, its digest, and what its `docker_image` is pinned to. | +| `datasets///generated.lock.json` | For a generated version: each task's hbench package digest. | +| `adapters/` | The adapters that generate task directories, and `adapters/common`, what they share. | +| `images/` | The environment images this repository builds (`images.yaml`) and the base images they start from (`bases.yaml`). | +| [`schema/source.v1.json`](schema/source.v1.json) | The JSON Schema for `source.yaml`. | +| `scripts/` | Descriptor validation, fetching a source at its pinned commit, the pushed-manifest check, the Croissant generator, a verified hbench install, and the image mirror and pin scripts. | +| [`docs/licensing.md`](docs/licensing.md) | Which datasets can be republished, and why. | +| [`docs/PUBLISHING.md`](docs/PUBLISHING.md) | For maintainers: mirroring, pinning and building images, and publishing a version. | -1. optionally adds the repository secrets `DOCKERHUB_USERNAME` and `DOCKERHUB_TOKEN`. Without them, - the copy pulls from Docker Hub anonymously, in shards on separate runners, and waits and retries - when it is rate limited; -2. runs **Mirror images** from `main` with the dataset and the version whose tasks declare the images - (`terminal-bench`, `2.0`). A mirror tag that already holds the source's digest is skipped and one - that holds another digest is never overwritten, so a failed run is finished by re-running it; -3. after the first run, makes the package public: a new GHCR package starts private, and only its - [settings page](https://github.com/orgs/node-q/packages/container/hbench-envs%2Fterminal-bench/settings) - changes that. Publishing a `pinned-mirror` version refuses to run until every mirrored image can - be pulled anonymously; -4. commits the run's `images.lock.json` artifact to the `pinned-mirror` version's directory - (`datasets/terminal-bench/2.0.1/`) in a pull request. CI checks it against every task. +No task files are committed. CI fetches each git source at its pinned commit, or generates a +generated source's tasks from its pinned upstream files, and packs them. -## Generated datasets +### Generated datasets A dataset version whose source kind is `generated` has task directories this repository writes itself. Its descriptor names an adapter, `adapters/`, and the upstream files the adapter @@ -177,97 +288,31 @@ package digest, as `hbench tasks digest` prints it. CI generates every generated fails if a digest differs from the lock, then runs each task's checker on its oracle and on a wrong solution. `fetch_source.py` generates instead of fetching, so publishing works as for any source. -A generated version's environment policy is `built`: every task runs in an image this repository -builds, a tag of the one GHCR package `ghcr.io/node-q/hbench-envs`. `images/bases.yaml` lists base -images, copied from Docker Hub once with their digests unchanged and tagged `base--` -(`base-python-3.12-slim`); `images/images.yaml` lists the built images, tagged `-` -(`humaneval-plus-1.0`), each with its build context and base. Every built image's Dockerfile starts -`FROM` its base's GHCR copy pinned by digest, so neither a build nor a run needs Docker Hub, and the -task's `environment/Dockerfile`, the same file, is what `--force-build` builds. Versions whose tasks -need the same environment share an image: `humanevalfix@1.0`, `mbpp-plus@1.0` and -`cruxeval-input@1.0` run in `humaneval-plus-1.0`, and `generated_source.py lock-image` writes the -locks of every version built on a tag at once. - -To build an image, a maintainer: - -1. runs **Build images** from `main` with its tag (`humaneval-plus-1.0`). The workflow copies any base - not on GHCR yet, builds the image for its platforms, pushes it, refusing an existing tag, and - signs it with cosign keyless; -2. the first time, makes [`hbench-envs`](https://github.com/orgs/node-q/packages/container/hbench-envs/settings) - public: a new GHCR package starts private, and it is one package for every built image and base; -3. commits the run's `locks-` artifact, the `images.lock.json` and `generated.lock.json` of every - version built on the image, in a pull request. - -```sh -cosign verify ghcr.io/node-q/hbench-envs@sha256: \ - --certificate-identity-regexp '^https://github.com/node-q/hbench-datasets/.github/workflows/build-images.yml@' \ - --certificate-oidc-issuer https://token.actions.githubusercontent.com -``` - -Until then `images.lock.json` is `{}`: the version validates with a warning, its tasks name no -`docker_image`, and the publish workflow refuses it. - -## Offline use +### Verification -hbench's `export` command, which is not in a release yet, writes an OCI image layout to a -directory, and hbench reads it with `--registry oci-layout:`: - -```sh -hbench datasets export --oci-layout ./hbench-layout terminal-bench@2.0 --registry oci://ghcr.io/node-q/hbench-datasets -hbench run -d terminal-bench@2.0 --registry oci-layout:./hbench-layout +```bash +python -m pip install --require-hashes -r requirements.txt +python scripts/validate_sources.py # every source.yaml, its LICENSE, NOTICE and SPDX id +python scripts/environment_images.py check # the environment images and their bases +python -m unittest discover --start-directory tests --verbose ``` -`oras copy -r --to-oci-layout ghcr.io/node-q/hbench-datasets/terminal-bench:2.0 ./hbench-layout:2.0` -writes the same layout today. The tasks' environment images are not part of the layout: pull or -load them before going offline. +CI (`.github/workflows/validate.yml`) runs these on GitHub-hosted runners with Python 3.12, fetches +every dataset version's task directories at their pinned commits, and generates each generated +version twice and checks every task. -## Licensing +## License -The tooling in this repository is Apache-2.0 ([`LICENSE`](LICENSE)). Each dataset keeps its own -licence, in its directory and published with it. +The tooling in this repository is Apache-2.0. See [LICENSE](LICENSE). Each dataset keeps its own +licence (see [Licensing](#licensing)). -- **terminal-bench@2.0** is Apache-2.0 and configured. -- **terminal-bench@2.0.1** is the same tasks, Apache-2.0. Its environment images are redistributed - unmodified from upstream's published images; the software in them carries its own licences. -- **humaneval-plus@1.0** is generated by node-q from HumanEval+ (EvalPlus, Apache-2.0), which - extends OpenAI's HumanEval (MIT). Its task directories are Apache-2.0, with both notices in its - NOTICE. -- **humanevalfix@1.0** (from BigCode's HumanEvalPack, MIT), **mbpp-plus@1.0** (from EvalPlus's - MBPP+, Apache-2.0, derived from Google's MBPP, CC BY 4.0) and **cruxeval-input@1.0** (from Meta's - CRUXEval, MIT) are generated by node-q. Their task directories are Apache-2.0, with every upstream - notice in their NOTICE. -- **swebench-verified@1.0 and aider-polyglot@1.0** have MIT data, but their task directories come - from a third-party shared task repository with no LICENSE file. They need that repository's - permission, or task directories of our own, which would be new versions. -- **livecodebench@6.0** has an unspecified "cc" licence and problems scraped from contest sites. It - is not republished. -- **SWE-bench's prebuilt images** embed GPL-2.0 code among other licences. They are pinned upstream - by digest, not mirrored. - -The details are in [`docs/licensing.md`](docs/licensing.md). +## Contributors and contact -## Publishing - -A maintainer runs the **Publish** workflow from `main` with the dataset, the version and the -node-q/hbench release to pack with. It runs on GitHub-hosted runners and: - -1. refuses to run unless that hbench release has `hbench datasets push`, installing it only after - checking `SHA256SUMS` and its build provenance; -2. refuses to overwrite a tag that already exists; -3. fetches the source at its pinned commit, or generates it, and checks its task count and images - against the descriptor; for a version that pins its images, refuses an `images.lock.json` that is - not filled yet, or mirrored or built images that cannot be pulled anonymously, and pins each - task's `docker_image` from the lock; -4. packs and pushes with `hbench datasets push`, then checks the pushed manifest; -5. signs the manifest digest with cosign, attests it with `actions/attest`, and attaches the - Croissant file; -6. writes the digest to the job summary; -7. once an anonymous pull of the new digest works, creates the GitHub release - `-` (for example `terminal-bench-2.0`): the reference and digest, how to run - and verify it, and its `croissant.json`, `source.yaml`, `LICENSE` and `NOTICE` attached. - -A new GHCR package starts private. After the first publish of a dataset, make its package public -in the package's settings, then re-run the release job. +hbench-datasets is maintained by [node-q](https://github.com/node-q). See the +[contributors graph](https://github.com/node-q/hbench-datasets/graphs/contributors) for everyone who +has contributed. +Report bugs and request datasets in [GitHub issues](https://github.com/node-q/hbench-datasets/issues). To propose a dataset, see [CONTRIBUTING.md](CONTRIBUTING.md). To report a problem with a published -dataset, see [SECURITY.md](SECURITY.md). +dataset, such as a signature that does not verify or a tag that moved, use private reporting as +[SECURITY.md](SECURITY.md) describes. diff --git a/docs/PUBLISHING.md b/docs/PUBLISHING.md new file mode 100644 index 0000000..978270e --- /dev/null +++ b/docs/PUBLISHING.md @@ -0,0 +1,116 @@ +# Publishing + +For maintainers: how a dataset version's environment images are mirrored, pinned or built, and how +the version is published. Every step runs in this repository's workflows, from `main`, on +GitHub-hosted runners. Contributors propose versions as [CONTRIBUTING.md](../CONTRIBUTING.md) +describes. + +- [Mirroring images](#mirroring-images) +- [Pinning upstream images](#pinning-upstream-images) +- [Building images](#building-images) +- [images.lock.json](#imageslockjson) +- [Publishing a version](#publishing-a-version) + +## Mirroring images + +A version whose environment policy is `pinned-mirror` runs its tasks' prebuilt images from node-q's +mirror. The **Mirror images** workflow copies each image, registry to registry, unmodified and with +its digest unchanged, to one GHCR package per dataset, `ghcr.io/node-q/hbench-envs/`, tagged +`-` (for example +`ghcr.io/node-q/hbench-envs/terminal-bench:adaptive-rejection-sampler-20251031`). It signs each digest +with cosign keyless and writes `images.lock.json`. Each task's `task.toml` then names its image by +digest, `ghcr.io/node-q/hbench-envs/terminal-bench@sha256:…`, and nothing else in it changes. + +To mirror a dataset's images, a maintainer: + +1. optionally adds the repository secrets `DOCKERHUB_USERNAME` and `DOCKERHUB_TOKEN`. Without them, + the copy pulls from Docker Hub anonymously, in shards on separate runners, and waits and retries + when it is rate limited; +2. runs **Mirror images** from `main` with the dataset and the version whose tasks declare the images + (`terminal-bench`, `2.0`). A mirror tag that already holds the source's digest is skipped and one + that holds another digest is never overwritten, so a failed run is finished by re-running it; +3. after the first run, makes the package public: a new GHCR package starts private, and only its + [settings page](https://github.com/orgs/node-q/packages/container/hbench-envs%2Fterminal-bench/settings) + changes that. Publishing a `pinned-mirror` version refuses to run until every mirrored image can + be pulled anonymously; +4. commits the run's `images.lock.json` artifact to the `pinned-mirror` version's directory + (`datasets/terminal-bench/2.0.1/`) in a pull request. CI checks it against every task. + +## Pinning upstream images + +A version whose policy is `pinned-upstream` pins upstream's images by digest without copying them, +for images that are not ours to redistribute or are too large to mirror. The **Pin images** workflow +resolves each tag with a registry HEAD request, which Docker Hub does not count as a pull, and writes +`images.lock.json`. Commit it to the version's directory in a pull request, as for a mirror. + +## Building images + +A generated version's environment policy is `built`: every task runs in an image this repository +builds, a tag of the one GHCR package `ghcr.io/node-q/hbench-envs`. `images/bases.yaml` lists base +images, copied from Docker Hub once with their digests unchanged and tagged `base--` +(`base-python-3.12-slim`); `images/images.yaml` lists the built images, tagged `-` +(`humaneval-plus-1.0`), each with its build context and base. Every built image's Dockerfile starts +`FROM` its base's GHCR copy pinned by digest, so neither a build nor a run needs Docker Hub, and the +task's `environment/Dockerfile`, the same file, is what `--force-build` builds. Versions whose tasks +need the same environment share an image: `humanevalfix@1.0`, `mbpp-plus@1.0` and +`cruxeval-input@1.0` run in `humaneval-plus-1.0`, and `generated_source.py lock-image` writes the +locks of every version built on a tag at once. + +To build an image, a maintainer: + +1. runs **Build images** from `main` with its tag (`humaneval-plus-1.0`). The workflow copies any base + not on GHCR yet, builds the image for its platforms, pushes it, refusing an existing tag, and + signs it with cosign keyless; +2. the first time, makes [`hbench-envs`](https://github.com/orgs/node-q/packages/container/hbench-envs/settings) + public: a new GHCR package starts private, and it is one package for every built image and base; +3. commits the run's `locks-` artifact, the `images.lock.json` and `generated.lock.json` of every + version built on the image, in a pull request. + +Until then `images.lock.json` is `{}`: the version validates with a warning, its tasks name no +`docker_image`, and the publish workflow refuses it. + +## images.lock.json + +`images.lock.json` maps each task, sorted, to the image upstream declares or the tag this repository +built (`source`), its digest (`source_digest`) and what `task.toml`'s `docker_image` becomes +(`pinned`); a mirrored image also has its tag in the mirror (`mirror_tag`): + +```json +{ + "adaptive-rejection-sampler": { + "source": "alexgshaw/adaptive-rejection-sampler:20251031", + "source_digest": "sha256:…", + "pinned": "ghcr.io/node-q/hbench-envs/terminal-bench@sha256:…", + "mirror_tag": "adaptive-rejection-sampler-20251031" + } +} +``` + +For `pinned-upstream`, `pinned` is `alexgshaw/adaptive-rejection-sampler@sha256:…` and there is no +`mirror_tag`. A version is committed with `{}` as its lock; it validates with a warning, and the +publish workflow refuses it until the lock is filled. + +## Publishing a version + +A maintainer runs the **Publish** workflow from `main` with the dataset, the version and the +node-q/hbench release to pack with. It: + +1. refuses to run unless that hbench release has `hbench datasets push`, installing it only after + checking `SHA256SUMS` and its build provenance; +2. refuses to overwrite a tag that already exists; +3. fetches the source at its pinned commit, or generates it, and checks its task count and images + against the descriptor; for a version that pins its images, refuses an `images.lock.json` that is + not filled yet, or mirrored or built images that cannot be pulled anonymously, and pins each + task's `docker_image` from the lock; +4. packs and pushes with `hbench datasets push`, then checks the pushed manifest; +5. signs the manifest digest with cosign, attests it with `actions/attest`, and attaches the + Croissant file; +6. writes the digest to the job summary; +7. once an anonymous pull of the new digest works, creates the GitHub release + `-` (for example `terminal-bench-2.0`): the reference and digest, how to run + and verify it, and its `croissant.json`, `source.yaml`, `LICENSE` and `NOTICE` attached. + +A new GHCR package starts private. After the first publish of a dataset, make its package public in +the package's settings +(`https://github.com/orgs/node-q/packages/container/hbench-datasets%2F/settings`), then +re-run the release job.