terminal-bench-2@1.0, and each dataset numbers its own versions from 1.0

.github/workflows/mirror-images.yml

@@ -27,7 +27,6 @@ env:
   DATASET: ${{ inputs.dataset }}
   VERSION: ${{ inputs.version }}
   REGISTRY: ghcr.io
-  MIRROR_REPOSITORY: ghcr.io/node-q/hbench-envs/${{ inputs.dataset }}
   IMAGES_PER_SHARD: 10
   CERTIFICATE_IDENTITY_REGEXP: ^https://github.com/node-q/hbench-datasets/.github/workflows/mirror-images.yml@
   CERTIFICATE_OIDC_ISSUER: https://token.actions.githubusercontent.com
@@ -210,13 +209,14 @@ jobs:
           overwrite: true
       - name: Write the lock to the job summary
         run: |
+          mirror_repository="$(jq -r .mirror_repository "$RUNNER_TEMP/plan/mirror-plan.json")"
           {
             echo "## Mirrored images of $DATASET@$VERSION"
             echo
-            echo "- Repository: \`$MIRROR_REPOSITORY\`"
+            echo "- Repository: \`$mirror_repository\`"
             echo "- Images: $(jq length "$RUNNER_TEMP/lock/images.lock.json"), each copied with its digest unchanged and signed by this workflow"
             echo
-            echo "Commit this as \`images.lock.json\` in the $DATASET version whose environment policy is \`pinned-mirror\`. A new GHCR package starts private: make \`$MIRROR_REPOSITORY\` public in its package settings (https://github.com/orgs/node-q/packages/container/hbench-envs%2F$DATASET/settings) before publishing that version."
+            echo "Commit this as \`images.lock.json\` in the $DATASET version whose environment policy is \`pinned-mirror\`. A new GHCR package starts private: make \`$mirror_repository\` public in its package settings (https://github.com/orgs/node-q/packages/container/hbench-envs%2F${mirror_repository##*/}/settings) before publishing that version."
             echo
             echo '```json'
             cat "$RUNNER_TEMP/lock/images.lock.json"

.github/workflows/publish.yml

@@ -127,7 +127,7 @@ jobs:
           images_lock="$(dirname "$SOURCE_DESCRIPTOR")/images.lock.json"
           policy="$(python -c 'import sys, yaml; print(yaml.safe_load(open(sys.argv[1]))["environment"]["policy"])' "$SOURCE_DESCRIPTOR")"
           case "$policy" in
-            pinned-mirror) package="hbench-envs/$DATASET" ;;
+            pinned-mirror) package="hbench-envs/$(python -c 'import sys, yaml; d = yaml.safe_load(open(sys.argv[1])); print(d["environment"].get("mirror", d["name"]))' "$SOURCE_DESCRIPTOR")" ;;
             built) package="hbench-envs" ;;
             built-locally) package="" ;;
             *)

CONTRIBUTING.md

@@ -114,10 +114,14 @@ When every task needs its own prebuilt environment, as in SWE-bench, the environ
 
 - A published tag is never moved. A version is published once the publish workflow has finished:
   pushed, signed, attested and with its Croissant file attached.
-- A change of bytes is a new version, in a new directory: a task fixed upstream, a new upstream
-  commit, or images pinned by digest. A change that keeps upstream's version and changes only how it
-  is packaged bumps the last component, as 2.0 to 2.0.1. The old version's directory stays, and
-  [`superseded.yaml`](superseded.yaml) maps it to its successor so the dataset index leaves it out.
+- A dataset's version is this repository's own and starts at 1.0, whatever upstream numbers its
+  release; upstream's release is in the title and `upstream`. When upstream's number names the
+  benchmark rather than a revision of it, it goes in the name: `terminal-bench-2`.
+- A change of bytes is a new version, in a new directory. A new upstream release or commit bumps the
+  first component, as 1.0 to 2.0. A change of ours that keeps upstream's data, such as an adapter fix
+  or images pinned by digest, bumps the last, as 1.0 to 1.0.1. The old version's directory stays, and
+  [`superseded.yaml`](superseded.yaml) maps it to its successor so the dataset index leaves it out
+  once the successor is in it.
 - A publish run that fails after pushing leaves a tag that is not signed. A maintainer deletes that
   package version before running the workflow again.
 

README.md

@@ -37,8 +37,9 @@ new version is a new tag. Each dataset is also mirrored to a second registry, no
 
 | Dataset | Tasks | Licence | Status |
 | --- | ---: | --- | --- |
-| `terminal-bench@2.0` | 89 | Apache-2.0 | Superseded by `terminal-bench@2.0.1`; not listed |
-| `terminal-bench@2.0.1` | 89 | Apache-2.0 | Configured |
+| `terminal-bench@2.0` | 89 | Apache-2.0 | Superseded by `terminal-bench-2@1.0` |
+| `terminal-bench@2.0.1` | 89 | Apache-2.0 | Superseded by `terminal-bench-2@1.0` |
+| `terminal-bench-2@1.0` | 89 | Apache-2.0 | 2.0.1's tasks, byte for byte, under this repository's version numbers |
 | `humaneval-plus@1.0` | 164 | Apache-2.0 | Generated; waits for its image to be built |
 | `bigcodebench@1.0` | 1,135 | Apache-2.0 | Generated, with the Hard subset's 147 tasks tagged `hard`; waits for its image to be built |
 | `humanevalfix@1.0` | 164 | Apache-2.0 | Generated; waits for its image lock |
@@ -50,49 +51,49 @@ new version is a new tag. Each dataset is also mirrored to a second registry, no
 | `swebench-multilingual@1.0` | 283 | Apache-2.0, with its repositories' licences | Generated; environments built locally |
 | `multi-swe-bench@1.0` | 1594 | Apache-2.0, with its repositories' licences | Generated; environments built locally |
 
-A version that [`superseded.yaml`](superseded.yaml) maps to a successor is left out of the dataset index, so
-`hbench datasets list` and the interactive run offer only the successor; the registry keeps serving it to jobs
-that name it.
+A version that [`superseded.yaml`](superseded.yaml) maps to a successor is left out of the dataset index once
+the successor is in it, so `hbench datasets list` and the interactive run offer only the successor; the
+registry keeps serving it to jobs that name it.
 
 ## Running a dataset
 
 This needs an hbench release with the OCI read path (`--registry oci://…`), which is being built.
 
 ```sh
-hbench run -d terminal-bench@2.0.1 --registry oci://ghcr.io/node-q/hbench-datasets
+hbench run -d terminal-bench-2@1.0 --registry oci://ghcr.io/node-q/hbench-datasets
 ```
 
-hbench resolves `terminal-bench:2.0.1` to its manifest digest and checks every layer against it
+hbench resolves `terminal-bench-2:1.0` to its manifest digest and checks every layer against it
 before unpacking a task. Each task's environment image is pinned by digest to its mirror (see
-[Mirrored environments](#mirrored-environments)); in `terminal-bench@2.0`, which 2.0.1 supersedes, the
+[Mirrored environments](#mirrored-environments)); in `terminal-bench@2.0`, which it supersedes, the
 tasks pull them from Docker Hub by tag (`alexgshaw/<task>:20251031`).
 
 ## Verifying a dataset
 
 The signature, made by this repository's publish workflow:
 
 ```sh
-cosign verify ghcr.io/node-q/hbench-datasets/terminal-bench:2.0.1 \
+cosign verify ghcr.io/node-q/hbench-datasets/terminal-bench-2:1.0 \
   --certificate-identity-regexp '^https://github.com/node-q/hbench-datasets/.github/workflows/publish.yml@' \
   --certificate-oidc-issuer https://token.actions.githubusercontent.com
 ```
 
 The build provenance:
 
 ```sh
-gh attestation verify oci://ghcr.io/node-q/hbench-datasets/terminal-bench:2.0.1 --repo node-q/hbench-datasets
+gh attestation verify oci://ghcr.io/node-q/hbench-datasets/terminal-bench-2:1.0 --repo node-q/hbench-datasets
 ```
 
 The Croissant metadata is a referrer of the dataset's manifest:
 
 ```sh
-oras discover --artifact-type application/ld+json ghcr.io/node-q/hbench-datasets/terminal-bench:2.0.1
+oras discover --artifact-type application/ld+json ghcr.io/node-q/hbench-datasets/terminal-bench-2:1.0
 ```
 
 ## Mirroring
 
 ```sh
-oras copy -r ghcr.io/node-q/hbench-datasets/terminal-bench:2.0.1 <registry>/<namespace>/terminal-bench:2.0.1
+oras copy -r ghcr.io/node-q/hbench-datasets/terminal-bench-2:1.0 <registry>/<namespace>/terminal-bench-2:1.0
 ```
 
 `-r` copies the referrers too: the signature, the attestation and the Croissant metadata. Digests do
@@ -107,7 +108,9 @@ and with its digest unchanged, to one GHCR package per dataset, `ghcr.io/node-q/
 tagged `<task>-<upstream tag>` (for example
 `ghcr.io/node-q/hbench-envs/terminal-bench:adaptive-rejection-sampler-20251031`). It signs each digest
 with cosign keyless and writes `images.lock.json`. Each task's `task.toml` then names its image by
-digest, `ghcr.io/node-q/hbench-envs/terminal-bench@sha256:…`, and nothing else in it changes.
+digest, `ghcr.io/node-q/hbench-envs/terminal-bench@sha256:…`, and nothing else in it changes. A
+dataset renamed from another keeps that dataset's package, named by `mirror` in its descriptor's
+`environment`: `terminal-bench-2` keeps `terminal-bench`.
 
 To verify a mirrored image, take its digest from the version's `images.lock.json`:
 
@@ -303,11 +306,11 @@ hbench's `export` command, which is not in a release yet, writes an OCI image la
 directory, and hbench reads it with `--registry oci-layout:<dir>`:
 
 ```sh
-hbench datasets export --oci-layout ./hbench-layout terminal-bench@2.0.1 --registry oci://ghcr.io/node-q/hbench-datasets
-hbench run -d terminal-bench@2.0.1 --registry oci-layout:./hbench-layout
+hbench datasets export --oci-layout ./hbench-layout terminal-bench-2@1.0 --registry oci://ghcr.io/node-q/hbench-datasets
+hbench run -d terminal-bench-2@1.0 --registry oci-layout:./hbench-layout
 ```
 
-`oras copy -r --to-oci-layout ghcr.io/node-q/hbench-datasets/terminal-bench:2.0.1 ./hbench-layout:2.0.1`
+`oras copy -r --to-oci-layout ghcr.io/node-q/hbench-datasets/terminal-bench-2:1.0 ./hbench-layout:1.0`
 writes the same layout today. The tasks' environment images are not part of the layout: pull or
 load them before going offline.
 
@@ -316,9 +319,12 @@ load them before going offline.
 The tooling in this repository is Apache-2.0 ([`LICENSE`](LICENSE)). Each dataset keeps its own
 licence, in its directory and published with it.
 
-- **terminal-bench@2.0** is Apache-2.0; 2.0.1 supersedes it.
-- **terminal-bench@2.0.1** is the same tasks, Apache-2.0. Its environment images are redistributed
-  unmodified from upstream's published images; the software in them carries its own licences.
+- **terminal-bench@2.0** is Apache-2.0; terminal-bench-2@1.0 supersedes it.
+- **terminal-bench@2.0.1** is the same tasks with mirrored images, Apache-2.0; terminal-bench-2@1.0
+  supersedes it.
+- **terminal-bench-2@1.0** is terminal-bench@2.0.1's tasks, byte for byte, Apache-2.0. Its
+  environment images are redistributed unmodified from upstream's published images; the software in
+  them carries its own licences.
 - **humaneval-plus@1.0** is generated by node-q from HumanEval+ (EvalPlus, Apache-2.0), which
   extends OpenAI's HumanEval (MIT). Its task directories are Apache-2.0, with both notices in its
   NOTICE.