From 5aeb64f5ef0de1f9fb0ade651da44aa96a37d3d8 Mon Sep 17 00:00:00 2001 From: rzonyx <77541804+rzonyx@users.noreply.github.com> Date: Fri, 23 Oct 2026 18:27:08 +0700 Subject: [PATCH] README: name the action hbench-action and say when Check tasks runs The README called the action "Harness Benchmark" and said Check tasks runs on every push; it runs on pushes to main and on pull requests. Run a dataset now says that n-tasks (5 by default) runs part of a dataset, and which runs a score compares with. The local commands show `hbench tasks check`, and a contact section links the Discord invites and each repository's issues. --- README.md | 35 +++++++++++++++++++++++++++-------- 1 file changed, 27 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index a686231..27711a4 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ # hbench-action-example -An example repository for [Harness Benchmark](https://github.com/node-q/hbench-action) -(`node-q/hbench-action`): three small benchmark tasks, and the workflows that check them and run coding -agents on them with [hbench](https://github.com/node-q/hbench). +An example repository for [hbench-action](https://github.com/node-q/hbench-action), the GitHub Action for +[hbench](https://github.com/node-q/hbench): three small benchmark tasks, and the workflows that check them and +run coding agents on them. Fork it, add a key, and run an agent from the Actions tab. @@ -12,6 +12,7 @@ Fork it, add a key, and run an agent from the Actions tab. - [Run it on your machine](#run-it-on-your-machine) - [Add your own task](#add-your-own-task) - [License](#license) +- [Contributors and contact](#contributors-and-contact) ## What's here @@ -22,7 +23,7 @@ tasks/ count-words/ find the three most frequent words in a text file registry.json the three tasks as a dataset, example@1.0 .github/workflows/ - check-tasks.yml on every push and pull request: the oracle passes every task, an idle agent none + check-tasks.yml on pushes to main and pull requests: the oracle passes every task, an idle agent none run-agent.yml by hand: run an agent on the tasks run-dataset.yml by hand: run an agent on example@1.0 or a public dataset ``` @@ -54,7 +55,8 @@ fix-median/ 3. In the Actions tab, open **Run an agent**, choose **Run workflow**, pick the agent and its model, and run it. The run's summary shows each task's score and the mean reward, and the job's files (trajectories, logs, -results) are attached as an artifact. **Check tasks** needs no key: it runs on every push. +results) are attached as an artifact. **Check tasks** needs no key: it runs on every push to main and every +pull request. ## The workflows @@ -85,7 +87,9 @@ reads only its own. **Run a dataset** runs a dataset instead of a folder: `example@1.0`, listed in this repository's `registry.json`, or one of the public datasets at `oci://ghcr.io/node-q/hbench-datasets`, such as -`humaneval-plus@1.0` or `terminal-bench@2.0.1`, at most `n-tasks` of its tasks. +`humaneval-plus@1.0` or `terminal-bench@2.0.1`. It runs at most `n-tasks` of the dataset's tasks, 5 unless you +change it, so the score covers those tasks only. A score compares only with runs of the same tasks: the same +dataset and version from the same registry, with the same `n-tasks`. ```yaml - uses: node-q/hbench-action@v1 @@ -105,6 +109,7 @@ You need Docker, on Linux or macOS. From a clone of this repository: ```bash curl -fsSL https://raw.githubusercontent.com/node-q/hbench/master/scripts/install.sh | sh +hbench tasks check -p tasks # validate the tasks' files, no key hbench run -p tasks -a oracle # every task, with the oracle hbench datasets list --registry-path registry.json # the datasets in registry.json hbench run -d example@1.0 --registry-path registry.json -a oracle # example@1.0, from registry.json @@ -125,11 +130,25 @@ URL. `/logs/verifier/reward.txt` when the work is done and 0 otherwise. Keep the expected answer in `tests/`, which the agent can't see. 3. Write `solution/solve.sh` so the oracle can pass it. -4. Add it to `registry.json` if it belongs in the dataset, and push: **Check tasks** runs it with the oracle - and with `nop`. +4. Add it to `registry.json` if it belongs in the dataset, and open a pull request: **Check tasks** runs it + with the oracle and with `nop`. To run the same checks on your machine, see [Run it on your machine](#run-it-on-your-machine). ## License Apache-2.0. See [LICENSE](LICENSE). + +## Contributors and contact + +hbench-action-example is maintained by [node-q](https://nodeq.tech). See the +[contributors graph](https://github.com/node-q/hbench-action-example/graphs/contributors) for everyone who has +contributed. + +- Questions and help: [#support on Discord](https://discord.gg/5b6wCn5sEK) +- New releases: [#announcements on Discord](https://discord.gg/dEvy6DCMTn) +- Bugs in this example: [GitHub issues](https://github.com/node-q/hbench-action-example/issues); for the + action, [node-q/hbench-action's issues](https://github.com/node-q/hbench-action/issues); for hbench itself, + [node-q/hbench's issues](https://github.com/node-q/hbench/issues) +- Every input and output of the action: its + [reference](https://github.com/node-q/hbench-action/blob/main/docs/REFERENCE.md)