Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 27 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# hbench-action-example

An example repository for [Harness Benchmark](https://github.com/node-q/hbench-action)
(`node-q/hbench-action`): three small benchmark tasks, and the workflows that check them and run coding
agents on them with [hbench](https://github.com/node-q/hbench).
An example repository for [hbench-action](https://github.com/node-q/hbench-action), the GitHub Action for
[hbench](https://github.com/node-q/hbench): three small benchmark tasks, and the workflows that check them and
run coding agents on them.

Fork it, add a key, and run an agent from the Actions tab.

Expand All @@ -12,6 +12,7 @@ Fork it, add a key, and run an agent from the Actions tab.
- [Run it on your machine](#run-it-on-your-machine)
- [Add your own task](#add-your-own-task)
- [License](#license)
- [Contributors and contact](#contributors-and-contact)

## What's here

Expand All @@ -22,7 +23,7 @@ tasks/
count-words/ find the three most frequent words in a text file
registry.json the three tasks as a dataset, example@1.0
.github/workflows/
check-tasks.yml on every push and pull request: the oracle passes every task, an idle agent none
check-tasks.yml on pushes to main and pull requests: the oracle passes every task, an idle agent none
run-agent.yml by hand: run an agent on the tasks
run-dataset.yml by hand: run an agent on example@1.0 or a public dataset
```
Expand Down Expand Up @@ -54,7 +55,8 @@ fix-median/
3. In the Actions tab, open **Run an agent**, choose **Run workflow**, pick the agent and its model, and run it.

The run's summary shows each task's score and the mean reward, and the job's files (trajectories, logs,
results) are attached as an artifact. **Check tasks** needs no key: it runs on every push.
results) are attached as an artifact. **Check tasks** needs no key: it runs on every push to main and every
pull request.

## The workflows

Expand Down Expand Up @@ -85,7 +87,9 @@ reads only its own.

**Run a dataset** runs a dataset instead of a folder: `example@1.0`, listed in this repository's
`registry.json`, or one of the public datasets at `oci://ghcr.io/node-q/hbench-datasets`, such as
`humaneval-plus@1.0` or `terminal-bench@2.0.1`, at most `n-tasks` of its tasks.
`humaneval-plus@1.0` or `terminal-bench@2.0.1`. It runs at most `n-tasks` of the dataset's tasks, 5 unless you
change it, so the score covers those tasks only. A score compares only with runs of the same tasks: the same
dataset and version from the same registry, with the same `n-tasks`.

```yaml
- uses: node-q/hbench-action@v1
Expand All @@ -105,6 +109,7 @@ You need Docker, on Linux or macOS. From a clone of this repository:
```bash
curl -fsSL https://raw.githubusercontent.com/node-q/hbench/master/scripts/install.sh | sh

hbench tasks check -p tasks # validate the tasks' files, no key
hbench run -p tasks -a oracle # every task, with the oracle
hbench datasets list --registry-path registry.json # the datasets in registry.json
hbench run -d example@1.0 --registry-path registry.json -a oracle # example@1.0, from registry.json
Expand All @@ -125,11 +130,25 @@ URL.
`/logs/verifier/reward.txt` when the work is done and 0 otherwise. Keep the expected answer in `tests/`,
which the agent can't see.
3. Write `solution/solve.sh` so the oracle can pass it.
4. Add it to `registry.json` if it belongs in the dataset, and push: **Check tasks** runs it with the oracle
and with `nop`.
4. Add it to `registry.json` if it belongs in the dataset, and open a pull request: **Check tasks** runs it
with the oracle and with `nop`.

To run the same checks on your machine, see [Run it on your machine](#run-it-on-your-machine).

## License

Apache-2.0. See [LICENSE](LICENSE).

## Contributors and contact

hbench-action-example is maintained by [node-q](https://nodeq.tech). See the
[contributors graph](https://github.com/node-q/hbench-action-example/graphs/contributors) for everyone who has
contributed.

- Questions and help: [#support on Discord](https://discord.gg/5b6wCn5sEK)
- New releases: [#announcements on Discord](https://discord.gg/dEvy6DCMTn)
- Bugs in this example: [GitHub issues](https://github.com/node-q/hbench-action-example/issues); for the
action, [node-q/hbench-action's issues](https://github.com/node-q/hbench-action/issues); for hbench itself,
[node-q/hbench's issues](https://github.com/node-q/hbench/issues)
- Every input and output of the action: its
[reference](https://github.com/node-q/hbench-action/blob/main/docs/REFERENCE.md)