▝▜█▄▗▟█▀ ▗▟█ ██╗ ██╗██████╗ ███████╗███╗ ██╗ ██████╗██╗ ██╗
▙▖▝▜█▀ ▗▟█▀ ██║ ██║██╔══██╗██╔════╝████╗ ██║██╔════╝██║ ██║
▀█▙▖ ▗▟██▙ ███████║██████╔╝█████╗ ██╔██╗ ██║██║ ███████║
▀███▀ ▝▜█▄ ██╔══██║██╔══██╗██╔══╝ ██║╚██╗██║██║ ██╔══██║
▗▟█▀ ▄█▙▖▝▜ ██║ ██║██████╔╝███████╗██║ ╚████║╚██████╗██║ ██║
▟█▀ ▄█▛▘▀█▙▖ ╚═╝ ╚═╝╚═════╝ ╚══════╝╚═╝ ╚═══╝ ╚═════╝╚═╝ ╚═╝
hbench is the benchmark engine behind Harness Playground, by node-q.
Run coding agents (Claude Code, Codex, Gemini CLI and more) on benchmark tasks, and see how many they solve.
You need Docker, on Linux or macOS.
curl -fsSL https://github.com/node-q/hbench/releases/latest/download/install.sh | shLater, hbench update installs the newest release.
The script checks the archive it downloads against the release's SHA256SUMS. To check the script itself
before running it, download it, read it, and verify that this repository's release workflow built it (releases
after v0.3.7; gh attestation verify needs the GitHub CLI):
curl -fsSLO https://github.com/node-q/hbench/releases/latest/download/install.sh
curl -fsSLO https://github.com/node-q/hbench/releases/latest/download/SHA256SUMS
grep ' install.sh$' SHA256SUMS | sha256sum -c # macOS: shasum -a 256 -c
gh attestation verify install.sh --repo node-q/hbench
less install.sh
sh install.shThe release archives verify the same way: gh attestation verify hbench-<version>-<target>.tar.gz --repo node-q/hbench.
Or build from source:
cargo install --path .hbench run # pick a dataset, tasks, agent and model
hbench view # see the results in your browserTo run Claude Code, set your Anthropic key first, then pick claude-code:
export ANTHROPIC_API_KEY=sk-ant-...
hbench runOther agents need their own key (see Agents). Every run is saved in jobs/.
The full docs are at nodeq.tech/docs. Good places to start:
- Run your first arena on your machine
- Read your result
- Use a local model server
- Agents and the hbench command line
- Troubleshooting
# Run
hbench run # interactive
hbench run -p examples/tasks/hello-world # a local task
hbench run -d terminal-bench@2.0.1 \
-a claude-code -m anthropic/claude-opus-5-5 # a dataset, with Claude Code
hbench run -c job.yaml # from a job config (YAML/JSON)
# Results
hbench view # web viewer
hbench jobs summary -p jobs/<job>
hbench jobs resume -p jobs/<job>
# Tasks and datasets
hbench init <org/name> # a new task
hbench tasks check -p <tasks> # validate tasks: config, files, tests
hbench check <tasks> # review tasks against a rubric, with claude-code
hbench datasets list|download
# Other
hbench agents list # every agent
hbench agents schema <agent> # an agent's options
hbench --helpUseful options: -l 5 runs only 5 tasks, -k 3 runs each task 3 times, -n 4 runs 4 at a time.
| Agent | What it runs | Example model (-m) |
Key |
|---|---|---|---|
claude-code |
Claude Code | anthropic/claude-opus-5-5 |
ANTHROPIC_API_KEY |
codex |
OpenAI Codex CLI | openai/gpt-6.1-sol |
OPENAI_API_KEY |
gemini-cli |
Google Gemini CLI | google/gemini-3-pro |
GEMINI_API_KEY |
deepseek-harness |
DeepSeek's harness (dsh) | deepseek/deepseek-v4-flash |
DEEPSEEK_API_KEY |
cursor-cli |
Cursor CLI | <provider>/<model> |
CURSOR_API_KEY |
copilot-cli |
GitHub Copilot CLI | <provider>/<model> |
GITHUB_TOKEN |
opencode |
opencode | <provider>/<model> |
the provider's key |
aider |
Aider | <provider>/<model> |
the provider's key |
openhands |
OpenHands | <provider>/<model> |
the provider's key |
oracle |
Each task's reference solution, to check the tasks | ||
| … and a lot more | hbench agents list |
Apache-2.0. See LICENSE.
hbench is built and maintained by node-q. See the contributors graph for everyone who has contributed.
- Questions and help: #support on Discord
- New releases: #announcements on Discord
- Bugs and feature requests: GitHub issues
- The full reference: docs/REFERENCE.md
- Adding an agent: docs/ADDING_AN_AGENT.md