A job summary that says how many tasks were solved, at what cost

.github/workflows/test.yml

@@ -19,7 +19,7 @@ defaults:
 
 jobs:
   lint:
-    name: actionlint, shellcheck and the report's syntax
+    name: actionlint, shellcheck and the report's tests
     runs-on: ubuntu-24.04
     timeout-minutes: 10
     env:
@@ -41,8 +41,8 @@ jobs:
         run: '"$RUNNER_TEMP/actionlint" -color'
       - name: Lint the scripts
         run: shellcheck scripts/*.sh
-      - name: Compile the report
-        run: python3 -m py_compile scripts/report.py
+      - name: Test the report
+        run: python3 -m unittest discover --start-directory tests --pattern 'test_*.py' --verbose
 
   install:
     name: Install ${{ matrix.version }} on ${{ matrix.os }}
@@ -99,10 +99,13 @@ jobs:
           MEAN_REWARD: ${{ steps.hbench.outputs.mean-reward }}
           N_TRIALS: ${{ steps.hbench.outputs.n-trials }}
           N_ERRORS: ${{ steps.hbench.outputs.n-errors }}
+          N_SOLVED: ${{ steps.hbench.outputs.n-solved }}
+          PASS_RATE: ${{ steps.hbench.outputs.pass-rate }}
           RESULT_PATH: ${{ steps.hbench.outputs.result-path }}
           ARTIFACT_ID: ${{ steps.hbench.outputs.artifact-id }}
         run: |
           [[ "$MEAN_REWARD" == 1.0 && "$N_TRIALS" == 1 && "$N_ERRORS" == 0 ]]
+          [[ "$N_SOLVED" == 1 && "$PASS_RATE" == 1.0 ]]
           [[ -f "$RESULT_PATH" && -n "$ARTIFACT_ID" ]]
 
   run-below-threshold:
@@ -128,8 +131,9 @@ jobs:
           OUTCOME: ${{ steps.hbench.outcome }}
           MEAN_REWARD: ${{ steps.hbench.outputs.mean-reward }}
           N_ERRORS: ${{ steps.hbench.outputs.n-errors }}
+          N_SOLVED: ${{ steps.hbench.outputs.n-solved }}
         run: |
-          [[ "$OUTCOME" == failure && "$MEAN_REWARD" == 0.0 && "$N_ERRORS" == 0 ]]
+          [[ "$OUTCOME" == failure && "$MEAN_REWARD" == 0.0 && "$N_ERRORS" == 0 && "$N_SOLVED" == 0 ]]
 
   run-oci-dataset:
     name: Run the oracle on an OCI dataset, locked by digest

.gitignore

@@ -0,0 +1 @@
+__pycache__/

README.md

@@ -116,7 +116,13 @@ For hbench itself, the docs are at [nodeq.tech/docs](https://nodeq.tech/docs):
 ```
 
 With none of `path`, `dataset` or `config`, the action only installs hbench, and later steps can call
-`hbench` themselves. Useful outputs: `mean-reward`, `n-trials`, `n-errors` and `job-dir`.
+`hbench` themselves. Useful outputs: `pass-rate`, `n-solved`, `mean-reward`, `cost-usd` and `job-dir`.
+
+The job summary shows, for each agent and model, how many tasks it solved, its mean reward and pass@k, its
+errors, the time it spent, its tokens and its cost per solved task; the trials by outcome and each eval's
+mean reward, each with a bar; the command that opens the job in
+hbench's interactive dashboard on your machine; what ran (the hbench release, the agent, the model, the tasks
+and their digest); the errors by type; and a row per task.
 
 A `registry.json` lists datasets and where each of their tasks is: a git repository at a commit, or a folder
 in your repository, read from `working-directory`:

action.yml

@@ -96,6 +96,15 @@ outputs:
   n-errors:
     description: Trials that ended in an error.
     value: ${{ steps.run.outputs.n-errors }}
+  n-solved:
+    description: Trials solved, with a reward of 1.
+    value: ${{ steps.run.outputs.n-solved }}
+  pass-rate:
+    description: Solved trials over all trials, from 0 to 1.
+    value: ${{ steps.run.outputs.pass-rate }}
+  cost-usd:
+    description: The agents' cost in US dollars, where hbench knows it. Empty for agents that report none, such as the oracle.
+    value: ${{ steps.run.outputs.cost-usd }}
   artifact-id:
     description: ID of the uploaded artifact.
     value: ${{ steps.upload.outputs.artifact-id }}
@@ -134,6 +143,7 @@ runs:
         INPUT_JOBS_DIR: ${{ inputs.jobs-dir }}
         INPUT_MIN_MEAN_REWARD: ${{ inputs.min-mean-reward }}
         INPUT_ARTIFACT_NAME: ${{ inputs.artifact-name }}
+        INPUT_UPLOAD_ARTIFACT: ${{ inputs.upload-artifact }}
       run: '"$GITHUB_ACTION_PATH/scripts/run.sh"'
 
     - name: Upload the job directory

docs/REFERENCE.md

@@ -47,10 +47,30 @@ The [README](../README.md) has the short version.
 | `mean-reward` | Mean reward over every trial of every eval. |
 | `n-trials` | Trials run. |
 | `n-errors` | Trials that ended in an error. |
+| `n-solved` | Trials solved, with a reward of 1. |
+| `pass-rate` | Solved trials over all trials, from 0 to 1. |
+| `cost-usd` | The agents' cost in US dollars, where hbench knows it. Empty for agents that report none, such as the oracle. |
 | `artifact-id` | ID of the uploaded artifact. |
 
-`mean-reward` weighs each eval's mean reward by its trials. The job summary has a row per eval and lists the
-trials below a reward of 1.
+A trial is solved when its reward is 1, so `pass-rate` is the share of trials that got full marks.
+`mean-reward` weighs each eval's mean reward by its trials; it differs from `pass-rate` when tasks give
+partial rewards.
+
+The job summary has:
+
+- a row per eval (agent, model and tasks): solved trials and the pass rate, mean reward, pass@k, errors,
+  the agent's time in total and per trial (median), tokens in and out, and the cost in total and per
+  solved trial;
+- whether the mean reward reached `min-mean-reward`;
+- two tables with bars of up to 20 blocks: the trials solved, failed and ended in an error, with each
+  share, and each eval's mean reward;
+- when the job is uploaded, the command that downloads it and opens it in hbench's interactive dashboard,
+  `gh run download <run> -R <repository> -n <artifact> -D hbench-jobs/<job> && hbench view hbench-jobs`;
+- what ran: the hbench release and results epoch, the agent and model, the tasks with their registry and
+  manifest digest, attempts and concurrency, and when the job ran;
+- the tasks' own metrics, the other keys of their `reward.json`, as means;
+- the errors by type, with the trials they hit;
+- a row per trial, errors and failures first: result, reward, agent time, tokens and cost.
 
 ## More examples
 
@@ -162,15 +182,16 @@ Pass any other `hbench run` flag through `args`, one argument per line:
 action.yml          the inputs, outputs and three steps: install, run, upload
 scripts/install.sh  download, checksum, provenance, tool cache
 scripts/run.sh      build the hbench run command line, run it, report
-scripts/report.py   result.json to outputs, job summary and the threshold
+scripts/report.py   the job's results to outputs, job summary and the threshold
+tests/test_report.py  the report, on the two job directories in tests/report/
 tests/tasks/        a one-task folder the tests run
 ```
 
-`.github/workflows/test.yml` runs on GitHub-hosted runners only: actionlint and shellcheck; installing
-`latest` and a pinned release on Linux x86_64 and arm64 and macOS arm64 and x86_64, with provenance checked;
-the oracle on the local task with `min-mean-reward: '1'`; the `nop` agent, which must fail the step on its
-score; two tasks of an OCI dataset with `--lock-write`; and a dataset listed in a `registry.json`, by path and
-by URL.
+`.github/workflows/test.yml` runs on GitHub-hosted runners only: actionlint, shellcheck and the report's unit
+tests; installing `latest` and a pinned release on Linux x86_64 and arm64 and macOS arm64 and x86_64, with
+provenance checked; the oracle on the local task with `min-mean-reward: '1'`; the `nop` agent, which must fail
+the step on its score; two tasks of an OCI dataset with `--lock-write`; and a dataset listed in a
+`registry.json`, by path and by URL.
 
 To try the scripts outside GitHub, set the variables the runner sets (`RUNNER_TEMP`, `GITHUB_ACTION_PATH`,
 `GITHUB_OUTPUT`, `GITHUB_PATH`, `GITHUB_STEP_SUMMARY`) and the `INPUT_*` ones `action.yml` passes, then run