Repository navigation
A job summary that says how many tasks were solved, at what cost - #7
Merged
Merged
Conversation
The summary had each eval's mean reward and pass@k. It now reads the job's config.json, lock.json and each trial's result.json too: - a row per eval: solved trials and the pass rate (a trial is solved when its reward is 1), mean reward, pass@k, errors, the agent's time in total and per trial, tokens in and out, and the cost in total and per solved trial; - what ran: hbench and its results epoch, the agent and model, the tasks with their registry and manifest digest, attempts, concurrency and when the job ran; - the tasks' own metrics, the errors by type, and a row per trial, errors and failures first. New outputs: n-solved, pass-rate and cost-usd. tests/test_report.py checks the report on two job directories, one an agent's with a solved, a failed and a timed-out trial, one the oracle's with reward.txt.
The previous commit added the __pycache__ the unit tests wrote.
hbench's dashboard (hbench view) is a web app with its own server, so it cannot run in a job summary, which renders sanitised Markdown only. When the job is uploaded, the summary now says "View interactive dashboard by hbench:" with the command that downloads this run's artifact into hbench-jobs/<job> and opens it: hbench view needs the folder above the job, or it takes the trials for jobs.
Job summaries render Mermaid, so the summary now draws a pie of the trials solved, failed and ended in an error (an empty slice left out) and a bar chart of each eval's mean reward, from 0 to 1.
The charts stretched to the summary's full width in Mermaid's default theme. They now keep a fixed size (useMaxWidth off; the bar chart 360 px wide per few evals, 220 px high) and use the viewer's palette from viewer/src/tokens.css: good, bad and warn for solved, failed and errored trials, the accent for the bars, the muted ink for text. Slices are sorted by count so each colour stays on its label whether or not Mermaid sorts them.
A row each for solved, failed and errored trials: the count, and the share as a bar of up to 20 blocks with its percentage. It reads as numbers first and renders anywhere Markdown does; any outcome above 0 gets at least one block. The mean reward by eval stays a Mermaid bar chart.
Like the trials by outcome: a row per eval with its mean reward as a bar of up to 20 blocks and the number. The summary no longer needs Mermaid, so it reads the same wherever Markdown renders.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The summary now shows, per eval, solved trials and pass rate, mean reward, pass@k, errors, agent time, tokens and cost per solved trial; then what ran (hbench, agent, model, tasks and digest), the tasks' own metrics, errors by type and a row per trial. New outputs
n-solved,pass-rate,cost-usd. Unit tests cover the report; checked on real jobs with hbench 0.3.0.