Skip to content

Polish README and fix benchmark reproducibility - #1

Merged
Hydrodynamical merged 1 commit into
mainfrom
polish-readme-reproducibility
Oct 21, 2026
Merged

Hydrodynamical merged 1 commit into
mainfrom
polish-readme-reproducibility

Conversation

@Hydrodynamical

Copy link
Copy Markdown
Owner

The README pointed to archived entry points, the offline mock never reached a scored prediction, and sweep resumption could conflate prompt and sampling settings. This change gives visitors a tested Python 3.12 quick start, explains the experimental-design task, and records the limits of the available historical evidence.

Changes:

  • Pin the numerical environment and optional provider dependencies; add an offline GitHub Actions workflow.
  • Update parser tests to the active protocol, repair the difficulty-aware mock, and label mock logs correctly.
  • Preserve prompt, temperature, and reasoning settings in sweep identities and logs; prevent raw-log filename collisions.
  • Replace stale paths, output descriptions, budget claims, and the unsupported single-seed comparison in the README.
  • Reconcile manuscript protocol/evidence descriptions and rebuild the PDF, preserving numerical result tables.

Validation:

  • 47 offline tests pass on Python 3.12.12, covering all four mock difficulties and configuration-specific retries/merges.
  • The documented mock command writes a scored JSON log and dashboard.
  • Optional API dependency pins resolve; no live provider benchmark was run.
  • Manuscript compiles, affected pages were visually checked, and no new layout warnings were introduced.

The complete source-run archive behind the historical tables and figures is still absent. The README and manuscript now state that limitation explicitly.

@Hydrodynamical
Hydrodynamical merged commit 763d116 into main Oct 21, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant