Skip to content

Feat/assessment benchmark - #83

Open
tebanieo wants to merge 9 commits into
mainfrom
feat/assessment-benchmark
Open

Feat/assessment benchmark#83
tebanieo wants to merge 9 commits into
mainfrom
feat/assessment-benchmark

Conversation

@tebanieo

Copy link
Copy Markdown
Contributor

Closes #82

Assessment Quality Benchmark

Adds a staged benchmark that scores pipeline output against human-reviewed reference answers, so we can tell good runs from bad and catch quality regressions. v1 implements the engine-assignment stage end-to-end.

What's included

  • Harness: case loader + manifest, acceptable-set scoring, assignment stage adapter, outcome-aware reporting with a trust banner, CLI (python -m benchmarks.runner.run --stage assignment)
  • Assignment stage (deterministic-regression mode that is rules based, where a score drop = a code regression)
  • Seed dataset: 2 human-reviewed cases + benchmarks/README.md
  • ADR-024 (Proposed) recording the design

Design highlights

  • Dataset is the durable asset; runner is thin/replaceable (SWE-bench model)
  • Two modes: deterministic-regression (assignment) vs. LLM-quality (triage/schema/synthesis, later)
  • Acceptable-set-per-query scoring (avoids penalizing defensible choices)
  • Gold-input isolation (no error amplification across stages)
  • Infra failures (throttles) quarantined, never scored as wrong; incomplete runs flagged

Testing

13 benchmark tests + full unit suite pass. src/ untouched (purely additive).

Out of scope (v1)

LLM-quality stages, performance/cost benchmark, synthetic case generation, CI wiring.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature][Proposal] Assessment Quality Benchmark

1 participant