Evaluator: survive a failed question, and keep what was already paid for

[adamjohnwright]

A run is bought question by question — a rephrase call, a retrieval, a generation, then a judge pass over all of it. The report was written once, at the very end, after every model and every repeat.

So a rate limit on question 18 of 20 raised, unwound the loop, and discarded the seventeen answers already purchased. Nothing was on disk.

That is why the evaluation is unpleasant to run, and why it has not been run on the four retrieval changes that are owed one.

A failed question no longer ends the run

It is named on stderr, listed under failed_questions in the report, and the rest are scored without it.

The references are re-aligned to the questions that survived

This is the part that had to be got right. Dropping a question from the middle and leaving the reference list alone would score every later answer against the wrong reference — and that does not raise. It produces numbers, and they look like a result.

_kept filters the question list by the same indices, and a test pins the pairing.

Answers are written as they are produced

--transcript-log run.jsonl appends one JSON object per answer and flushes, so whatever was bought survives the process that bought it.

--concurrency answers several at once

Questions are independent, so this does not change what is measured. Results are placed by index, not appended as they arrive — appending would order them by completion time, which is the reference-misalignment bug by another route.

Default is 1, so nothing changes unless asked for.

Tests

Six, with fakes rather than API calls. Four fail against an append-based implementation, which is the mutation worth guarding against: ordering by arrival is plausible, passes casual inspection, and silently scores the wrong pairs.

FAILED test_answer_questions_keeps_what_succeeded
FAILED test_answer_questions_preserves_order_under_concurrency
FAILED test_answer_questions_order_holds_when_a_middle_question_fails
FAILED test_surviving_questions_line_up_with_their_references

Verification

• ruff and ruff format clean, mypy clean, full suite 237 passed
• One note: run ruff from .venv. The repo pins ^0.7.1; a newer ruff reports RUF100 in tests/evaluation/ for a rule this repo does not enable — the same version trap that made LNG parameterize OpenAI model #75 look broken.

Why this one first

It is the gate on everything else in the backlog. #116, #133, #144 and #153 all change what reaches the LLM, and the constitution requires a before-and-after measurement on real questions for exactly that. It is also the gating work for spec 002's D1–D3, and it is owed anyway for four retrieval changes that shipped unevaluated.

🤖 Generated with Claude Code