[hudsonaikins]

The method judge could penalize an omitted cutoff predicate even when the bound source was already truncated, or speculate that a join was required. Supply evaluator-only schema, key counts and source-validated observation cutoffs; accept equivalent calculations and retain uncertainty for unqualified sources.

The context builder and payload are separately versioned, the complete judge implementation participates in its identity, and current evaluator code is copied into both frozen comparison arms. Numerical/evidence scores and historical feedback remain unchanged. Private reference values and planted truth are excluded from prompts.

Offline validation: 159 tests passed with 14 optional skips before the additional development-builder integration test; the final focused context suite passes 8 tests, including all six actual development snapshots. Ruff, formatting, Vulture, whitespace and Gitleaks checks pass.

Live judge validation — 2026-11-13 UTC

Validated PR head 029da01995c259bf10a6bea2171aed0eb64eb959 against the previous evaluator from main 51846a48e52e203eabefd7354182a1bd49ba72b6. Four fixed authored answers and source profiles were graded once by each evaluator using the existing openrouter/anthropic/claude-haiku-5.5 judge, pinned to Anthropic with no fallback or SDK retries.

Control	Expected method score	Previous evaluator	PR #8 evaluator
Valid pre-truncated snapshot	1	0.5	1
Query incorrectly includes future observations	0	0.5	0
Valid equivalent calculation	1	0.5	1
Flat table needs no join	1	1	1

The candidate passes 4/4 targeted method checks, versus 1/4 for the previous evaluator. These are normalized method_matches_question scores, not aggregate rubric scores or population accuracy. The candidate recognizes the validated cutoff and source grain, rejects actual future leakage, and accepts equivalent SQL.

Residual finding: caveats_material remains 0.5 for all three valid controls. Reasons include treating a complete bounded extract as a small sample and requiring snapshot/completeness caveats for an extract-only revenue total. One reason also understates pending-count evidence despite a profile reporting zero null delivery dates. The overall rubric therefore remains imperfect; scores were not adjusted.

Execution and cost: 8 judge calls, no new analyst calls, provider-reported total $0.004787244 against a $0.01 cap. No missing charges or retries. LangSmith read-back verified all 8 root outputs and 16 feedback records. No credits purchased. Remaining API credit after the run: $0.011072809.

Compare the LangSmith experiments.

Reproduction identities: baseline judge fca43a717d4f; candidate judge 2f4c4a230fb7; context dd44858434c4; frozen case SHA-256 c1aaec26f43ce9f5b23d126a05316bb441dbf59293461255be4bce738a74ed0a. Local evidence bundle: data/output/judge-validation-pr8-20261113T051402Z/ (plan, SQLite controls, answers, runner, raw results, read-back receipt).

Limits: four authored controls and one trial per case/evaluator. Human calibration remains waived. This validates the targeted context behavior; it does not establish broad judge accuracy or complete the fresh paired analyst campaign.

PR remains draft. INT-431 remains in progress: resolve the residual caveat-scoring concern and complete the required fresh paired agent evaluation with the same new evaluator. Historical feedback remains unchanged.