You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Carved out of #1406 (closed as superseded — see its closing comment for what landed elsewhere and what was dropped). These are the two surviving work items, both deterministic PR-time quality improvements to the help-conformance bench, which is now the repo's single non-gating small-model oracle (SkillGym was removed in the #1412 descope).
1. Falsification fixtures for benchmark oracles
Every important oracle must demonstrate it can fail. Extend case/expectation definitions so each named expectation carries:
a minimal passing witness;
at least one known-bad counterexample;
where useful, a metamorphic variant that changes irrelevant nouns/values.
Add a completeness test that rejects a named expectation without falsification fixtures. Include regressions for swallowed lifecycle commands, unsupported flags/selectors, pseudo refs, shell operators, and invalid positional ordering.
2. Typed runner outcomes — infrastructure errors cannot enter model scoring
Replace raw-string success inference with a discriminated outcome:
No duplication of command grammar inside the benchmark — the production positional contract landed in fix(cli): reject excess positionals #1433; the bench derives from CommandSchema.
Carved out of #1406 (closed as superseded — see its closing comment for what landed elsewhere and what was dropped). These are the two surviving work items, both deterministic PR-time quality improvements to the help-conformance bench, which is now the repo's single non-gating small-model oracle (SkillGym was removed in the #1412 descope).
1. Falsification fixtures for benchmark oracles
Every important oracle must demonstrate it can fail. Extend case/expectation definitions so each named expectation carries:
Add a completeness test that rejects a named expectation without falsification fixtures. Include regressions for swallowed lifecycle commands, unsupported flags/selectors, pseudo refs, shell operators, and invalid positional ordering.
2. Typed runner outcomes — infrastructure errors cannot enter model scoring
Replace raw-string success inference with a discriminated outcome:
successoutcomes may reach command validation and scoring.runnerErrorand model-validation failures.N/A, not0/0 (0%).--evidencemulti-trial reporting mode from Guard help-conformance correctness at production and CI boundaries #1406 is optional follow-up, on demand — not part of this issue.)Acceptance criteria
N/A.Non-goals (decided in the #1412 descope)
pnpm bench:help-conformance).CommandSchema.Related: #1406 (superseded parent), #1433 (production positional contract, landed), #1411 (renderer-pinned bench gates, landed).