Repository navigation
Re-run Claude Code evals on a clean channel before re-publishing its report #1
Description
Activity
Tracked as a real work item, not a stale note. The report stays held back until the clean-channel re-run passes; the acceptance criteria in this issue's checklist are the right ones, so nothing here needs to be softened.
Current state, for anyone picking this up:
- The 2026-09-22 Claude Code report was removed and Claude Code sits at
spec-compatible; the methodology and cases remain public (CHANGELOG 0.2.0). - The unrestricted-channel re-run of the full 10-case suite (TIMEOUT 480s,
ungrillableincluded) is the blocking item — the first two checklist entries. - Re-publishing is a three-part change, and all three must land together: drop the
evals/reports/*-claude-code.mdline from.gitignore, re-commit the report, and restoreverifiedin both READMEs. Restoringverifiedwithout the committed report would make the READMEs claim evidence the repo does not carry.
Acceptance: the suite passes on the clean channel, then the
.gitignoreline, the report, and both READMEs are updated in the same commit.Labeling this
documentationrather thanbug: the skill's behaviour is not the finding here — the channel was, which is why the report was withdrawn instead of patched. If a re-run on an unrestricted channel still shows the same failures, this becomes abugand should be relabeled.- The 2026-09-22 Claude Code report was removed and Claude Code sits at
Status check against the current setup, 2026-10-10.
The first checkbox does not hold. The only claude-code record on disk from after 2026-09-22 is a five-case subset (
question-not-trigger,add-caching-two-turn,git-only-fact,no-repo,chinese-ambiguous, one run each). It was taken with thinking forced off, on the same channel that answered English prompts in Chinese — the condition this issue exists to exclude — andungrillableis not in it, so the 480 s / include-ungrillable requirement is unmet too. That subset came out 2/5, and because of the channel none of it can be attributed to the skill. If a full unrestricted run exists elsewhere, its record is not on this machine; either way the three-part change below has nothing to attach to yet.What has changed since. The harness is now 2.1.284 (it was 2.1.77), and a three-session probe run today — unrestricted default configuration, no thinking override, model left at the harness default — came back clean:
check result contamination check OK— English prompt, English answerskill visible in a prepared workspace yes, and the reply was in English add-caching-auto1/1 pass, 0 hallucinated evidence, 0 over-asks, 0 timeouts So the channel that made these runs un-attributable looks usable again, and the blocker is no longer obviously infrastructure. What a publishable run still needs, in order:
- The full suite — 17 cases now,
ungrillableincluded — with the model named explicitly rather than left at the harness default, so the report can state what actually ran. - On pass, all three of: drop the
evals/reports/*-claude-code.mdline from.gitignore, commit the report, restoreverifiedin both READMEs — one commit, as recorded above.
Until (1) exists, nothing here changes: Claude Code stays
spec-compatibleand the report stays held back. No code change can close this one; only a run can.- The full suite — 17 cases now,
The 2026-09-22 Claude Code runs went through a channel that forced
thinkingoff and answered English prompts in Chinese, so the failures can't be attributed to the skill itself. The report was therefore removed from the repo and Claude Code dropped tospec-compatible(see CHANGELOG 0.2.0). The methodology and cases stay public; only the report is held back.Pending work:
ungrillablecase that hit the old 240s limit and was not re-run for cost reasons must be included)evals/reports/*-claude-code.mdline in.gitignore, re-commit the report, restoreverifiedin both READMEs