Raise the plan-review codex ceiling 1200s -> 2400s - #757
Merged
Conversation
The 1200s cap was calibrated to the plan-review runtimes Campaign 1
observed (up to ~430s), whose plans were far smaller than production
ones. A dense ~500-line plan whose reviewer must verify claims against
15+ repo files runs 3-4x that: codex timed out on 6 of 9 rounds of a
single v4 plan, every failure a genuine
codex exec timed out after 1200.0s (model=gpt-5.6-sol, effort=xhigh)
never an auth or availability error. Retrying is the wrong remedy - a
failed attempt burns the full ceiling before the retry starts (one round
retried once and timed out twice, 40 minutes for no review).
This does not touch the graded engine. The 'do not change without
re-validation' note binds CODEX_MODEL and CODEX_EFFORT (what the
campaign graded); the ceiling is an operational knob, and the campaign
itself ran unattended at 3600s, so 2400 is still below the value the
graded configuration used.
SKILL.md states the ceiling to the reader ("caps at Ns and exits 3"), so
it is updated in the same commit - otherwise the skill's own
instructions describe behavior the code no longer has. The existing
contract test asserted only the code constant and could not catch that,
so it now DERIVES both values and compares them instead of hard-coding
the number twice; verified by injecting the old value and confirming the
test fails.
Trade-off worth stating: 2400s is a 40-minute interactive ceiling. It is
a cap, not a wait - fast reviews still return fast - but a genuinely
hung codex now takes twice as long to reach the loud single-Claude
fallback. Accepted because the observed failures were clean timeouts on
real verification work, not hangs.
Overall Assessment✅ Looks good — no unmitigated P0 or P1 findings. Executive Summary
MethodologyNo findings. No registered causal method or methodological assumption is affected.
Code QualityNo findings. The timeout remains a named constant and its rationale is documented.
PerformanceThe maximum wait before fallback increases by 20 minutes.
MaintainabilityNo findings. The new test compares the documented ceiling with the runtime constant, preventing drift.
Tech DebtNo untracked technical debt introduced.
SecurityNo secrets, unsafe command construction, or permission changes are present.
Documentation/TestsDocumentation and both timeout assertions were updated consistently. The synchronization assertion covers future code/documentation drift.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Raises
CODEX_TIMEOUT_Sin the plan-review skill from 1200s to 2400s, updates the one line ofSKILL.mdthat states the ceiling to the reader, and strengthens the contract test so the two cannot drift apart again.Why. The 1200s cap was calibrated against Campaign 1's plan-review runtimes (up to ~430s), but those plans were far smaller than production ones. A dense ~500-line plan whose reviewer must verify claims against 15+ repo files runs 3–4× that: codex timed out on 6 of 9 rounds of a single v4 plan, every failure a genuine
— never an auth or availability error. Plan review is simply the slow surface: its reviewer reads the plan and cross-checks every claim against the live tree.
Why not retry instead. Retrying was tried first and is strictly worse: a failed attempt burns the full ceiling before the retry starts, so a round that retried once timed out twice — 40 minutes for no review. Retrying a too-low ceiling just pays it twice.
Why this needs no engine re-validation. The "do not change without re-validation" note binds
CODEX_MODELandCODEX_EFFORT— what Campaign 1 actually graded. The ceiling is an operational knob graded by nothing, and the campaign itself ran unattended atCODEX_TIMEOUT_S=3600, so 2400 is still below the value the graded configuration used. The detection prompts, model, and effort are untouched.Doc/code consistency.
SKILL.mdtells the readercodex_review.py"caps at Ns and exits 3", so bumping the constant alone would leave the skill's own instructions describing behavior the code no longer has. The existing contract test asserted only the code constant and could not catch that. It now derives both values and compares them rather than hard-coding the number twice — verified to bite by injecting the old value and confirming the test fails.Trade-off, stated rather than buried: 2400s is a 40-minute interactive ceiling. It is a cap, not a wait — reviews that finish in five minutes still finish in five — but a genuinely hung codex now takes twice as long to reach the loud single-Claude fallback. Accepted because every observed failure was a clean timeout on real verification work, not a hang. The opposite reading is defensible: a plan needing >20 minutes of verification may be a plan that should be split.
Methodology references (required if estimator / math changes)
.claude/skills/plan-review/tooling and its contract test only.Validation
tests/test_plan_review_skill.py—test_reviewer_invocations_are_pinnedextended to derive the ceiling from bothcodex_review.pyandSKILL.mdand assert they agree.1200svalue intoSKILL.mdmakes the new assertion fail, confirming the pin is not vacuous.black --checkandruffclean on both changed files.Security / privacy