fix(codex): report per-turn tokens instead of the thread-cumulative total - #113
Conversation
…otal The Codex SDK's ThreadTokenUsage.total counts the whole thread, not the turn, and the thread is created once per task and reused for every turn. Each turn reported the running total, and the orchestrator sums per-turn usages into the task total, so a task's tokens became a sum of prefix sums — inflating an N-turn task by roughly (N+1)/2. Track the cumulative snapshot on the agent and report each turn's delta. The crash/timeout fallback reads per-generation tokens off the flushed messages, so it advances the same baseline to stay in step. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Claude finished @bai-uipath's task in 2m 4s —— View job Code Review
|
Multi-model code reviewReviewers: gemini-3.1-pro-preview, gpt-5.6-sol, and an Opus sub-agent (4 lenses + mutation testing), reviewing Scope:
🟠 High —
|
uipreliga
left a comment
uipreliga
left a comment
There was a problem hiding this comment.
Fix what you agree with and 🚢
2 participants
What's wrong
The Codex SDK's thread token total counts the whole thread, not the turn. The Codex thread is created once per task and reused for every turn, so by turn N that total still carries turns 1 through N-1. Each turn was reporting the running total, and the orchestrator sums per-turn usages into the task total. A task's reported tokens therefore became a sum of prefix sums, inflating a task by roughly (N+1)/2 in the number of turns.
Scope: model calls within a turn are fine, simulation turns are not
These are two different levels of nesting, and only the outer one is affected.
Model calls inside a single turn (each tool call leading to the next generation) were always correct. Those are booked from the SDK's per-generation delta, and the thread total at the end of turn 1 is exactly the sum of turn 1's generations. A task that runs 14 model calls in one turn reports all 14 correctly.
Simulation dialog turns (each
communicate()call, driven by a task'ssimulation:block) are where the double-count happens, because the thread persists across them while the orchestrator adds up what each turn claims to have spent.So single-turn tasks were never affected regardless of how many tool calls they made. On the 2026-08-13 nightly, 490 of the 618 tasks that generated anything ran a single turn and were exactly right. The damage was confined to the 128 multi-turn tasks.
Impact
The affected tasks are a minority, but an expensive one. On the 2026-08-13 nightly the 128 multi-turn tasks were 21% of the tasks that generated anything, yet carried 46% of the reported agent spend, because simulation dialogs are the long tasks to begin with. They were then inflated on top of that.
Multi-turn tasks reported 2.07x their real tokens on average, and individual tasks ran as high as 5.3x. At the run level that is roughly $41 of a $175.03 nightly, about 23.5%.
The $1.28 in the single-turn row is not part of the defect. It is sub-agent tokens that the per-generation ground truth does not see, since children run on their own threads. The real figure is the multi-turn $41.16.
Token and USD run budgets read the same aggregate, so a multi-turn Codex task could also trip
max_total_tokensormax_usdwell before it actually should.The distortion is not a constant
It tracks how much of the suite is multi-turn, which has been moving:
07-28 carries the largest dollar error despite the smallest multi-turn share, because that run was bigger and those 92 tasks were deeper dialogs. So run-over-run Codex cost comparisons have been reading a distortion that changes size each night, which is harder to spot than a fixed bias would be.
Why it went unnoticed
Every task stayed internally consistent. The synthetic reconciliation message absorbed the gap turn by turn, so the message stream still summed to the turn total, just an inflated one. Any audit checking "do the messages sum to the turn total?" passed. Summing every reconciliation row across that run recovers the double-counted amount to within 0.06%, which is the clearest signal that the residual was the bug rather than the expected small prompt slice it is documented to carry.
It is also Codex-only, so in cross-harness comparisons it read as "Codex costs more than Claude" rather than as a defect.
Verification
Live end-to-end run
skill-troubleshoot-no-host-pendingrun on the eval VM against a real Codex thread and a real Bedrock-backed simulator, once onmainand once on this branch. Same task, same command, same model.Before, 5 simulation turns. Turn 1 is exact; every later turn reports the running total:
Task total 2,948,104 reported against 891,026 spent, 3.36x, priced at $1.4503.
After, 3 simulation turns (the simulated dialog ends where it ends, so the two runs are not the same trajectory and the dollar figures are not directly comparable — the invariant is):
Every turn now reports exactly what it spent, and the task total matches ground truth exactly: 1.0000x, priced at $0.2585. On this same trajectory the old code would have booked 1,676,555 input tokens against 615,260 real, 2.72x, or about $0.70.
The reconciliation rows corroborate it independently. They ran 382,894 / 425,633 / 635,114 / 824,522 across the turns of the pre-fix run, and collapse to 0 / 977 / 2 after — back to the near-zero residual Codex is documented to produce, since its message stream is already complete.
Worked example from the 2026-08-13 nightly
skill-troubleshoot-export-pdf-com-hang, 4 simulation turns, 21 model calls in total:Every row's reported figure is the previous row's reported figure plus that turn's own spend, and each reconciliation row is almost exactly the previous turn's cumulative total. Turn 1 is correct including all 14 of its model calls. The last row, 730,350, is the correct task total; summing the reported column instead gives 2,209,246 (3.02x), and the task was priced at $1.0660 instead of ~$0.3524.
Replay against production data
All 128 multi-turn tasks from that nightly, replayed through the corrected path:
123 of the 128 land within 0.5% of ground truth. The remaining 0.2% is sub-agent tokens, which are folded in separately and are not part of the parent thread's total.
Tests
tests/test_codex_token_mapping.pycovers the multi-turn delta, the invariant that summed turns equal the thread's final cumulative total, per-turn cost, per-agent baseline isolation, a restarted thread, and the turn following a crash. Four of them were confirmed to fail against the pre-fix behavior rather than passing vacuously.make format,make check,make typecheckandmake lintare clean.make testhas 4 failures, all pre-existing and unrelated: 3 intest_reports_stats_nonfinitethat reproduce on an unmodifiedorigin/main(a Python 3.13.7statisticschange), and 1 intest_litellm_routethat depends on a local.envmodel pin.Other harnesses
Unaffected and unchanged. Claude Code and Antigravity already report per-turn figures, confirmed empirically on their own full nightlies rather than by reading the code: both come out at 1.00x against ground truth at every turn count, including tasks running 3, 4 and 6 turns.