Token-Optimization Benchmark for Coding-Agent LLM Workflows

A small, self-contained, zero-cost benchmark that measures how much you can cut an LLM coding-agent's context cost by being smart about what you send.

It runs a 6-step coding task over a tiny fake repository twice:

variant	strategy
naive	every step re-sends the FULL conversation history + FULL uncompressed repo, every time
optimized	context selection + history summarization + prompt-caching-friendly prompt structure

It then reports real, measured numbers for both: input tokens (counted locally with tiktoken), output tokens (reported by the free LLM), projected USD cost at published paid rates, and task correctness - plus the % savings.

---

The problem

Real coding agents hold multi-step conversations. That means constantly re-feeding the context window with:

1. the full repo or files,
2. the full message history - which grows, and grows, and is re-sent every turn,
3. $ per token input that is often the dominant cost line.

The naive agent ships everything every time: tokens grow ~quadratically with steps. The whole point of this repo is to quantify, honestly, how much an optimized context strategy saves - and to verify it does not sacrifice task correctness.

The scenario

A fake repository with 3 small Python files (+1 reference doc):

fake_repo/
├── config.py       # constants; TIMEOUT_SECONDS must be changed 30 -> 15
├── utils.py        # contains a planted bug in uniq_preserving_order
├── processor.py    # logic; summarize() semantics asked about
└── docs/STYLE.md   # read-only project conventions (stable + cacheable)

The agent must complete 6 sequential steps:

#	task	kind	graded how
1	Find & fix the bug in uniq_preserving_order	edit	execute the returned function on 4 test cases
2	Add a chunked(items, n) generator to utils.py	edit	execute it on chunk-size/edge cases
3	Change TIMEOUT_SECONDS 30 → 15 in config.py	edit	parse the returned line == 15
4	What does summarize([2, 4, 6]) return?	question	parse dict == {"count":3, "total":12}
5	What is the current RETRY_DELAY_MS?	question	parse int == 500
6	What is len(list(chunked(range(10), 3)))?	question	parse int == 4 (depends on step 2!)

The model's code answers are written back to the variant's own copy of the repo, so later steps act on a genuinely evolved codebase - not a static dump. Correctness is graded per step from the model's actual output, plus an independent end-of-run smoke test that imports the edited files and checks the final repository state. Both variants must pass the same 6 steps; if the optimized one fails it is reported honestly as a failure.

The optimization techniques

1. Context selection (src/context_optimized.py, scenario.render_relevant)

Only the file(s) a step actually touches are included - and for two steps even just the single function that answers them. The repo is small, but the same ratio applies to real repos: an agent fixing a bug in cashier.py does not need analytics.py, models.py and migrations/.

Why it saves tokens: token spend on input is proportional to what you send. Shipping 1 relevant file instead of 50 real files is a ~50× cut.

2. History summarization (OptimizedContext.summary_request / absorb_summary)

Raw history turns accumulate until they exceed the SUMMARY_TOKEN_THRESHOLD (default 600 tokens); then the previous turns are collapsed by the LLM itself into a 3-sentence summary ("fixed uniq_preserving_order in utils.py; added chunked(...); TIMEOUT_SECONDS = 15").

Why it saves tokens: an O(n²) transcript becomes O(n) - a constant-sized block regardless of how many steps the agent has taken. The summary's own LLM call is counted transparently in the report (it is far cheaper than re-sending every raw turn on every remaining step).

3. Prompt-caching-friendly structure (OptimizedContext.static_prefix)

The messages are built as a stable byte-identical prefix followed by a dynamic tail:

[system prompt] + [project conventions doc] + [ack]
        ^--------- constant every step -> cache-eligible -----------^
+ [summarized history] + [current task + relevant files]

The prefix reuses the exact same tokens on every request - precisely what prompt caching matches on - so the provider can serve it at the discounted cached-input rate (GPT-4o-mini: $0.075/M vs $0.15/M). The benchmark measures the real cache-eligible prefix per request as the longest token-exact common prefix between consecutive requests (see caching.py), using tiktoken, and reports:

• cache-eligible = raw repeated-prefix tokens,
• cache-qualified = the subset that would actually qualify, given OpenAI's documented minimum cacheable prefix length (default 1024 tokens, --cache-min-prefix).

Honest note: at this toy scale most prompts are below 1024 tokens, so cache-qualified is often 0 - the code still shows the mechanics and the README explains how the same structure saves real money once the repeated context (docs, tool schemas, unchanged files) exceeds that threshold. cache-eligible is reported separately so the structural gain is visible.

Why these win over the naive approach

The naive build (NaiveContext) does [system] + [every previous turn verbatim, each already containing a full repo dump] + [current task + full repo again]. Input tokens grow journal-fashion every step. The optimized build replaces whole-repo dumps with one relevant file and replaces raw history with a tiny summary - while keeping a cacheable prefix that the naive blob never isolates.

---

Zero-cost guarantee

No paid API is used, ever. The model is resolved in this order:

1. Groq free tier - if GROQ_API_KEY is set (default model llama-3.3-70b-versatile).
2. Google Gemini free tier - if GEMINI_API_KEY is set (default model gemini-2.0-flash).
3. Local Ollama - talks to http://localhost:11434 (default model qwen2.5-coder:3b; falls back to llama3.1:8b style models if you prefer --model llama3.1:8b).

If a provider is unavailable at runtime the script prints a clear message and tries the next free backend (then, if the current call fails mid-run, it re-routes to another free backend). If all backends are unavailable it exits with setup instructions: set GROQ_API_KEY / GEMINI_API_KEY, or run ollama serve and ollama pull qwen2.5-coder:3b.

All input-token counts are computed locally with tiktoken (o200k_base - the GPT-4o family tokenizer matching the priced model), so they are real and reproducible with zero spend. Output tokens come from the provider's usage report for the free completion.

Cost model (projected, not real spend)

The USD figures are projected costs: the real token counts from the run, multiplied by published paid pricing. The benchmark itself spends $0.00.

model	input $/1M	cached input $/1M	output $/1M	source
gpt-4o-mini (primary)	0.150	0.075	0.600	https://developers.openai.com/api/docs/pricing (accessed 2026-08-30)
claude-haiku-4.5 (comparison)	1.00	0.10	5.00	https://platform.claude.com/docs/en/about-claude/pricing (accessed 2026-08-30)

Cached-input pricing applies only to cache-qualified tokens (see above). Prices were current on the access date; always re-check the vendor pricing page before extrapolating to your workload.

How to run

git clone <this repo> && cd token_optimization_benchmark

python3 -m venv .venv && source .venv/bin/activate   # optional, recommended
pip install -r requirements.txt                      # tiktoken + requests

# Zero-cost default: auto-detects Groq -> Gemini -> local Ollama
python benchmark.py

# Force the local model (recommended for a fully offline, 100% free run)
ollama serve &                      # if not already running
ollama pull qwen2.5-coder:3b        # ~1.9 GB download, local+free
python benchmark.py --provider ollama

# Use a free-tier API instead
export GROQ_API_KEY=gsk_...         # or GEMINI_API_KEY=...
python benchmark.py --provider groq

Useful flags: --model, --summary-threshold, --cache-min-prefix, --results-dir, --encoding.

Outputs:

• a results table on stdout,
• results/results.json (full detail incl. per-step records),
• results/results.csv (variants summary),
• results/results_steps.csv (per-step token/cache/correctness rows).

Project layout

token_optimization_benchmark/
├── benchmark.py              # CLI entrypoint, orchestration
├── requirements.txt
├── README.md
├── fake_repo/                # seed codebase copied per variant
├── src/
│   ├── tokenizer.py          # tiktoken counting (real, local)
│   ├── pricing.py            # published pricing + projections
│   ├── llm.py                # groq/gemini/ollama backends + fallback
│   ├── caching.py            # exact cache-eligible prefix measurement
│   ├── scenario.py           # 6 steps, graders, repo edits
│   ├── context_naive.py      # the baseline strategy
│   ├── context_optimized.py  # the three techniques
│   ├── runner.py             # per-variant agent loop
│   └── report.py             # table + JSON/CSV writers
└── results/                  # generated outputs

Measured results

The numbers below come from an actual run of benchmark.py - nothing is hardcoded. Backend: local Ollama, qwen2.5-coder:3b (free, offline, $0.00 spend). Tokens counted with tiktoken (o200k_base); output tokens are the provider-reported counts from the real completions; USD is projected at published gpt-4o-mini rates (cached-input discount applied only to cache-qualified tokens, i.e. prefixes ≥ 1024 tokens).

Run date: 2026-08-30. results/results.json (and .csv) contain the same data plus full per-step detail.

variant     input tok  output tok  total tok  cache-elig  cache-qual  cost(no-cach)  cost(cache)  steps  overall  smoke
---------------------------------------------------------------------------------------------------------------------------
naive          14697         171           14868       10545        9830     $0.002307    $0.001570    6/6      PASS    yes
optimized       4374         230            4604        2039           0     $0.000794    $0.000794    6/6      PASS    yes
SAVINGS (opt vs naive)   70.2%                       69.0%                                             65.6%       49.4%

metric	naive	optimized	reduction
total input tokens sent	14,697	4,374	70.2%
total tokens (input + output)	14,868	4,604	69.0%
projected cost, no cache credit	$0.002307	$0.000794	65.6%
projected cost, with cache credit*	$0.001570	$0.000794	49.4%
steps completed correctly	6/6	6/6	equality preserved
end-of-run repo smoke test	pass	pass	equality preserved

* Naive gets a large cache credit here because its append-only transcript re-sends an ever-longer verbatim prefix that providers match almost exactly. That is a real property of prompt caching - but the naive agent still pays to ship ~3.2× more tokens over the wire, carries stale file dumps into every turn, and would blow past context limits far sooner. The optimized agent never even has a ≥1024-token prefix to cache at this toy scale (cache-eligible = 2,039 tokens measured), yet is still ~half the price.

TL;DR: on this 6-step run the optimized agent used 70% fewer input tokens and projected to cost ~49-66% less, while completing every step correctly - same correctness as the naive agent, no paid API involved.