Repository navigation
fix(mcp): stop execute programs at a CodeMode step budget - #105
Merged
Merged
Conversation
A program that only computes ran on the product's own thread until its five-minute wall-time budget. On Nov 18 one execute spent 203 s in the interpreter with no tool calls and kept production's product isolate near 100% CPU. The CodeMode patch adds ExecutionLimits.maxSteps: statements and expressions evaluated across the whole program. Waiting on tools evaluates nothing, so only interpreter work counts. Past the limit every step throws a TimeoutExceeded the program's own catch, finally or reactions cannot stop. McpLimits gains maxSteps (default 5,000,000, about 9 s in workerd); self-host and local read EXECUTOR_MCP_MAX_STEPS. Execute keeps CodeMode's message, which names the budget and asks for heavy work to move out of the program, and tags the span executor.timeout.budget=steps.
A loop of large native calls, such as split on a 1 MB string, was one step per call and ran 83 s inside a 5,000,000-step budget. invokeCallable now charges every native call for its input before it runs and its result after, at one step per 2 elements or keys and one per 256 characters; spread and new are charged at the same rate. The review's split PoC now stops in 3.4 s on the self-host e2e server.
…am holds Review round 1 found two holes in the native-call charge. Binary operators deep-walked both operands for one step, so `id in index` over a 20,000-record index ran to its 120 s wall time; and every call's result was charged by size, even an existing reference, so grouping 50,000 rows with Map.get into 10 or 50 groups stopped at the budget in under a second. Results the program already holds (receiver, an argument, or an element or accumulator from get, at, pop, shift, find, findLast, reduce) are no longer charged. `in` reads one key and no longer walks its object; other operators' walks are charged as they go and stop partway, and primitives skip them. String comparisons, templates of arrays, rest patterns, for...in key lists and string-consuming Set, Map and spread are charged too. Record width samples 8 elements, capped at 64; scans charge per 512 items; normalize and localeCompare charge 8x. The e2e scenario adds the operator runaway, the `in` dedupe and the 10- and 50-group group-by. Backtracking regexes remain uncharged and documented.
Storing a value (assignment, push, unshift, splice, fill, Object.assign) walked it to reject cycles without charging, so attaching one large shared value to every row, or reassigning a group after each push, ran for minutes. The walk is now charged per 2 values visited, and remembers objects that hold no objects so it stops there next time. switch walks are charged the same way. `+` yielding a string was one step however long the result, and reading the rope it made copied it all. It is now charged one step per 4,096 characters of the result. The doc states the stop times seen under load and lists the walks left uncharged.
`g[k] = g[k] || []` stored the group back into its own slot on every row, and the store walked the whole group each time, so grouping 50,000 row objects stopped at the budget. Storing the value a slot already holds cannot change the graph, so it no longer walks. A template literal is charged one step per 4,096 characters of its result, like string `+`; interpolating a megabyte string in a loop ran to the wall clock.
shuv1337
added a commit
that referenced
this pull request
Nov 19, 2026
…search in use (#107) * Stop the store walk at any value it has walked, not only leaves A shared value of many small records, such as a meta object of 1,000 records attached to every row, was walked record by record on every store: about 500 steps each, 10 million for 20,000 rows, so the program stopped at the 5,000,000-step budget. The store walk now remembers every object whose whole graph it checked. A leaf stays remembered until an object is stored into it; any other object is remembered for the current epoch, and storing an object into a remembered object that another remembered object holds starts a new epoch. A differential fuzz of 3.9 million stores against a fresh reachability check agrees on all of them and fails at once when the epoch is not advanced. mcp-execute-step-budget: 20,000 rows sharing a 1,000-record meta now complete in 283 ms (TimeoutExceeded at 1,317 ms before); storing a row into that shared value is still rejected as a cycle; runaways still stop with TimeoutExceeded. * Never evict a search still being paged, and keep oversized searches apart Kept search orders (#106) had three gaps: - room was checked before the ranking await and eviction after it was strict LRU, so new searches ranking at once evicted searches still in use; an order is now admitted after the await, evicting only searches unread for 30 s, or not kept; - idle time read the wall clock; it now reads the monotonic clock; - a search too large to keep was remembered in the same cache with its key's bytes, and a key past 6 MB evicted every kept search; such searches are now remembered in their own bounded map, and a key too large to keep is never remembered. mcp-search-kept-orders covers the first and third: both assertions fail on the previous store and pass on this one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On Nov 18 2026 (about 3:26–3:33 PM PT) two
executeprograms on production each spent about 203 s in the CodeMode interpreter with no tool calls. The interpreter runs on the product workerd's main thread, and its only budgets were wall time (timeoutMs, 5 minutes) andmaxToolCalls(100). A program that only computes ran inside its 5 minutes and kept the product isolate near 100% CPU. Effect preempts the program's fiber every 2,048 operations, so other requests still ran, but slowly, and clients gave up.What
patches/@opencode-ai%2Fcodemode@0.0.0-dev-19272.patch, notes inpatches/codemode-step-budget.md): addsExecutionLimits.maxSteps. AStepBudgetis shared by everyInterpreterof one execution (invokeFunctioncreates one per call).evaluateStatementandevaluateExpressioneach count one step. Waiting on a tool call or approval evaluates nothing, so only interpreter work counts.StepBudgetExceeded, anInterpreterRuntimeErrorof kindTimeoutExceeded.trypasses it on without runningcatch, and because each later step throws again, nocatch,finallyor promise reaction can keep the program running.McpLimits.maxStepsdefaults to 5,000,000. Self-host and local readEXECUTOR_MCP_MAX_STEPS, documented in the local README. Execute keeps CodeMode's message instead of replacing it with the wall-time message, and tagsmcp.executewithexecutor.timeout.budget=steps.EXECUTOR_MCP_TIMEOUT_MSis unchanged.The message the agent gets:
What is charged
Round 0 showed a loop of
s.split("")on a 1 MB string running 83 s inside a 5 M budget. Round 1 showed two more holes: operators that walk a whole value for 1 step, and a regression where results the program already holds were charged as if copied. The rule now is charge data that is actually read, copied or scanned, never the size of a reference that is handed back.invokeCallable, the one dispatch every call goes through, charges each native call. Calls that run program code or wait on the host count their own steps and aren't charged: tools, program functions, promise and generator methods, andtools.search.JSON,Object,Array,Math…). Two exceptions:Object.assign's target, which it only writes, andObject.hasOwnandArray.isArray, which read one key or the type.Map.set), an argument (Object.assign), or an element or accumulator (Map.get,at,pop,shift,find,findLast,reduce).sort,reverseandfillstill pay for their pass.===/!==, and unary+,-and~, walk their operands to reject interpreter references before coercing them. The walk is now charged as it goes, 1 step per 2 values visited, so it can stop partway. Primitives skip the walk entirely.inreads one own key, so it no longer walks its right-hand object at all; it only checks that the object itself is data. One behavior change:"a" in onow works whenoholds a function somewhere inside, where it used to throw.+(and+=) that yields a string is charged 1 step per 4,096 characters of its result. Round 4: so is a template literal;`${s}${i}`on a 1 MB string ran to the wall clock and now stops in about 1.5 s. The engine joins strings as ropes, so the join is cheap but the next read copies the whole result:t = s + ion a 1 MB string, thent.charCodeAt(…), ran to the wall clock and now stops in about 1.5 s. 4,096 rather than 256 becauseout += lineis charged its full length on every append, which grows with the square of the line count: a 554 KB report of 5,000 lines costs 0.43 M steps, and about 18,000 lines (2.2 MB) spend the whole default budget. Building in an array and callingjoinis charged once.push,unshift,splice,fillandObject.assignwalk the stored value to reject cycles, andswitchwalks its discriminant and case values to reject interpreter references. These walks are now charged as they go, 1 step per 2 values visited. The store walk also remembers, in aWeakSet, every object it found holding no objects, and stops there next time. Storing an object into one of them drops it from the set, and every store into existing data runs this walk first, so the walk still catches every cycle (20 cycle and sharing checks, below). Round 4: a property or index assignment of the value the slot already holds skips the walk, since it can't change the graph. Round 3 walked the whole group on everyg[k] = g[k] || []and stopped grouping 50k row objects at the budget in about 2 s; it now completes in 0.7–0.9 s, in thereduceform too. Attaching one 50,000-columnmetato each of 20,000 rows went from minutes to 0.36 s; grouping into an object witha = g[k] || []; a.push(v); g[k] = atakes 0.7 s.for…inis charged for its key list. Spread,new Setandnew Mapcharge a string 1 step per 2 characters, because they consume it one character at a time.Map/Setentries, and 1 per 256 string characters. An array of records counts each element as wide as the widest of up to 8 evenly spaced samples, capped at 64. Round 1 used only the first element, capped at 32, and[0, ...wide records]got through.slice,at,push,pop,concatand similar charge only their result.indexOf/includes/lastIndexOfcharge 1 step per 512 elements scannable from their start position (round 1: 4,096).normalizeandlocaleComparecharge 8× their receiver./^(a+)+$/on a 25-character string takes about 40 ms per match, and a loop of 1,000 matches ran 44 s in Node, inside its 60 s wall budget. Codemode has no backtracking limit to set, and refusing nested quantifiers would refuse ordinary patterns.++/--on a property (o.x++), the entriesObject.fromEntriesreads, and non-string arguments of string methods ("a".padEnd(1, big)).console.tableandObject.iswalk too but are already stopped by the charge on their arguments (7.6 s, 4.4 s).metaon each of 20k rows stops at the budget in about 2 s.g[k] = (g[k] || []).concat([v])copies the group every row and is charged for it, so 50k rows in 10 groups stop at the budget in 0.7 s.pushinto the group instead.main. Medians of 7 Node 24 runs,main's patch / round 2 / round 3: 300k-iterations += i % 71,370 / 1,291 / 1,283 ms; comparisons andin884 / 790 / 780 ms;push+Math.max+slice1,667 / 1,710 / 1,713 ms; storeso.x = {v: i, t: [i]}anda[0] = i1,109 / 1,134 / 1,129 ms; string joins'key-' + i + '-' + (i % 7)768 / 703 / 714 ms (+1.6% vs round 2).Review PoCs
Every PoC from round 0 (
/workspace/pr105-poc/), round 1 and round 2 (/workspace/pr105b/) was run against the round 3 patched CodeMode under Bun, with the default 5 M budget unless a budget is named."Stops" means
TimeoutExceededat the step budget. Times are wall time under Bun.Round 3 review findings (round 4 build; before = round 3, e158010)
g[r.k] = g[r.k] || []; g[r.k].push(r), row objects, 50k rows into 10 / 50 groups; 20k into 10(acc[r.k] = acc[r.k] || []).push(r)inreduce, 50k rows, 10 groups`${s}${i}`thencharCodeAt, 1 MB / 4 MBs, 1M times`row ${i}: ${i*2}`), 100k timesg.a = g.atheng.a.push(g),g[0] = g[0],g.a = g.a || [g],(acc.k = acc.k || []).push(acc), replacing with a cycle)hunt:templateLoop(`${s}${s}`.length on 1 MB, 100k times)s + sisEvery other PoC below gives the same outcome on the round 4 build as on round 3.
Round 2 review PoCs (round 3 build; before = round 2, 4bfe724)
realistic2:r.meta = meta(50k columns) on 20k rowsrealistic2: object group-by with reassignment, 500 / 50 / 10 groupsrealistic2: object group-by without reassignment, 10 groupsrealistic2(added):g[k] = (g[k] || []).concat([v]), 10 groupsconcat:t = s + ion 1 MB, thencharCodeAt, 1M timeshunt3:h.x = big,h[0] = big,push/unshift/splice/fillofbig,Object.assign(t, {a: big}), 1M timeshunt3:switch (big)1M timeshunt3: object/array literal holdingbig,Map.set,!big,&&,??hunt3:Object.assigngrowing a 200k-key target;reducegrowing an accumulatorhunt3:big.sort((x, y) => 0)100k timeshunt3:copyWithin100k timeshunt3:push(...big)hunt3:s += "x".repeat(100)1M times;a == bon 4 MB stringssec:inagainst tools, a tool namespace, a tool function, a promise, a search resulttrue, a returned object holding a tool is refuseda.push(a),o.x = o, cycles throughsplice,fill,unshift,Object.assign, index assignment, and indirect cycles after the objects were cached as leavesEarlier PoCs (round 0 and round 1)
Re-run on the round 3 build, all unchanged from round 2 except
concatOpLoop, which now stops (1.42 s). The table keeps round 2's numbers; "before" is round 1's build (4213e74).pr105b/realistic: dedupe within(20k into 20k)pr105b/realistic: dedupe withMap.haspr105b/group: group-by 50k rows, 10 / 50 / 500 groups, at 5 Mpr105b/group: same with an object indexpr105b/g2: group-by, 50 groups, at no limit / 5 M / 20 M / 40 Mpr105b/hunt2:"zz" ina 200k-key objectpr105b/hunt:"zz" ina 1k-key object (LOW)pr105b/hunt:a == "x",String(a), template,"" + aon 1M elements==, template and"" +ran to wallpr105b/hunt:===and<on 4 MB stringspr105b/hunt: rest destructuring of 1M array / 100k-key objectpr105b/hunt:for…in200k keys, thenbreakpr105b/hunt:new Setof a 1M-character stringpr105b/hunt:includeson 1M strings (LOW)pr105b/hunt:JSON.stringify([0, ...20k 200-key records])(LOW)pr105b/hunt:normalizeon 1M characters (LOW)pr105b/hunt: the other 26 amplification loops (repeat,padStart,fill,sort,reverse,new Map,Object.assign, spread,entries,join,localeCompare,Math.max(...a),flat,indexOf, …)pr105b/hunt:templateLoop,concatOpLoop(1 MB string concatenation, ropes)concatOpLoopOK 7.28 s in round 2, stops 1.42 s in round 3+(the review's 12 s run of it is the MED)pr105b/hunt:bigint,typedArray,structuredClone,String.rawpr105b/hunt:/^(a+)+$/and/^(\w+\s?)*$/loops of 1,000 matchespr105b/hunt2:a === a,typeof, ternary, property read,new Set(array)in long loopspr105b/hunt2:new Setof a 1M-character string, 20 times (setStrSmall)pr105b/hunt2:for…of1M array, thenbreak, 100k timespr105b/rerun+ round 0bypass:split("")of 1 MB, 100k timespr105b/rerun:JSON.stringify(all).includes(id), 20k recordspr105b/rerun: 10 amplification loops on 1M dataamp: 10 ops × 5 on 1M dataincludesandindexOfpassescape: 34 escape attempts at 20,000 stepsgetterunsupported anderrorCtorToStringcaught data error, as before; one ReDoS match OK in 0.23 spr105b/wait+ round 0wait: 5 s tool on 50 steps, 10 × 3 s on 200, race and background tool on 100pr105b/wait+ round 0wait: comparatorsortof 20k / 50k at 5 Mpr105b/legit+ round 0legit: budget sweeps (sort 100k,bigJsonTransform, group-by 500, text lines, paginate)env: 13EXECUTOR_MCP_MAX_STEPSvaluesCalibration
Each runaway workload ran under the default 5 M budget until it stopped. Times are wall seconds of one program on shuvdev; in workerd, one request with nothing else running. Many setups build a 1M-element array with
Array.fromand a mapper. That's a million callback calls, and callbacks cost about twice a loop step in workerd, so the setup alone takes up to 7 s there.split("")of 1 MB (round 0 PoC)JSON.stringify20k records +includesJSON.parseof 1.3 MBjoinof 1M elementsObject.keysof 100k keysreplaceAllon 1 MBJSON.stringifyof[0, ...20k 200-key records]includeson 1M stringsnormalizeof 1M characters"zz" ina 200k-key objecta == "x"on 1M elements"" + aon 1M elements===of two 4 MB strings<of two 4 MB strings[x, ...r] =1M elements{k0, ...r} =100k keysfor…in200k keys, thenbreaknew Setof a 1M-character stringOn a quiet host every workload stops within about 7.5 s in Node and Bun. In workerd, everything stops within about 15 s, against 8.5 s for the interpreted loop alone; the ones above 8.5 s spend most of that time building their input with callbacks. Under load these take longer: PR Reviewer saw 8.3–9.6 s in Bun with other programs running, and a loop that sorts a 1M array in place with a comparator takes 10–16 s. They stop at the budget either way; how long that takes is the step rate the host gives them. (This table is from round 2; round 3 adds a charge to string
+only, at most about 1.7k steps per op for these sizes.)Ordinary programs fit. These are the steps each one is charged (smallest passing budget, Bun), with Bun and workerd wall times:
Map.get, 10, 50 or 500 groupsinObject.hasOwnObject.assignone key at a time, 20k times+=(round 3)sortof 50k objectsA comparator
sortof 100k objects needs about 20 M steps (every comparison is interpreted), so the default stops it. That's documented with how to raise the budget throughEXECUTOR_MCP_MAX_STEPS. In workerd, a callback-heavy program near the budget can take up to about 20 s, as the 50k sort shows.Choosing the default
All rates were measured on one host (shuvdev):
Proof
Scenario
mcp-execute-step-budget.spec.ts(self-host) runs on the real self-host server through MCP with the default budget. It passed on the final code:while (true)in nestedtry/catch/finallyand tries toreturnfrom its outer catch. It fails withTimeoutExceedednaming the 5,000,000-step budget, withtoolCalls: [], in 6.5 s (limit: under 60 s; the wall budget is 300 s).s.split("")on a 1 MB string, 100,000 times) fails with the sameTimeoutExceededin 3.3 s (limit: 20 s).a == "x"on a 1M-element array, 100,000 times, fails with the sameTimeoutExceededin 5.1 s (limit: 20 s).in. Merging 20k records into a 20k-record index with!(r.id in byId)completesokwith 10,000 added in 502 ms (limit: 20 s).g.get(k) ?? []/push/setcompletesokin 888 ms for 10 groups and 900 ms for 50.a = g[r.k] || []; a.push(r.v); g[r.k] = aover 50,000 rows completesokin 765 ms (limit: 20 s). On round 2 it took 10.8 s in Node.r.meta = meta(50,000 columns) on each of 20,000 rows completesokin 370 ms (limit: 20 s). On round 2 it ran past a 20 s timeout.g[r.k] = g[r.k] || []; g[r.k].push(r)over 50,000 row objects into 10 groups completesokin 868 ms, and thereduceform in 884 ms (limit: 20 s). On round 3 both stopped at the budget.`${s}${i}`on a 1 MB string, read withcharCodeAt, 1M times, fails withTimeoutExceededat the budget in 1.6 s (limit: 20 s). On round 3 it ran past a 20 s timeout.+runaway.t = s + ion a 1 MB string, read withcharCodeAt, 1M times, fails with the sameTimeoutExceededin 1.5 s (limit: 20 s). On round 2 it ran past a 20 s timeout.JSON.stringify, completesokin 131 ms.okin 570 ms, all three callssuccess.Other runs on the final code:
bun run checkpassed:patches:check(all 24 patches apply), format, lint, typecheck and the rest.bun run e2e:checkpassed.mcp-execute-concurrencyscenarios, and themcp-execute-lazystalled-app timeout scenario (itswhile (true) {}stops at the step budget and still reportsTimeoutExceeded).a.push(a),o.x = o, nested and indirect cycles after the objects were cached as leaves, throughsplice,fill,unshift,Object.assignand index assignment) is still rejected; a sorted leaf, a shared value and a diamond still store.in,==withnull, coercion of arrays holding functions, string comparison, unary operators, templates, rest,for…in,Setfrom strings,Object.assign,Object.hasOwnandMap.getgive identical results. The one exception is"a" in {a: {f: fn}}, which now returnstruewhere it used to throw.Limits and follow-ups
++/--on a property,Object.fromEntriesentries, non-string string-method arguments) aren't charged yet (see residual gaps above). These are for a follow-up PR.+=is charged quadratically: about 18,000 appended lines (2.2 MB) spend the default budget.joinan array instead.EXECUTOR_MCP_MAX_STEPSraises it without a code change.This fork's CI (
fork-ci.yml) runsbun run checkand a fixed list of scenarios. It doesn't read the block below, so the e2e runs above are the proof.e2e:check-descriptionagainst upstream's layout fails on a scenario this PR doesn't touch (cloudOAuthProxyRefusalhas no CI job).