Skip to content

Keep a search's order for its later pages, so walking the catalog is no longer quadratic - #106

Merged
shuv1337 merged 2 commits into
v2from
fix/pr104-catalog-listing-quadratic
Nov 19, 2026
Merged

shuv1337 merged 2 commits into
v2from
fix/pr104-catalog-listing-quadratic

Conversation

@shuv1337

@shuv1337 shuv1337 commented Nov 19, 2026 •

Copy link
Copy Markdown
Owner

Summary

Paging tools.search re-ranked every row its targets hold on every page (tool-search-store.ts page()). So walking a catalog of n tools in pages of k cost n²/k on the product's single thread. On Nov 18 (3:10–3:24 PM PT), three concurrent walks with neither a query nor a namespace, over 11,467 tools, ran at 0.8–5 s per page. That held the self-hosted product isolate at ~90–100% CPU in bursts.

The fix keeps the paging contract (offset, limit, remaining, next and totals are unchanged) and makes a walk cost one ranking plus n row reads:

  • First page (offset 0): unchanged. It still ranks and reads in one statement; most searches end here.
  • On a miss: the first page past offset 0, or any page whose kept order is gone or stale, runs one statement. That statement ranks, reads the page, and returns the full order plus a stamp of each ranked listing's digest. There's no second read, and the page comes straight from that ranking.
  • Later pages: they read only their own rows by id. The same statement re-reads the digest stamp; if any listing was written, added or removed since, the order is dropped and the page re-ranks as on a miss. A page never mixes two rankings.
  • Compact storage: each item is 3 Int32 cells: its target, the position of its first row in that target's listing (row ids are listing:position), and the end of its other merged targets in a separate Int32 array. That's 12 B per item, about 140 KB per order at 11,467 tools and 720 KB at 60,000.
  • Bounds:
    • at most 32 kept searches and 6 MB in total, least recently used evicted first;
    • at most 2 MB per order, about 160k tools. A larger order isn't kept, and its pages rank as before.
    • The byte count includes the key and stamp.
  • No thrash: a new order may evict a kept search only if that search has been unread for 30 s. While every kept search is in use, a new search's pages rank exactly as before. So more concurrent walks than the cap cost what they did before the cache existed, instead of evicting each other on every page.
  • Cache key: targets (listing, app, account digest, labels, scopes, below) plus terms. Callers with different scopes never share an order.
  • Span attribute: sdk.tools.search spans carry executor.search.order, which is ranked, ordered or kept.

I considered capping listings that have no query at 200 items and dropped the idea: it would break the documented "page through a large catalog" contract, and it doesn't help a single large namespace.

Verification

All runs were on a Linux x64 dev host with Node 24.22.2 and Bun.

  • bun run check: passes (format, lint, typecheck of 9 projects, protocols/telemetry/API-errors checks, no-unit-tests, boundary).

  • New scenario mcp-search-walk.spec.ts: "An MCP search without a query pages through 8,000 tools ranking them once, each later page reading only its own rows". It deploys 16 apps × 500 tools and walks the whole catalog with tools.search({ limit: 150 }) plus next, twice, in one execute each (55 pages).

    • It asserts that every tool is listed once.

    • It asserts the order sequence: first walk ranked, ordered, kept…; second walk ranked, kept….

    • It asserts that the median kept page's SQL time is under half the first page's ranking SQL.

    • Before and after, from the same scenario (SQL time summed from sql.execute spans under each sdk.tools.search; "after" is head 08fabf9):

      before (f482d19) after
      SQL, whole walk (55 pages), repeat walk 1,848 ms 165 ms
      SQL per later page ~34 ms (max 40) ~2 ms (max 3)
      execute wall time, repeat walk 2,385 ms 477 ms
      execute wall time, first walk (includes evaluating 16 listings) 4,644 ms 2,975 ms

      The before run fails the new scenario's assertions; the after run passes.

  • Memory, with PR Reviewer's PoC on the real store and PGlite, 16 listings. "Held" is the memory freed when the store and its kept orders are dropped, as heap plus external, since Int32Array bytes are external on V8.

    • Node 24 (V8):

      tools orders kept held per order
      11,467 8 1.13 MB 0.14 MB
      11,467 32 3.95 MB 0.12 MB
      60,000 8 4.42 MB 0.55 MB
      60,000 32 4.40 MB byte cap holds it at about 8 orders

      Before this commit: 5.39 MB per order and 43 MB for 8 at 11,467 tools; 28 MB per order and 224 MB for 8 at 60,000.

    • Bun: process.memoryUsage() didn't show the freed memory after dropping the store, so I report growth while the orders were kept, after warm-up:

      tools 8 orders 32 orders
      11,467 2.5 MB 7.9 MB
      60,000 8.4 MB 8.8 MB

      Before this commit: 12.5 MB per order and 100 MB for 8 at 11,467 tools.

    • The reviewer's original mem.ts, which reads heapUsed with no warm-up, reports 0.05 MB per order on Node, because it doesn't count the external typed arrays. On Bun it reports a flat 80.5 MB at both 11,467 and 60,000 tools. I read that as first-use warm-up of the ranking-with-order statement: it doesn't grow with the catalog, and with a warm-up it doesn't show.

  • Thrash (PR Reviewer's thrash.ts: 8,000 tools, walkers in lockstep, 6 pages each, ms per page):

    walkers base before this commit now
    8 ~30 15.5 8.5
    9 ~30 57.3 8.8
    16 ~30 58.4 8.0
    32 ~30 8.2
    40, past the 32 cap 32.8 12.5

    At 40 walkers, the 8 walkers the cache has no room for rank each page as before, and nothing is evicted while in use.

  • Correctness, PR Reviewer's PoCs on this head:

    • fuzz against the pre-PR store over seeds 1–5: 1,787 page comparisons, 0 mismatches, with the kept path exercised in every seed;
    • concurrency: 406 checks, 0 mismatches;
    • edge cases: same results as the pre-PR store.
  • Self-host scenarios: on f5995c3, one combined run of 48 scenarios, 43 passed. The 5 failures were the 4 mcp-catalog-scale scenarios described below and the first draft of mcp-search-walk. Passing files: mcp-search-walk, mcp-search-statements, search-statements, mcp-search-concise, mcp-catalog, catalog-agent-search, mcp-execute-reach, mcp-execute-lazy, mcp-execute-concurrency, mcp-execute-failures, and Cold MCP discovery (8,549 OpenAPI tools).

    • The mcp-search-walk draft failed in this combined run on an assertion that summed SQL time across pages, which proved load-sensitive. I replaced it with the median-page check; the new scenario then passed alone twice (11.5 s, 10.5 s).
  • Local: MCP search resolves namespace-relative dotted paths and A broad MCP tool search returns concise pages pass (2/2), as in fork-ci.

  • mcp-catalog-scale.spec.ts on self-host: 3/7 pass. The same 4 fail identically on unpatched f482d19 on this host (TraceUnsettled on "over 7,000 tools" and "beside stalled apps", plus "still aged" and "not yet reported" on the two aged-rows scenarios). They are pre-existing and environmental, and not caused by this change: those scenarios only search page 0, whose statement is unchanged.

  • Not run: the Cloud target and the packaged workerd Docker image. fork-ci doesn't run the new scenario, and I made no workflow edits.

  • bun run e2e:check-description fails on this repo before reaching this PR's block. It reports scheduled scenario cloudOAuthProxyRefusal with no CI job, which is unrelated and pre-existing; fork-ci has no select job.

  • Re-run on 08fabf9:

    • bun run check passes;
    • the new walk scenario passes;
    • mcp-search-statements, search-statements, mcp-execute-lazy and mcp-catalog pass (11/11 self-host);
    • the two local MCP search scenarios pass (2/2).

Behavior

No API or contract change. The only new state is up to 32 kept search orders in the store's memory, together at most 6 MB. A miss costs one statement, the ranking plus that order, with no second read.

Deployment

Not deployed. Shipping it to prod needs a product restart, which needs Lauren's sign-off.

mcp-search-walk.spec.ts
mcp-search-statements.spec.ts
search-statements.spec.ts
mcp-search-concise.spec.ts
mcp-catalog.spec.ts
mcp-catalog-scale.spec.ts
mcp-execute-lazy.spec.ts
mcp-execute-failures.spec.ts

Executor Owner (Grok Bot) added 2 commits November 18, 2026 17:14
Paging tools.search ranked every row its targets hold again for each page
(tool-search-store page()), so walking a catalog of n tools in pages of k
cost n²/k on the product's one thread. On Nov 18 three walks with neither
query nor namespace over 11,467 tools held a self-hosted product at full
CPU for minutes.

A page past the first now ranks once, keeps the order (eight searches,
least recently used dropped) and reads later pages' rows by id, in a
statement that also checks every ranked listing still has its digest; a
listing written since is ranked again in one statement, as before. The
first page, which most searches end at, is unchanged. Paging, offsets and
totals keep their contract.

mcp-search-walk walks 8,000 tools twice: SQL across 55 pages falls from
1,848 ms to 153 ms, and a later page from ~34 ms to ~3 ms.
A kept order held each item as a [row id, target, ordinals] array: about
5 MB per order at 11,467 tools on V8 and 28 MB at 60,000, eight of them
with no byte cap. An item is now three Int32 cells (target, row position
in its listing, end of its other merged targets), about 140 KB per order
at 11,467 tools. Kept orders are bounded at 32 searches and 6 MB together,
2 MB each; a larger order is not kept and its pages rank as before.

A miss reads its page from the same statement that ranks, so it no longer
makes a second read. A new order evicts only a search unread for 30 s:
while every kept search is in use, a new one's pages rank as before, so
more concurrent walks than the cap cost what they did before, instead of
evicting each other on every page.
@shuv1337
shuv1337 merged commit bce0520 into v2 Nov 19, 2026
4 checks passed
shuv1337 added a commit that referenced this pull request Nov 19, 2026
…search in use (#107)

* Stop the store walk at any value it has walked, not only leaves

A shared value of many small records, such as a meta object of 1,000
records attached to every row, was walked record by record on every
store: about 500 steps each, 10 million for 20,000 rows, so the program
stopped at the 5,000,000-step budget.

The store walk now remembers every object whose whole graph it checked.
A leaf stays remembered until an object is stored into it; any other
object is remembered for the current epoch, and storing an object into a
remembered object that another remembered object holds starts a new
epoch. A differential fuzz of 3.9 million stores against a fresh
reachability check agrees on all of them and fails at once when the
epoch is not advanced.

mcp-execute-step-budget: 20,000 rows sharing a 1,000-record meta now
complete in 283 ms (TimeoutExceeded at 1,317 ms before); storing a row
into that shared value is still rejected as a cycle; runaways still stop
with TimeoutExceeded.

* Never evict a search still being paged, and keep oversized searches apart

Kept search orders (#106) had three gaps:
- room was checked before the ranking await and eviction after it was
  strict LRU, so new searches ranking at once evicted searches still in
  use; an order is now admitted after the await, evicting only searches
  unread for 30 s, or not kept;
- idle time read the wall clock; it now reads the monotonic clock;
- a search too large to keep was remembered in the same cache with its
  key's bytes, and a key past 6 MB evicted every kept search; such
  searches are now remembered in their own bounded map, and a key too
  large to keep is never remembered.

mcp-search-kept-orders covers the first and third: both assertions fail
on the previous store and pass on this one.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant