fix(dataflow): resolve each callable under its owning tsconfig - #137
Merged
Conversation
…PLETE) Extraction indexed every callable against the ROOT program alone, so a file a deeper program owns was absent from that index and hit `if (!fn) continue` -- skipped silently, not mis-resolved. On vscode (92 programs, no root tsconfig) that extracted 1,204 of 174,767 callables. This threads BuiltProgram[] through startExtraction so a callable is looked up in the program that owns its file, matching what the symbol table and the L2 call graph already do. Worker tasks are partitioned within a program so each task's files share one tsconfig -- previously every task got a single config, which is also why -j N and -j 1 could diverge on multi-program repos. Adds a coverage line so a near-empty L3 can no longer pass for a successful one. INCOMPLETE -- DO NOT MERGE. Correct on the multi-tsconfig fixture (break-checked: reverting to root-only lookup fails the new test) but it does not fit in memory at vscode scale. Walking a project forces tsc to parse and bind every file in it; previously only the root program was ever walked, so the other 91 stayed lazy. Three runs, all killed by a JSC heap OOM (exit 133): sequential 26.9GB, sequential with one index resident at a time 28.6GB, -j 4 29.2GB. Bounding the index did not help because the memory is in the Projects, not the index. Freeing them needs core.ts to stop running extraction concurrently with the call-graph solve (both hold every program live), or a project pool that materializes a bounded number at a time. See #111.
Two memory ceilings on top of the per-program extraction fix, both from #112. Step 2 -- bound the resident program set. Extraction now runs AFTER the call-graph solve rather than "concurrently" with it. That costs nothing: at the default -j 1, startExtraction evaluated extractSequential eagerly, so the two were already serial and the concurrency comment described only -j N > 1. Ordering them explicitly makes extraction the last reader of every program's Project, which is what lets each one's source files be released as extraction finishes with it. The root project is spared -- finalizeAnalysis still needs it for the config-use dataflow tier. Measured on vscode: peak 29.2GB -> 24.0GB. Step 1 -- stream the JSON emit. `JSON.stringify(application)` materialised the entire output before writing a byte, so peak carried the tree and its serialisation at once and a large enough analysis exceeded the runtime's maximum string length outright (vscode -a 4: `RangeError: Out of memory` in stringify). writeAnalysisJson walks the envelope in insertion order and streams `symbol_table` per module and the application-scope arrays per element, holding at most one of either as a string. Byte-identical output, verified against the previous whole-string path on six fixtures at -a 4 with every graph selector -- the first cut hoisted symbol_table to the front and produced the same bytes in a different key order, which the comparison caught. vscode at -a 4 STILL does not complete: 30.8GB, killed. The earlier run reporting a stringify error at 24.0GB was not "the analysis finished, only emit failed" -- JS threw a catchable RangeError while already near the ceiling, and removing that early exit only let the run continue until the OS killed it. Both changes are real improvements and neither is sufficient there. What they do buy, measured on superset-frontend (39 programs, 1,841 modules): callables with CFG 8,138 (73.4%) -> 11,023 (99.4%), param_in 6,475 -> 17,920, for +0.3GB and no extra wall-clock.
This was referenced Sep 2, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #111. Advances #112 (steps 1 and 2). Against
main, not stacked — no file overlap with the open Neo4j chain (#117/#119/#136).The bug
extractSequentialbuilt its AST index from one project — the root program's — then skippedevery callable it could not find:
ownerProgramassigns files deepest-scope-wins, so a file undersrc/belongs tosrc/'s program,not the root's. Anything a deeper program owned was absent from that index and dropped — silently,
since the
continuealso covers the legitimate bodiless cases.The run reported success either way. Nothing compared callables collected against callables
extracted.
Impact, measured
superset-frontend (39 tsconfigs, 1,841 modules) — before and after on the same repo:
param_in+2,885 callables, +594 modules, 2.8× the SDG param vertices — for 0.3 GB and no extra wall-clock.
Severity scales with how much of the tree the root program does not own: mild on a repo with a
real root tsconfig, near-total on one without. vscode has no root tsconfig at all, which is why it
showed 1,204 of 174,767 (0.7%).
What is in this PR
Three changes that only became visible one after another — each fix exposed the next, and vscode
needs all three:
that owns their file — the same assignment
buildSymbolTableand the L2 call graph already use.Worker tasks partition within a program so each task's files share one tsconfig, which also
fixes a latent
-j Nvs-j 1divergence on multi-program repos.solve rather than "concurrently" — free, because at the default
-j 1startExtractionevaluated
extractSequentialeagerly, so they were already serial. That ordering makesextraction the last reader of each Project, so its source files can be released as it goes. The
root project is spared for
finalizeAnalysis. vscode peak 29.2 GB → 24.0 GB.JSON.stringify(application)built the whole outputbefore writing a byte and exceeded the runtime's maximum string length outright. Now written
element-wise, holding at most one module or one array element as a string.
Also adds a coverage line (
dataflow: extracted N of M callables (P%), warning under 50%) so anear-empty L3 can no longer pass for a successful one.
vscode still does not complete
-a 4on vscode is 30.8 GB, killed. Both memory changes help and neither is sufficient there.Worth stating plainly because I got it wrong mid-investigation: an earlier run reported
RangeError: Out of memoryin stringify at 24.0 GB, and I read that as "the analysis completed,only emit failed". It did not. JS threw a catchable error while already near the ceiling;
removing that early exit just let the run continue until the OS killed it. vscode remains over
budget and #112's remaining steps (incremental reuse, sharding) are what would close it.
Verification
at
-a 4with every graph selector. The first cut hoistedsymbol_tableto the front andproduced identical bytes in a different key order — the comparison caught it.
test/multi-tsconfig.test.tsgains an L3 assertion that callables owned by the nestedweb/program get a CFG, break-checked: reverting to root-only lookup fails it. That fixture existed
but was only exercised at level 2, which is how this shipped.