Skip to content

fix(dataflow): resolve each callable under its owning tsconfig - #137

Merged
rahlk merged 2 commits into
mainfrom
fix/issue-111-per-program-dataflow
Sep 2, 2026
Merged

fix(dataflow): resolve each callable under its owning tsconfig#137
rahlk merged 2 commits into
mainfrom
fix/issue-111-per-program-dataflow

Conversation

@rahlk

@rahlk rahlk commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Closes #111. Advances #112 (steps 1 and 2). Against main, not stacked — no file overlap with the open Neo4j chain (#117/#119/#136).

The bug

extractSequential built its AST index from one project — the root program's — then skipped
every callable it could not find:

const astIndex = indexCallableDecls(project, opts.input);   // ONE project

const fn = astIndex.get(sig);
if (!fn) continue;                                          // silently skipped

ownerProgram assigns files deepest-scope-wins, so a file under src/ belongs to src/'s program,
not the root's. Anything a deeper program owned was absent from that index and dropped — silently,
since the continue also covers the legitimate bodiless cases.

The run reported success either way. Nothing compared callables collected against callables
extracted.

Impact, measured

superset-frontend (39 tsconfigs, 1,841 modules) — before and after on the same repo:

callables with CFG modules param_in peak
main 8,138 (73.4%) 871 / 1,841 6,475 10.3 GB
this PR 11,023 (99.4%) 1,465 / 1,841 17,920 10.6 GB

+2,885 callables, +594 modules, 2.8× the SDG param vertices — for 0.3 GB and no extra wall-clock.

Severity scales with how much of the tree the root program does not own: mild on a repo with a
real root tsconfig, near-total on one without. vscode has no root tsconfig at all, which is why it
showed 1,204 of 174,767 (0.7%).

What is in this PR

Three changes that only became visible one after another — each fix exposed the next, and vscode
needs all three:

  1. Per-program extraction (L3/L4 populate 0.7% of callables on multi-program repos — dataflow uses one root tsconfig #111). One AST index per program, callables looked up in the program
    that owns their file — the same assignment buildSymbolTable and the L2 call graph already use.
    Worker tasks partition within a program so each task's files share one tsconfig, which also
    fixes a latent -j N vs -j 1 divergence on multi-program repos.
  2. Dispose programs after extraction (Scaling L1-L4 past vscode: three memory ceilings and a sequenced path #112 step 2). Extraction now runs after the call-graph
    solve rather than "concurrently" — free, because at the default -j 1 startExtraction
    evaluated extractSequential eagerly, so they were already serial. That ordering makes
    extraction the last reader of each Project, so its source files can be released as it goes. The
    root project is spared for finalizeAnalysis. vscode peak 29.2 GB → 24.0 GB.
  3. Stream the JSON emit (Scaling L1-L4 past vscode: three memory ceilings and a sequenced path #112 step 1). JSON.stringify(application) built the whole output
    before writing a byte and exceeded the runtime's maximum string length outright. Now written
    element-wise, holding at most one module or one array element as a string.

Also adds a coverage line (dataflow: extracted N of M callables (P%), warning under 50%) so a
near-empty L3 can no longer pass for a successful one.

vscode still does not complete

-a 4 on vscode is 30.8 GB, killed. Both memory changes help and neither is sufficient there.

Worth stating plainly because I got it wrong mid-investigation: an earlier run reported
RangeError: Out of memory in stringify at 24.0 GB, and I read that as "the analysis completed,
only emit failed". It did not. JS threw a catchable error while already near the ceiling;
removing that early exit just let the run continue until the OS killed it. vscode remains over
budget and #112's remaining steps (incremental reuse, sharding) are what would close it.

Verification

  • superset-frontend before/after above, same machine, same flags.
  • Streaming emit is byte-identical to the previous whole-string write, checked on six fixtures
    at -a 4 with every graph selector. The first cut hoisted symbol_table to the front and
    produced identical bytes in a different key order — the comparison caught it.
  • test/multi-tsconfig.test.ts gains an L3 assertion that callables owned by the nested web/
    program get a CFG, break-checked: reverting to root-only lookup fails it. That fixture existed
    but was only exercised at level 2, which is how this shipped.
  • Suite: 244 pass, 0 fail.

…PLETE)

Extraction indexed every callable against the ROOT program alone, so a file a
deeper program owns was absent from that index and hit `if (!fn) continue` --
skipped silently, not mis-resolved. On vscode (92 programs, no root tsconfig)
that extracted 1,204 of 174,767 callables.

This threads BuiltProgram[] through startExtraction so a callable is looked up
in the program that owns its file, matching what the symbol table and the L2
call graph already do. Worker tasks are partitioned within a program so each
task's files share one tsconfig -- previously every task got a single config,
which is also why -j N and -j 1 could diverge on multi-program repos. Adds a
coverage line so a near-empty L3 can no longer pass for a successful one.

INCOMPLETE -- DO NOT MERGE. Correct on the multi-tsconfig fixture (break-checked:
reverting to root-only lookup fails the new test) but it does not fit in memory
at vscode scale. Walking a project forces tsc to parse and bind every file in
it; previously only the root program was ever walked, so the other 91 stayed
lazy. Three runs, all killed by a JSC heap OOM (exit 133): sequential 26.9GB,
sequential with one index resident at a time 28.6GB, -j 4 29.2GB. Bounding the
index did not help because the memory is in the Projects, not the index.

Freeing them needs core.ts to stop running extraction concurrently with the
call-graph solve (both hold every program live), or a project pool that
materializes a bounded number at a time. See #111.
Two memory ceilings on top of the per-program extraction fix, both from #112.

Step 2 -- bound the resident program set. Extraction now runs AFTER the
call-graph solve rather than "concurrently" with it. That costs nothing: at the
default -j 1, startExtraction evaluated extractSequential eagerly, so the two
were already serial and the concurrency comment described only -j N > 1.
Ordering them explicitly makes extraction the last reader of every program's
Project, which is what lets each one's source files be released as extraction
finishes with it. The root project is spared -- finalizeAnalysis still needs it
for the config-use dataflow tier. Measured on vscode: peak 29.2GB -> 24.0GB.

Step 1 -- stream the JSON emit. `JSON.stringify(application)` materialised the
entire output before writing a byte, so peak carried the tree and its
serialisation at once and a large enough analysis exceeded the runtime's
maximum string length outright (vscode -a 4: `RangeError: Out of memory` in
stringify). writeAnalysisJson walks the envelope in insertion order and streams
`symbol_table` per module and the application-scope arrays per element, holding
at most one of either as a string.

Byte-identical output, verified against the previous whole-string path on six
fixtures at -a 4 with every graph selector -- the first cut hoisted
symbol_table to the front and produced the same bytes in a different key order,
which the comparison caught.

vscode at -a 4 STILL does not complete: 30.8GB, killed. The earlier run
reporting a stringify error at 24.0GB was not "the analysis finished, only emit
failed" -- JS threw a catchable RangeError while already near the ceiling, and
removing that early exit only let the run continue until the OS killed it.
Both changes are real improvements and neither is sufficient there.

What they do buy, measured on superset-frontend (39 programs, 1,841 modules):
callables with CFG 8,138 (73.4%) -> 11,023 (99.4%), param_in 6,475 -> 17,920,
for +0.3GB and no extra wall-clock.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

L3/L4 populate 0.7% of callables on multi-program repos — dataflow uses one root tsconfig

1 participant