feat(cli): --program shard selector, --no-repo-sections, and shard IR - #149
Open
rahlk wants to merge 2 commits into
Open
feat(cli): --program shard selector, --no-repo-sections, and shard IR#149rahlk wants to merge 2 commits into
rahlk wants to merge 2 commits into
Conversation
Whole-repository analysis holds every program's ts-morph Project at once. On vscode that is 92 programs, and it is why -a 4 is killed there (exit 133, JSC heap). This adds the shard selector the two-wave L4 design (#112 step 4) is built on. `--program <scope...>` restricts the run; `--list-programs` enumerates the shards so an orchestrator can find them. Programs are named by SCOPE dir, not tsconfig path: a nested tsconfig.json that only `references` others resolves to its LEAF config, so `web/tsconfig.json` becomes a program named `web` whose configPath is `web/src/tsconfig.app.json` -- and two specs can share one leaf config under different scopes, which makes the config path unusable as an identity. Ownership is computed against ALL discovered programs and filtered afterwards. Filtering first would be silently wrong: ownerProgram falls back to the root program, so a selected ancestor would absorb every file its deeper unselected descendants own and compile them under the wrong tsconfig. The test for this is mutation-checked -- reordering the two steps fails it. An unmatched --program is a hard error, not an empty shard: an orchestrator typo must not produce a shard that unions cleanly into a graph missing a third of the repo. Verified on vscode: a `--program src` shard reproduces the standalone run's 6,758 modules and 1,053,108 call-graph edges exactly, and completes -- where the whole-repo run at the same level does not.
…epository The repository-artifact layer is scoped to --input, not to --program, so a shard recomputed the WHOLE repository's inventory: measured on vscode, a `--program src` shard emitted 4,961 artifacts and 4,107 dependency records for a code analysis covering 6,758 modules, against 1,759 and 5 for the same code analysed standalone. That is 5.3GB per shard (39.16GB -> 33.85GB measured) for a result identical in every shard, and a full 92-shard run would also union 92 duplicate copies of it. Repo-scoped sections belong to the repository, not to a shard: one run computes them, every other run passes --no-repo-sections. Also adds `--emit-ir` and the NDJSON shard-IR reader/writer (unit 2 of the design): what a later cross-shard stitch needs, and nothing tsc owns. NDJSON because JSON.stringify on the whole IR would build one multi-hundred-megabyte string -- the emit-time wall #112 lists as ceiling 3. The header pins ir_version, k_limit, --input and --app-name, because a divergence there does not produce a partial union but a silently wrong one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #146. Units 1-2 of the sharded two-wave L4 design.
What this adds
--program <scope...>— restrict a run to named programs;--list-programsenumerates them--no-repo-sections— skip the repo-scoped artifact layer--emit-ir— persist the shard's graph IR as NDJSON for a later cross-shard stitchMeasured on vscode (
-a 4, all graph selectors)--program src--program src --no-repo-sections--input vscode/src(baseline)The shard's code analysis is byte-equivalent to the baseline: 6,758 modules and 1,053,108 call-graph edges, exact match.
Two findings that change the design
Repo-scoped sections were being recomputed per shard. The artifact layer derives from
--input, not from the selected programs, so thesrcshard inventoried all of vscode — 4,961 artifacts and 4,107 dependency records, against 1,759 and 5 for the same code standalone. That is 5.3 GB per shard for an identical result, and a 92-shard run would union 92 duplicate copies. Hence--no-repo-sections: one run computes them, the rest skip. This needs to land in the spec as a design element, not a flag.The ceiling was never a fixed RSS number. Whole-repo L4 dies at 28–31 GB, but this shard reached 33.85 GB and completed. Those deaths are JSC heap exhaustion (exit 133), not RSS pressure — which is why per-shard analysis clears the wall despite a higher peak. Worth correcting wherever the earlier framing implied an RSS limit.
Honest gap
The shard still sits 5.1 GB above the standalone baseline (33.85 vs 28.75 GB) and runs ~30% slower. I ruled out the analysis cache (identical 0.54 GB in both layouts) and have not identified the cause. It does not block the design — every shard completes, which is the property that matters — but it is unexplained, not explained-and-accepted.
Correctness
The load-bearing rule is that selection must not change file→program assignment.
ownerProgramfalls back to the root program, so filtering the spec list before assignment would hand a selected ancestor every file its deeper unselected descendants own, compiling them under the wrong tsconfig. Ownership is computed globally and filtered after; the test for this is mutation-checked — reordering the two steps fails it (and three sibling tests).An unmatched
--programis a hard error rather than an empty shard, so an orchestrator typo cannot produce something that unions cleanly into a graph missing a third of the repository.252 tests pass (7 new), typecheck clean.