Skip to content

feat(callgraph)!: replace Jelly with the defuse linker — tsc + per-callable tiers - #99

Closed
rahlk wants to merge 15 commits into
mainfrom
feat/issue-098-defuse-linker
Closed

feat(callgraph)!: replace Jelly with the defuse linker — tsc + per-callable tiers#99
rahlk wants to merge 15 commits into
mainfrom
feat/issue-098-defuse-linker

Conversation

@rahlk

@rahlk rahlk commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Closes #98. Stacked on #97 (native v2 model). Design record: docs/design/specs/defuse-linker-call-graph.md · audited gate: docs/design/specs/defuse-linker-joern-ledger.md.

What

Jelly is removed wholesale — the whole-program flow analysis (the same cost class as python's removed PyCG), the __jelly multi-call binary mode, CANTS_SELF_JELLY, the @cs-au-dk/jelly dependency and its patch, and BREAKING: the released --call-graph-provider and --tsc-only flags (one code path, no backend flag — python 1.2.0 precedent). Bundle: 998 → 455 modules.

In its place, the defuse linker (src/semantic_analysis/defuseLinker.ts): a deterministic per-callable pass over the tsc resolver's leftovers, python's Jedi+defuse architecture instantiated for TS. Tiers, in order: T1 local value chase (bounded alias hops) · T2 decorator invocations (method/accessor/param owners; class/property decorators attribute to the module) · T3 external-callback rule (function value passed to an external/unresolved callee ⇒ enclosing→lambda — the documented divergence from python; JS is callback-central) · T4 bounded interprocedural votes (param-invoking sites, factory returns; two rounds, no fixpoint) · T5 CHA-by-name fallback (bounded fan, skip-not-truncate). Edges carry prov: ["defuse"]; resolutions reach the L1 call nodes out-of-band and are never persisted into callee_signature (cache provenance rule). Module-scope execution is now a call-graph SOURCE (python #131 parity): top-level calls, class decorators, and the top-level express idiom attribute to the MODULE, id-homed onto the module node.

The validation drove six real analyzer fixes

Iterating the Joern ledger surfaced L1/L2 gaps Jelly had been masking, each now regression-tested:

  1. Concise-arrow bodies that ARE the call (u => u.describe()) recorded no call site
  2. Module-scope calls had no caller at all
  3. Tagged template calls (inline`url(...)`) were invisible end to end
  4. Parameter-default initializer calls (f(sel, style = mkSheet())) were never walked
  5. JS sources were never discovered (SOURCE_EXTS was ts-only — vscode's vendored marked.js was analyzed through its bodiless .d.ts); .js/.jsx/.mjs/.cjs are first-class now with two-way compiled-sibling/declaration-sibling rules
  6. T4c, the ctor-field callback chainthis.field(...) through parameter properties with one bounded parameter hop; vscode's migrateOptions registration pattern closes end-to-end (apply → callback → apply.write), no fixpoint

Gates

  • Corpus superset (enforced): residual 0 — 74/74 Joern jssrc2cpg real pairs across sample-app, dataflow-app, anon-app, nestjs-realworld @ c1c2cc4 (ours 126 internal edges vs Joern's 30 there)
  • vscode scale audit (microsoft/vscode @ a3c9dc6, 8,735 files / 1.15M LOC): 99.66% of 55,074 single-candidate Joern real pairs covered after seven fix rounds; the 189 residual is classified per family in the ledger (Registry-pattern generics, Promise-executor name shadows, the static/instance signature-collision schema finding, escaped closure-locals = python docs(spec): entrypoint detection for typescript, matching python #150's staged tier, accessor tails)
  • Scale: cants L2 5m52s / 24.4GB for 1,024,232 edges (single-threaded) vs Joern parse 9m06s / 30.3GB on 10 cores; the engines this replaces DNF'd at this class (PyCG 3h19m, Fraunhofer OOM at 44GB)
  • Jelly recovery: 6/6 jelly-only fixture edges (3 decorator, 1 callback, 2 upgraded to typed tsc resolutions)
  • A/B determinism: byte-identical paired runs on every corpus app and the vscode edge dump
  • Full suite (136 pass), typecheck, Neo4j conformance + schema.neo4j.json regen byte-stable, binary builds

Follow-ups recorded (not in this PR)

Base automatically changed from refactor/issue-096-native-v2-model to main August 27, 2026 13:27
rahlk added 15 commits August 27, 2026 09:27
BREAKING: the released --call-graph-provider and --tsc-only flags are
removed (commander now rejects them); the tsc resolver is the one base
call graph, per the defuse-linker design
(docs/design/specs/defuse-linker-call-graph.md). Gone with them: the
union/jelly providers and selector, the __jelly multi-call binary mode
and CANTS_SELF_JELLY re-exec, the @cs-au-dk/jelly dependency, its patch,
and the --external @babel/preset-typescript bundling workaround (bundle:
998 → 455 modules). mergeCallGraphs stays — the defuse linker overlays
the tsc base through it. python-sdk's tsc_only kwarg keeps passing
--tsc-only until its tracked follow-up lands (spec, release plan).

Full suite + typecheck green; binary builds.
…#98)

walkBody only visited the body's CHILDREN unless the body was itself a
callable boundary, so `u => u.describe()` never recorded u.describe()
as a call site — an L1 gap the Jelly leg papered over with its own
approximated edge. Visiting the body node itself closes it: the site
lands in the anon's call_sites, the tsc resolver types it precisely
(users: User[] ⇒ u: User), and the body{} call node appears at L1.
Additive wire change (a real call site that was missing).
…base (#98)

The local pass that backfills what the resolver missed
(docs/design/specs/defuse-linker-call-graph.md; python's Jedi+defuse
architecture). Per-callable and bounded-round only, sorted iteration —
deterministic by construction, no whole-program fixpoint:

  T1 local value chase (alias chains via bounded symbol hops)
  T2 decorator invocations (method/accessor/parameter owners; edge-only)
  T3 external-callback rule (function value passed to an external or
     unresolved callee ⇒ enclosing→lambda; edge-only — the documented
     divergence from python)
  T4 bounded interprocedural votes (param-invoking sites resolve to the
     functions passed by resolved-internal callers, two rounds; factory
     returns through a unique returned function)
  T5 CHA-by-name fallback (bounded fan, skip-not-truncate over the cap)

Edges carry prov ["defuse"] and overlay the tsc base via
mergeCallGraphs; body-node resolutions return out-of-band and are
applied by backfillCallees — never persisted into callee_signature
(cache provenance rule).

Jelly-recovery on the fixtures: 6/6 jelly-only edges (3 decorator,
1 callback via T3, 2 upgraded to typed tsc resolutions by the
concise-arrow fix); edge counts equal the old union exactly
(57 / 22 / 2). Full suite + typecheck green.
…ual 0 (#98)

The reference-validation harness (scripts/joern: cpg call dump + the
signature-mapping comparator) and the audited exception ledger
(docs/design/specs/defuse-linker-joern-ledger.md). Corpus: the three
repo fixtures + nestjs-realworld-example-app @ c1c2cc4. Every Joern
real pair is covered; exclusions are classified per family (their
lowering synthetics, parameters-as-callees stubs, super-to-interface
fabrications, decorator-attribution variants) — python's ledger
discipline. A/B paired runs byte-identical on all four apps.
`inline\`url(...)\`` never recorded a call site — walkBody, the callee
resolver, and the call-expression index all matched Call/New only, while
the L3 exception model already treated tagged templates as calls. Found
by the vscode Joern ledger (432-strong same-file residual family).
Additive wire change: real call nodes that were missing.
…lable (#98)

`createCSSRule(sel, style = getSharedStyleSheet())` executes the default
in the callee's activation, but walkBody only covered getBody() — the
call (and any nested arrow) was invisible. Found by the vscode ledger
(domStylesheets family). Additive wire change + regression test.
…ied (#98)

Corpus gate holds at residual 0 (74/74). vscode @ a3c9dc6 (8,735 files,
1.15M LOC): 54,703/54,947 single-candidate Joern real pairs covered
after four ledger-driven fix rounds; the 244 residual is classified
(escaped closure-locals — python #150's staged tier, vendored marked,
the static/instance signature-collision finding, accessor tails).
Scale table: cants L2 5m52s/24.4GB for 1.02M edges vs Joern parse
9m06s/30.3GB. Comparator gained fan-row and misresolution-variant
rules (their multi-candidate rows are enumeration, not resolution).
…ain (#98)

Two residual families from the vscode ledger:

- .js/.jsx/.mjs/.cjs sources were never DISCOVERED (SOURCE_EXTS was
  ts-only) — vscode's vendored marked.js was analyzed through its
  bodiless .d.ts, so every internal edge was missing. JS files are now
  first-class, with a compiled-sibling guard (a .js next to its same-
  prefix .ts is build output, skipped — the compiler's own allowJs
  duplicate rule).
- T4c: `this.field(...)` where the field's value arrives through the
  constructor (parameter property or this.f = param) resolves to the
  function values passed at the class's `new` sites — with ONE bounded
  parameter hop (register(key, cb) → new Migration(key, cb)) and
  module-scope resolved calls now feeding the vote rounds. The
  registered callbacks' own param-invoking sites then resolve in the
  existing T4 rounds: the full migrateOptions chain
  (apply → callback → apply.write) lands, no fixpoint anywhere.
#98)

The compiled-sibling guard registered marked.d.ts's prefix and skipped
marked.js — inverted for the hand-written-declarations pattern. Sibling
rules are now two-way: .js beside a REAL .ts source is build output
(skipped); .d.ts beside an analyzed .js is its declaration file
(skipped as a module; the checker still reads it from disk).
@rahlk

rahlk commented Aug 31, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #103, which is now based on main.

These three branches were never a stack in history. feat/issue-101-artifacts and this branch share a merge-base of f49b3f8 — the #96 commit at the bottom — so the artifact branch forked there and re-developed this work in parallel rather than on top of it. The PR bases declared a stack that git never had.

Content-wise, feat/issue-101-artifacts is a superset:

  • It merges into main with zero conflicts (+7530/-936 across 90 files).
  • Against this branch it is +5939/-40, and every one of those 40 deleted lines is a replacement, not a loss — it carries the streaming PrintWriter in scripts/joern/dump-calls.sc (the JVM 2GB array-cap fix) and structuredClone in src/schema/emit.ts (the vscode L4 OOM fix), where this branch still holds the older StringBuilder and JSON.parse(JSON.stringify(...)).
  • src/semantic_analysis/defuseLinker.ts and docs/design/specs/defuse-linker-joern-ledger.md are both present there.

Nothing here is lost by closing. The work lands in #103, which now has a passing CI run — the first any branch in this group has ever had, since the old pull_request: branches: [main] filter exempted every non-main-based PR (fixed in #106).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace Jelly with a per-callable defuse linker (tsc + defuse call graph)

1 participant