Skip to content

Commit 2f88349

Browse files
authored
feat(l2): call graph — declared + RTA edges, external symbols, callee backfill (#181) (#192)
* feat(schema): oracle for L2 call_graph edges and external_symbols * feat(l1): synthesize record accessors and enum values()/valueOf * feat(l2): declaring-type hint plumbing for callee backfill Record the callee's declaring-type binary name per resolved call site so L2's tree walk can map it to a callee id without keeping ASTs alive. The binary name comes from a structural containerType() walk (BinaryNames), the spelling WALA emits and the L2 index will key on. Anonymous creations are left unhinted so L2 can bind them to the anon's own constructor by node identity rather than fabricating a super/interface endpoint. The hint is @cacheonly: a V2Json.cache() Gson keeps it while the payload writers exclude it in both directions, and both L1Cache paths switch to it so a warm-cache run does not silently drop the backfill. No payload change. * feat(l2): declared call_graph, callee backfill, and external_symbols L2CallGraph is a pure walk over the emitted L1 tree: it derives a binary-name type index from the tree, backfills each resolved call node's callee, accumulates one declared edge per (src, dst) with weight = call-site count, and homes out-of-project targets as external_symbols so no edge dangles. Self-edges are kept. Anonymous creations stay unhinted (no fabricated endpoint); their node-identity edges are a follow-up. -a 2 now emits under --schema v2: the level gate lifts to 2 and analyzeV2 runs the pass after saving the L1 cache, since callee is an L2 refinement that must not persist into an L1 artifact. declared edges need only the dependency jars, never a build. Adds JCallEdge/JExternalSymbol models, call_graph/external_symbols on JApplication, and CanId.externalId. The call-graph-test fixture gains a second package (Greeter) for a cross-package edge and a JDK call for external homing. L2CallGraphGateTest covers every definition-of-done item except rta: no dangling endpoints, non-empty prov, named and cross-package edges, weight equals backfilled-site count, L1 subset of L2, determinism, no in-project type homed as external, and schema conformance. * feat(l2): anonymous-class node-identity edges (resolution case 1) Complete the callee backfill for anonymous creations. L1 threads each anon creation's own generated-constructor can-id through the builders: CallableBuilder computes a body's local classes first, keyed to the creation by AST node identity, and hands that map to CallSiteBuilder, which records the constructor id as the site's declaring-type hint — the discriminated-union hint L2 dispatches on the can:// prefix. L2 uses it directly (case 1), checking membership so the endpoint never dangles. This points new Runnable(){} at the anonymous class's own <init>() rather than fabricating java.lang.Runnable.<init>() — an interface has no constructor. L2CallGraphTest covers the edge, the avoided fabrication, and that self-edges are kept. * feat(l2): RTA overlay — prov set-union over declared and WALA rta edges RtaCallGraph builds WALA's RTA call graph (mirroring the v1 scope/CHA/RTA construction) and reduces it to binary-name + erased-signature endpoint pairs; L2CallGraph joins them through the same tree-derived index the declared pass uses. One edge per (src, dst): prov is the set-union of the attesting analyses (sorted), weight is the declared count when declared attests it, else the RTA count. The overlay adds external symbols but never in-project nodes: an in-project WALA endpoint absent from the tree (a bridge/access$/lambda$ synthetic, or an $anon$N-vs-Outer$1 identity-join failure) is dropped via the same membership check, so no-dangling stays structural. A library target is homed external; the edge is kept. -a 2 attempts the build to feed RTA; --no-rta or a build failure degrades to declared-only rather than failing the level. RtaCallGraph defaults the CodeAnalyzer.projectRootPom global the build machinery reads, so a direct caller need not know that coupling. Tests: RtaCallGraphTest pins the descriptor converters; L2CallGraphTest covers the join (prov union, rta-only, external homing, synthetic/library drops) against synthetic endpoints; a realworld-tagged test asserts a real application yields at least one rta-attested edge. * docs(notes): L2 v1-vs-v2 call-graph comparison Adds the L2 companion to the L1 comparison note, generated from a full v1/v2 -a 2 run over the ten real-world fixtures. Both emitters run the same WALA RTA over the same bytecode, so the note reconciles v1's WALA- only, application-target-only graph against v2's declared+rta overlay: v1's edges match v2's rta in-project edges; v2 adds the declared (JavaParser) analysis and homes the external/library targets v1 drops. Records the app-dependent value of RTA (negligible on commons-lang, dominant dispatch fan-out on daytrader8), the 164 fabricated <<implicit>> callables v2 no longer emits, the callee backfill, and that v1 -a 2 failed on two apps where v2 degraded to declared-only. * docs(design): L2 spec amendments and ledger entries D17-D23 Amend schema-v2-l3-l4-design.md for the inverted producer posture: the L2 level table and precision-posture note name both analyses and their prov tags (JavaParser declares, WALA RTA attests + extends), and the L4 points-to note records that its dependency is on the rta overlay. Add SCHEMA_DECISIONS D17-D23: one edge per (src,dst) with prov set-union and kept self-edges; JavaParser-declares/WALA-attests production; the binary-name @external symbol map; true recursive erasure; implicit constructor synthesis; the cache-only declaring-type hint plumbing; and implicit record-accessor / enum values()/valueOf synthesis. * fix(wala): exclude dependency jars that shadow the project's own classes A project that depends on a released copy of itself (a common benchmark setup — e.g. commons-lang's test-scoped commons-lang3) had its own compiled classes duplicated in a dependency jar. WALA binds a class to the loader that defines it, so those duplicates loaded under the Extension (library) loader and were excluded from isApplicationClass — dropping them from entrypoints, the call graph, and IR construction. On commons-lang this left only 14 of 435 application classes in the hierarchy (73 entrypoints), so WALA's call graph covered ~3% of the project. ScopeUtils now builds the application classes first and skips any dependency jar that redefines one, keeping the project's own bytecode authoritative. commons-lang recovers all 435 application classes (5,141 entrypoints) and its RTA app->app edges rise from 37 to 6,944. This is shared scope construction, so v1's call graph and future L3/L4 (built on the same class hierarchy) benefit identically. RtaCallGraph now logs the application-class count so a future shortfall surfaces rather than silently thinning the graph. * feat(l2): --external-calls opt-in for external edges (default off, v1 parity) v1's L2 kept only application-target edges; default v2 -a 2 now matches by gating external_symbols and edges to out-of-project targets behind --external-calls (off by default). When off, a call resolving outside the project is dropped like an unresolved one (no callee, no edge, no external symbol); when on, targets are homed so no edge dangles. The gate is threaded through both the declared pass and the rta join. V2Emitter now omits an empty call_graph/external_symbols rather than emitting []/{} , so parity is a missing key (absence = no fact), not an empty one. The L2CallGraph.build library default keeps external on (it is intrinsic to L2); only the CLI defaults it off. Ledger D24. * docs(notes): regenerate L2 comparison after the WALA scope fix Re-run against the fixed jar and both --external-calls modes. Leads with the scope bug this comparison surfaced (test-scoped self-dependency shadowing 421 of commons-lang's 435 classes into WALA's Extension loader) and its fix, which lifts commons-lang from 21 to 16,183 v1 edges and restores WALA coverage for v1, v2, and future L3/L4 alike. Totals now show v2 in both modes: default (external off, v1 parity) and --external-calls (external targets homed). Concrete post-fix example edges included. * fix(l2): address pre-merge code-review findings - ScopeUtils: exclude module-info/package-info from the dependency-shadow check — every modular jar carries its own module-info, so counting it as an application class excluded every modular dependency from WALA scope; and skip a class ASM cannot read rather than aborting the whole scope. - RtaCallGraph: mute stdout/stderr across the entire WALA phase (scope, hierarchy, entrypoints, call graph), not just call-graph construction, so --stdout mode's JSON data channel is never corrupted; log the class-count diagnostic after restoring the streams. - L2CallGraph: clear a call node's callee when a run resolves it to nothing, so build() is idempotent on a reused tree and no stale @external callee dangles across a re-run with different --external-calls. - Tests: null-safe external_symbols access in the gate; cover the clear-on-reuse path and that unresolved local-class calls are never homed external.
1 parent b3b46bd commit 2f88349

33 files changed

Lines changed: 2420 additions & 52 deletions

.claude/SCHEMA_DECISIONS.md

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -287,6 +287,89 @@ this is a new fact, not a reshaping of an existing one.
287287
Existing convention (`J_CALLS`, …); dual-label `JSymbol` merge pattern retained.
288288
`SchemaCatalog` takes a major bump (families rename v1→v2).
289289

290+
### D17 — L2 call edges: one edge per `(src, dst)`, `prov` set-union, self-edges kept
291+
The `call_graph` carries one edge per ordered `(src, dst)` callable pair, not one per call site:
292+
`weight` counts the sites behind it and `prov` is the set-union of the analyses that attest it
293+
(`["declared"]`, `["rta"]`, or both), sorted alphabetically. Where `declared` and `rta` disagree the
294+
`dst` differs, so they are different edges — not parallel edge sets. **Self-edges are kept**: direct
295+
recursion is a real edge; v1 dropped it via a `!source.equals(target)` guard. When both analyses attest
296+
an edge the `declared` count wins, because those are the sites a consumer can navigate to.
297+
298+
### D18 — Call-graph production: JavaParser declares, WALA RTA attests (inverts spec line 103)
299+
The earlier spec made WALA the sole call-graph producer. L2 inverts this: JavaParser's symbol solver
300+
produces the *declared* graph — every edge to a resolved target, needing only the dependency jars, no
301+
build — and WALA RTA is an *overlay* that attests those edges and adds dynamic-dispatch fan-out. The
302+
measured payoff is decisive on library code (on `commons-lang` WALA RTA resolves 37 in-project edges to
303+
JavaParser's 16,605) and complementary on dispatch-heavy code (on `daytrader8` RTA adds more in-project
304+
edges than declared). So `-a 2` never fails for want of a build: `--no-rta`, or a build failure,
305+
degrades to `declared`-only rather than failing the level. The overlay adds external symbols but never
306+
in-project nodes — a WALA endpoint absent from the tree (a bridge/`access$`/`lambda$` synthetic, or an
307+
`$anon$N`-vs-`Outer$1` identity-join failure) is dropped, not fabricated.
308+
309+
### D19 — `external_symbols`: flat map keyed by binary-name `@external` can-id
310+
Out-of-project call targets are homed in an application-scope `external_symbols{}` map so no edge
311+
dangles, keyed by an `@external` can-id (`can://java/<app>/@external/<binary-type>/<signature>`),
312+
positionally parallel to an in-project callable id with `@external` in the file slot. The key carries
313+
the **binary** type name (`java.util.Map$Entry`) — unambiguous about where the package ends, and the
314+
spelling WALA emits natively so the `rta` overlay joins — while the `declaring_type` field carries the
315+
legible dotted form. v1 dropped external targets entirely (it filtered its graph to application
316+
classes); v2 homes 5,265 distinct symbols across the ten fixtures and keeps the 32,644 edges reaching
317+
them.
318+
319+
### D20 — Signature erasure: true recursive erasure, replacing `erasure().describe()`
320+
`Signatures` erases parameter types to a fixpoint rather than one level: JavaParser's
321+
`ResolvedType.erasure()` erases a type variable to its bound but leaves the bound's own type arguments,
322+
so a generic parameter leaked them into a durable id (`copy(java.util.function.Consumer<?>[])`), which
323+
no WALA descriptor (`java.util.function.Consumer[]`) could ever join. Iterating to a fixpoint erases the
324+
bound too. This is provably collision-free (two methods in one class cannot share a JVM erasure —
325+
0 collisions measured across 10,989 signatures), information-neutral (`type_parameters`/D16 preserves
326+
the bounds), and required by the `rta` join. Landed in #188, before any id became durable.
327+
328+
### D21 — L1 synthesizes implicit constructors under the JLS rule
329+
A `new Foo()` where `Foo` declares no constructor needs a callable to point at, so L1 emits the
330+
compiler-guaranteed constructor with `is_implicit:true` and no `body_span`. The rule follows the
331+
language, not a guess: a class or enum gets `<init>()` iff it declares **zero** constructors; a record
332+
gets the canonical `<init>(components…)` unless a *canonical* constructor is declared (a non-canonical
333+
one does **not** suppress it — `record R(int x,int y){ R(int x){…} }` compiles to both); interfaces and
334+
annotations never; an anonymous class always gets exactly one matching the creation site's resolved
335+
arguments. This replaces v1's fabricated `filePath:"<<implicit>>"` vertices (164 across the ten
336+
fixtures) with real tree nodes an edge can land on.
337+
338+
### D22 — Backfill plumbing: declaring-type hint, cache-serialized but payload-excluded
339+
`dst` needs the callee's *declaring* type, which diverges from the emitted `receiver_type` on every
340+
inherited call and exists only during L1's per-module resolution. L1 records it once per resolved call
341+
site as a hint on the `call` node — a binary type name, or, for an anonymous creation, the `can://` id
342+
of the anon's own constructor (a discriminated union L2 dispatches on the `can://` prefix, matched by
343+
AST node identity). L2 then maps it through a type index derived from the emitted tree (keyed by binary
344+
name, so one index serves both the JavaParser hint and WALA's native names). The hint must survive the
345+
incremental cache but never reach the payload, so it is marked `@CacheOnly`: `V2Json.cache()` keeps it
346+
while `V2Json.compact()`/`pretty()` exclude it in both directions, and **both** `L1Cache` paths use the
347+
cache writer — a `transient` field would have been dropped by the cache too, silently losing the
348+
backfill on a warm-cache run.
349+
350+
### D23 — L1 synthesizes implicit record accessors and enum `values()`/`valueOf(java.lang.String)`
351+
The symbol solver resolves `money.cents()` and `Op.values()` to the in-project declaring type itself,
352+
so without the generated member in the tree the call would resolve to a callable that is neither
353+
nameable nor homable as external (the type is ours). L1 therefore synthesizes, on **named** types only,
354+
one accessor per record component (suppressed by a declared accessor of that name; a varargs
355+
component's accessor returns an array) and, on every enum, `values()` and
356+
`valueOf(java.lang.String)` unconditionally (declaring either is a compile error, so nothing suppresses
357+
them). A record's generated `equals`/`hashCode`/`toString` are a stated blind spot — JavaParser cannot
358+
resolve those sites, so no `callee` is ever set and synthesizing them would add callables nothing points
359+
at.
360+
361+
### D24 — External call edges are opt-in (`--external-calls`), off by default for v1 parity
362+
v1's L2 call graph kept only edges whose target is an application class; calls into the JDK or a
363+
library were dropped. To keep the default v2 `-a 2` output at parity during the migration,
364+
`external_symbols` and the edges reaching them are gated behind `--external-calls` (off by default).
365+
When off, a call resolving out of the project is dropped exactly like an unresolved one — no
366+
{@code callee}, no edge, no {@code external_symbols} entry — so no-dangling holds trivially and the
367+
default graph matches v1's application-only shape. When on (the design's §2 posture, D19) they are
368+
homed so no edge dangles. The gate is threaded through both the `declared` pass and the `rta` join;
369+
an empty `external_symbols`/`call_graph` is omitted rather than emitted as `{}`/`[]` (absence = no
370+
fact, D10), so parity is a *missing* key, not an empty one. The `L2CallGraph.build` library default
371+
keeps external on (it is intrinsic to L2); only the CLI defaults it off.
372+
290373
### Scope guard
291374
The analyzer is a **pure graph provider**: it emits the CFG/PDG/SDG substrate and
292375
stops. Slicing, taint, and reachability are **SDK queries** over the emitted graph

0 commit comments

Comments
 (0)