diff --git a/.claude/SCHEMA_DECISIONS.md b/.claude/SCHEMA_DECISIONS.md new file mode 100644 index 0000000..f8ef755 --- /dev/null +++ b/.claude/SCHEMA_DECISIONS.md @@ -0,0 +1,156 @@ +# Schema decisions (codeanalyzer-typescript) + +Auditable record of the node-by-node schema design. Anchored on the two mature reference +analyzers — Java (`python-sdk/cldk/models/java/models.py`, rich-edge legacy) and Python +(`codeanalyzer-python/codeanalyzer/schema/py_schema.py`, identity-only, the model we mirror). +Every divergence below was decided **with the user**. + +## Invariant spine (never drifts) +- Root: `TSApplication { symbol_table: Dict[path, TSModule], call_graph: List[TSCallEdge], + entrypoints: Dict[str, List[TSEntrypoint]] }`. +- `symbol_table` keyed by **project-relative POSIX path with extension** (e.g. `src/user.ts`). +- `Module → Class/Callable` nesting; identity-only edges (`source`/`target` are bare signature + strings that byte-match a real `Callable.signature`). +- **One `signatureOf()`** produces every id, caller- and callee-side. + +## Decisions + +| # | Node / concept | Java | Python | **TS decision** | Rationale | +|---|---|---|---|---|---| +| 1 | **Signature scheme** | dotted FQN | `module.Class.method` (file-stem prefix, can collide) | **rel-path (no ext) + dotted members**: `src/services/user.UserService.getUser` | unique project-wide, file is recoverable from the id | +| 2 | **Constructor id** | `` | `Class.__init__` | **`Class.constructor`** | matches the TS keyword; reads naturally | +| 3 | **interface / type-alias / enum** | one Class + `is_interface`/`is_enum` flags | none | **separate sibling collections** `interfaces{}`, `type_aliases{}`, `enums{}` on Module/Namespace, each a typed node with its own signature | first-class & queryable; `base_classes`/edges can reference them | +| 4 | **Decorators** | flat `annotations: List[str]` | structured `PyDecorator` | **structured `TSDecorator`** (name, qualified_name, positional_arguments[], keyword_arguments{}, span) | entrypoint finders read `@Get('/path')` without re-parsing | +| 5 | **Generics** | — | — | **structured `TSTypeParameter[]`** (`{name, constraint?, default?}`) on class/interface/callable/type-alias | faithful ``; queryable | +| 6 | **extends / implements** | `extends_list` + `implements_list` | flat `base_classes` | **flat `base_classes` (spine) + typed `implements_types`** | `get_class_hierarchy` reads `base_classes`; split preserves class-vs-interface | +| 7 | **Member modifiers** | flat `modifiers: List[str]` | — | **typed fields**: `accessibility` (public\|private\|protected\|null), `is_static`, `is_abstract`, `is_async`, `is_generator`, `is_readonly`, `is_optional`, `accessor_kind` (getter\|setter\|null) | consumers branch on visibility/static directly | +| 8 | **Ambient / JSX / namespace / overloads** | — | — | **first-class**: `is_ambient` on declarations; `namespaces{}` collection (recursive, same containers as Module); `overload_signatures: List[TSOverloadSignature]` on the implementation callable; `is_tsx`/`is_declaration_file` on Module | the team wants these queryable, not buried in tags | +| 9 | **Anonymous callables** | lambdas not materialized (`LambdaExpr` absent from the model) | lambdas/comprehensions get no `PyCallable`; internals stay attributed to the enclosing function | **materialized as a `V2Callable`** in the enclosing callable's `callables{}`, with its own `body`/`cfg`/`cdg`/`ddg` and `@formal_in:N` at L4; call sites re-anchor to it and no compensating enclosing-callable edge is emitted | a Python lambda is one expression; a JS arrow is a full body and the dominant unit of behaviour (883 handlers in Juice Shop). Folding loses the application. Honours L10 for the unnamed case. Spec: `docs/design/specs/anonymous-callable-materialization.md` | +| 10 | **Anonymous callable identity** | — | — | **`contributorName` contributes ``** to the dotted chain: `routes/login.login.` | durable tier as the keystone requires, no collision with the `@line:col` ordinal namespace, and byte-identical from both the resolver and Jelly (both know positions; neither counts ordinals). TS-local pending roadmap candidate 4 ratification | +| 11 | **Legacy anonymous surfaces (2.1.0)** | — | — | **retained, re-meant**: `:TSAnonymousCallable` becomes a second label on the real tree node (reached by containment); `synthesized_callables` becomes a signature→`can://` id index, not a node registry | keeps the bump MINOR under the `neo4j/schema.ts:19` rule (no label/relationship/key removed), keeps existing `MATCH` queries working, and closes #75 — the wipe's containment traversal now reaches these nodes | + +## Derived conventions +- **module prefix** = file key minus extension (`src/services/user`); also stored as + `TSModule.module_name`. +- **scope chain**: namespace/class/function names are dot-joined onto the module prefix as we + descend, so `namespace Api { class V1 {} }` in `src/api.ts` → `src/api.Api.V1`. +- **implicit constructors**: a class instantiated with `new` but lacking an explicit + constructor still needs an edge target, so each class without an explicit constructor gets a + synthesized `Class.constructor` callable (`is_implicit = true`) — mirrors Java's default + constructor. Keeps the call graph free of dangling edges. +- **call-graph dispatch precision** (Tier-1): decided at the Call Graph Construction step. + +## What stays open-vocabulary +`TSCallEdge.provenance` (`["tsc"]`, later `["tsc","joern"]`), `TSCallEdge.tags`, +`TSEntrypoint.tags` — plain strings/maps so a persisted `analysis.json` round-trips even +without the producing pass installed. + +--- + +# Level 3 — program graphs (`program_graphs`, issue #2) + +The `-a 3` section: CFG / PDG (CDG+DDG) / SDG, per the cross-language dataflow contract. +Shared vocabulary is untouched; everything TS-specific below is additive. + +## Node identity +- Every node keyed by `(signature, node_id)` — the SAME `signatureOf()` as + `symbol_table`/`call_graph`; `node_id` = source-span order of the owning AST node within the + callable, ENTRY = 0, EXIT = last. Stable across runs on identical content. +- Node kinds: `entry`, `exit`, `param` (the SDG formal-in; span = the parameter declaration), + `statement`. Statement-level CFG (no basic-block compression). + +## Decisions + +| # | Concept | Decision | Rationale | +|---|---|---|---| +| L1 | **CFG edge kinds** | shared set + TS-native `await_resume` (await suspension) and `yield` (generator suspension) as the outgoing-normal edge of the suspending statement | additive per the parity clause | +| L2 | **Short-circuit / ternary / optional chaining** | intra-statement — never split into CFG nodes; reads of both arms attributed to the containing statement | statement-level identity stays stable; sound over-approximation | +| L3 | **Exceptional edges** | over-approximate: any call/`new`/`await`/tagged template (and `throw`) edges to the nearest catch node, else the finally entry, else EXIT; a finally region's exits also edge outward (`exception`) for the re-raise path; bare property reads do NOT throw | region splicing, not finally-duplication | +| L4 | **Infinite loops** | `while (true)` / `for (;;)` still emit the loop-exit `false` edge (dead) | keeps EXIT the unique post-dominance root — the contract's synthetic edge | +| L5 | **Call sites (actuals)** | collapsed onto the containing statement node: it is both actual-in and actual-out; `PARAM_IN` var `argN` sources there, `PARAM_OUT` var `return` targets it, and **SUMMARY edges are self-edges** on it (var = the input that flows to the result) | no synthetic actual nodes → every node keeps a real source span | +| L6 | **Formal-out** | EXIT doubles as the formal-out node: return-value nodes get a synthetic DDG edge `→ EXIT` var `return`; module-global writes get `→ EXIT` var `` | PARAM_OUT sources at EXIT must be reachable from the callee's PDG (slice descent) — the two documented non-syntactic DDG edges | +| L7 | **Globals** | canonical path `.` (same prefix as signatures); defined at ENTRY on entry, ride the SDG as extra params (`PARAM_IN` → callee ENTRY, `PARAM_OUT` ← callee EXIT); callee transitive global effects are re-applied at the caller's callsite node (uses/defs), so cross-function global flow is visible in the caller's own DDG | HRB "globals as extra formals" | +| L8 | **Base identity** | a DDG base is its *declaration node* (locals/params/captured), `this`, or the canonical module path — labels are names, identity is the decl | shadowed names in nested scopes can never leak edges | +| L9 | **Aliasing (MVP)** | flow-insensitive union-find over bases joined by bare copies (`const q = p`); field writes are always weak (no kill); strong kills only for whole-base local/param writes | sound-leaning stub per the substrate menu; Jelly points-to upgrade is staged PR F | +| L10 | **Closures** | nested callables get their own graphs; their reads of outer state are *capture uses* attributed to the declaring statement in the enclosing CFG (capture-at-declaration); captured bases are defined at the closure's own ENTRY | capture edges without cross-graph DDG | +| L11 | **Summaries** | node-granular relational summaries: `param_flows` (argN → return), `global_reads`/`global_writes`, `globals_to_return`; composed bottom-up over the Tarjan SCC condensation, co-defined to a monotone fixpoint inside an SCC; k-limited access paths (`--graph-field-depth`, default 3) bound the domain | statement-level precision cap, documented posture | +| L12 | **External / unresolved callees** | conservative pass-through SUMMARY self-edges (every arg may flow to the result); no CALL/PARAM edges (their graphs don't exist — no dangling endpoints); their global effects are unmodeled | the call-graph no-dangling rule extended to graphs | +| L13 | **Emission scoping** | `--graphs cfg,dfg,pdg,sdg` (strict validation); `dfg` = the DDG subset of `pdg` (no separate section); `program_graphs.schema_version` versioned independently ("1.0.0") | contract | + +## Known unsoundness (documented, not silently absorbed) +Dynamic `eval` / `Function`, reflection and monkey-patching, dynamic property names beyond +`[*]`, `this` flow across call boundaries (no this-param edges yet), exceptions carrying values +(the catch binding is a def but not data-linked to the throw site), npm-internal global effects. + +## Deferred (staged in issue #2) +Taint models-as-data + `taint_flows` (PR E), Jelly-backed alias-aware propagation (PR F), +CPG Neo4j projection + schema bump (PR G), incremental re-analysis over the recorded summary +dependency edges in `graphs_summaries.json` (PR H — the file is written today, read by nothing). + +# Native v2 model (#96, docs/design/specs/native-v2-model.md) + +The v1 compute model and the emit-time v1→v2 transform are retired: `src/schema/schema.ts` IS +schema v2 (envelope `TSAnalysis` → root `TSApplication` → `TSModule`/`TSType`/`TSCallable`/ +`TSField`/`TSBodyNode`), built directly by the builders. Wire unchanged (proved by deep-equal +goldens across `-a 1..4` + cypher during the transition; schema_version stays 2.1.0). + +| # | Concept | Decision | Rationale | +|---|---|---|---| +| N1 | **One model family + per-run passes** | builders emit the wire shapes; `assignIds`/`l1Body`/`heritage`/`homing`/`l2Callees`/`dataflow/attach` stamp the derived layers each run | python parity (`assign_ids.py` et al.); ids embed `--app-name` while the cache round-trips the tree, so ids can never be baked at build time | +| N2 | **INTERNAL fields, strip-at-emit** | `call_sites`, `callee_signature`, `abs_path`, `content_hash`/`last_modified`/`file_size` ride the model, stripped by key in `finalizeAnalysis`'s deep wire copy | the resolver span-joins on call sites and the cache needs them; the wire never saw them (python d0084cb precedent) | +| N3 | **Derived layers rebuilt wholesale per run** | `l1Body` rebuilds `body{}` from `call_sites` and deletes `cfg`/`cdg`/`ddg`/`summary`; every pass is overwrite-idempotent | cache safety across `--app-name` changes + repeated finalization at different levels stays correct | +| N4 | **Present-or-absent is the model's own convention** | nullable leaf fields became optional; builders omit instead of null (the sanctioned `callee: null` excepted) | the old recursive emit-time null-pruning defined the wire nowhere; now the types do | +| N5 | **types{} fill order = collision precedence** | classes → interfaces → enums → aliases → namespaces; later kind wins a member-key collision (declaration merging) | bit-for-bit the historical per-kind bucket merge | + +# Repository-artifact layer (#101, docs/design/specs/artifacts-and-dependencies.md) + +Python-parity port of 51ee29e: `application.artifacts{}` with contained dependency/config-key +children, `@artifact/` id marker, level-free. TS decisions: coined additive scope token `peer` +(npm's contract-with-host; shared vocabulary grows); JSON-lock family extracts, yarn/pnpm +inventory-only; declared-only records (`direct:false` reserved); the wire strip became +STRUCTURAL (structuredClone + targeted deletes) because artifact `content_hash` is wire payload +while the module trio is internal — and because the stringify-roundtrip clone OOM'd at +vscode-L4 scale (measured). SDK `extra="forbid"` ⇒ lockstep: python-sdk gains the families +before its pin moves. + +# Linker propagation tiers (#100) + +T4a property votes (object-literal callbacks through parameters), T4b chained return summaries +(one level, memoized), T4c ctor-field chain (parameter properties + one bounded parameter hop), +property-initializer attribution (initializers execute in the ctor; initializer arrows are +class-scoped anons — the property-arrow gap closes sig-consistently). vscode ledger: 99.72%, +residual 135 all-classified. Joern parameter tables prove the param-shadow fabrications. + +## Recalibration (2026-08-27): PR-160 is the anchor + +The first #101 cut mirrored `51ee29e` — an UNMERGED python branch (`feat/configuration-files`). +The ratified contract is python PR #160 / spec PR #158: language-neutral `can://artifact/` ids, +flat roles[] artifacts with unbounded verbatim source (rules-matched capture only), flat +evidence-tagged `dependencies[]` (kinds runtime|dev|optional|build + our coined `peer`), +`unresolved_imports[]`, neutral :Artifact/:Package (purl) with the prefix-gate exception, +LOCKS coarse fan, TS_PROVIDES/TS_UNRESOLVED_IMPORT ghosts, `--resolve-installed`. config_keys +dropped for this cut (python's unit 4 owned config extraction) — **superseded 2026-08-30**, see +below: units B/C/D shipped the family. Lesson recorded: parity anchors must be merged refs, not +local branch archaeology. + +## v1.3.0 parity (2026-08-30, #101 units A–D) + +Spec: `docs/design/specs/2026-08-30-artifact-layer-v130-parity.md`. Brings the branch to +codeanalyzer-python **v1.3.0**: the ConfigKey family, the level-graded `config_use` edge, and +deployment-env namespaces all shipped, reversing the "config_keys dropped" line above. + +| # | Concept | Decision | Rationale | +|---|---|---|---| +| A1 | **`direct: false` transitives** (`src/artifacts/deps.ts::transitiveRecords`) | every lock-pinned package no manifest declares becomes its own `TSDependency` record: `direct: false`, `kind: "runtime"` unconditionally, `declared_in` the lock artifact, `prov: ["lockfile"]`. Projected onto `DECLARES_DEPENDENCY.direct` in Neo4j | dependency *surface* and dependency *supply chain* are different questions — a lock file never records WHY a package is present, so `kind` asserts the safe default instead of inferring one from a whole-graph walk | +| B1 | **`config_access`** joins `call`/`entry`/`exit`/… as new L1 `body{}` vocabulary (`src/schema/l1Body.ts`) | a recognized env-root read (`process.env.X`, `import.meta.env.X`, `Bun.env.X`, incl. destructured bindings) mints one node per read into the SAME body-key space as `call` nodes, carrying `root`/`key?` but never `callee` | keeps `config_uses.src` a uniform ordinal id at every level without inventing a second body-node addressing scheme; a read is not a call, so it must not resolve through `TS_RESOLVES_TO` | +| B2 | **`arg.`/`env.` config-key id disambiguation** (`src/schema/assignIds.ts`) | the `key` FIELD always stays the bare variable name (env-namespace resolution joins on a plain `key ===` match); only the ID gets an internal prefix — `arg.` for a `namespace: "dockerfile"` (ARG) mint, `env.` for a YAML artifact's `namespace: "env"` dual-mint. Dockerfile's own `ENV` mint and every ordinary structural key stay unprefixed | a bare name can mint twice on one artifact (Dockerfile `ARG VERSION` beside its own `ENV VERSION=$VERSION`; a YAML top-level `PAYMENT_HOST:` leaf beside the `env` dual-mint under `services.web.environment.*`) and Task 10 MERGEs `:ConfigKey` on id — an undisambiguated collision would silently drop one key. python v1.3.0's scheme, adopted verbatim | +| C1 | **`config_reads` shrinks as `-a` rises — deliberately** | the layer's one non-monotonic section, unlike `config_uses` (asserted superset-monotonic, L2 ⊆ L3 ⊆ L4 — measured 21/25/29 uses, 10/9/8 reads on the `artifacts-app` fixture). A read unresolved at the literal tier can close at a higher dataflow tier, so it MOVES from `config_reads` to `config_uses` as the level climbs | mirrors python's documented same-shape caveat; a consumer diffing two levels must read a vanished `config_reads` record as "resolved at the higher tier," never "fixed in the code" — recorded here and in the consumer skill (`docs/skills/analyzing-cants-graphs/`) | +| D1 | **`SCHEMA_VERSION` held at 2.1.0** | PR #103's provisional `"2.2.0"` bump (recorded in the 2026-08-27 recalibration above) reverted in Task 10; the whole config layer (`ConfigKey`, `DEFINES_CONFIG`, `TS_USES_CONFIG`, `direct` on `DECLARES_DEPENDENCY`) landed additively under 2.1.0 instead | every analyzer re-baselines together, cross-language, only when the artifact layer settles across the org — the maintainer's call, tracked in the org epic; a solo version bump would commit the schema before that's decided | + +`config_reads` and `config_uses` both stay level-graded via the SAME three tiers: literal (L2, +`config_access`/detector-table call joined to a declared key by exact `(namespace, key)` match), +dataflow-intra (L3, reaching-definitions to a unique string literal), dataflow-interproc (L4, the +same chain crossing one call boundary via the SDG param/summary edges). Neither tier ever guesses: +an unresolved read is `reason: "non-literal"` (key never closes on one literal) or +`reason: "undefined-key"` (a literal key matching no declared `ConfigKey`) — first-class in +`config_reads`, never silently dropped. diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index be00f29..f8622a8 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -144,6 +144,14 @@ jobs: - name: Stage the install script (release asset) run: cp packaging/install/cants-installer.sh release-bins/cants-installer.sh + # The consumer query skill teaches an agent the Cypher recipes for the graph this binary + # produces. Ship it two ways, python v1.3.0 parity: a tarball (drop into a skills/ + # directory) and the bare SKILL.md (read without unpacking). + - name: Stage the consumer query skill (release asset) + run: | + tar -czf release-bins/analyzing-cants-graphs-skill.tar.gz -C docs/skills analyzing-cants-graphs + cp docs/skills/analyzing-cants-graphs/SKILL.md release-bins/analyzing-cants-graphs-SKILL.md + - name: Build changelog (auto-generated from commits/PRs) id: changelog if: startsWith(github.ref, 'refs/tags/') @@ -191,6 +199,8 @@ jobs: | [cants-win_amd64.exe]($BASE/cants-win_amd64.exe) | x64 Windows | | [cants-installer.sh]($BASE/cants-installer.sh) | Shell installer | | [schema.json]($BASE/schema.json) | Neo4j schema contract | + | [analyzing-cants-graphs-skill.tar.gz]($BASE/analyzing-cants-graphs-skill.tar.gz) | Consumer query skill (tarball) | + | [analyzing-cants-graphs-SKILL.md]($BASE/analyzing-cants-graphs-SKILL.md) | Consumer query skill (SKILL.md only) | ## Changelog diff --git a/.gitignore b/.gitignore index 80ba338..b761141 100644 --- a/.gitignore +++ b/.gitignore @@ -12,3 +12,11 @@ packaging/python/README.md !CLAUDE.md !AGENTS.md + +# .claude is globally ignored (agent scratch state), but this branch's decision log is reviewable +# project record, tracked like CLAUDE.md/AGENTS.md above. A bare parent-directory exclusion can +# never be re-included by negating a child alone (git gitignore docs) -- un-ignore the dir, then +# re-ignore its contents, then except this one file. +!.claude/ +.claude/* +!.claude/SCHEMA_DECISIONS.md diff --git a/CLAUDE.md b/CLAUDE.md index 6565c84..bd76fcd 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -50,11 +50,12 @@ while cache round-trips tree), `l1Body` (`call_sites` → `body{}`), `heritage`, (`src/schema/emit.ts`) runs them + assembles envelope + strips INTERNAL fields (`call_sites`, `abs_path`, cache trio). -Call graph defaults to **union** of two backends: TS compiler resolver -and embedded [Jelly](https://github.com/cs-au-dk/jelly) flow analyzer (recovers -higher-order/callback edges resolver misses). Merged edges keep -`provenance` tag (`tsc` / `jelly`); `--tsc-only` or `--call-graph-provider jelly` -picks one alone. +Call graph = tsc resolver + **defuse linker** (#98): deterministic per-callable +pass over resolver leftovers — alias chains, decorator edges, library-callback +edges, bounded interprocedural votes, CHA-by-name fallback. No whole-program +fixpoint, no backend flag, one code path. Module-scope calls attributed to +MODULE (python #131 parity). prov tags: `tsc` / `defuse` / `import`. Joern +superset ledger: `docs/design/specs/defuse-linker-joern-ledger.md`. ## Architecture — follow the pipeline @@ -65,8 +66,9 @@ it first; everything else is stage it calls, in order: 2. **buildSymbolTable** (`src/syntactic_analysis`) — modules, classes, interfaces, enums, type aliases, namespaces, functions, methods, variables, decorators, JSDoc, with precise source spans. -3. **call graph** (`src/semantic_analysis`) — `selectProvider()` picks tsc / jelly / - union; each provider returns edges + external (phantom) symbols. +3. **call graph** (`src/semantic_analysis`) — tsc resolver (`callGraph.ts`, incl. + module-scope sweep + RTA + phantoms) then `defuseLinker.ts` tiers T1–T5; + merged with provenance union. 4. **program graphs** (`src/dataflow`) — levels 3–4 (`-a 3`/`-a 4`): CFG → post-dominance/CDG → access-path def-use → PDG → SCC-condensed bottom-up summaries → SDG. This is *compute* (IR in `src/schema/graphs.ts`); `src/dataflow/attach.ts` writes it **onto tree** @@ -96,13 +98,29 @@ test — treat both as contracts, keep in lockstep with JSON. | `src/core.ts` | `analyze()` orchestrator — the spine | | `src/options` | Parsed CLI options / `AnalysisOptions` | | `src/syntactic_analysis` | Symbol table (ts-morph traversal) | -| `src/semantic_analysis` | Call-graph providers (tsc, jelly, union), phantoms | +| `src/semantic_analysis` | Call graph: tsc resolver + defuse linker (T1–T5), phantoms | | `src/dataflow` | L3/L4 program-graph **compute** (CFG, dominance/CDG, def-use, summaries, SDG) + `attach.ts` (IR → tree) | | `src/schema` | **the native v2 model** (`schema.ts`) + per-run passes (`assignIds`/`l1Body`/`heritage`/`homing`/`l2Callees`) + `emit.ts` (`finalizeAnalysis`) + `signatureOf` + graphs IR | | `src/build` | Dep materialization; `build/neo4j` = the v2 graph projection (project/rows/cypher/bolt/schema) | | `src/utils` | fs, caching, logging, serialization (`serialize.ts` writes the envelope), version | | `test` | Bun tests + `fixtures/sample-app` + `fixtures/dataflow-app`; `schema-v2.test.ts` = the L1–L4 gates | +**Repository-artifact layer** (#101, python v1.3.0 parity): three level-free sections — +`application.artifacts{}` (never-drop inventory, LANGUAGE-NEUTRAL `can://artifact//` +ids, roles[], text-capture policy: `--no-artifact-text`/`--artifact-text-max-bytes`, `sha256`/ +`size_bytes` always full-file even when `source` is a truncated prefix), `dependencies[]` (npm +kinds incl. coined `peer`, `direct:false` lockfile-only transitives), `unresolved_imports[]` +(@types type-only rule; `--resolve-installed` opt-in probe). Each artifact also carries +`config_keys[]` (env/JSONC/YAML/TOML/INI/dockerfile namespaces, `@key/`-suffixed ids, +`arg.`/`env.` internal id disambiguation). `config_uses`/`config_reads` join a `config_access` L1 +body-node read (or a detector-table call) to a declared key through a level-graded +literal→dataflow-intra→dataflow-interproc tier (`src/semantic_analysis/configUse.ts`, +`src/dataflow/configUse.ts`); `config_reads` deliberately SHRINKS as `-a` rises — the layer's one +non-monotonic section. `src/artifacts/`. Neo4j contract 2.1.0 (SCHEMA_VERSION unmoved — every +analyzer re-baselines together later): NEUTRAL :Artifact/:Package/:ConfigKey (purl) — sanctioned +prefix exception — plus TS_PROVIDES/TS_UNRESOLVED_IMPORT into :TSExternal ghosts and +TS_USES_CONFIG into :ConfigKey. Consumer query skill: `docs/skills/analyzing-cants-graphs/`. + ## Commands - `bun run start -- --input /path/to/project` — run analyzer from source. diff --git a/README.md b/README.md index 55d1a92..10ea9f9 100644 --- a/README.md +++ b/README.md @@ -24,11 +24,11 @@ structure into a **Neo4j property graph**. It is the TypeScript backend behind [Python](https://github.com/codellm-devkit/codeanalyzer-python) and [Java](https://github.com/codellm-devkit/codeanalyzer-java) siblings. -By default the call graph is the **union** of two backends: the TypeScript compiler's resolver and -[Jelly](https://github.com/cs-au-dk/jelly) — a flow-based analyzer that resolves higher-order and -callback edges the resolver misses, embedded in the `cants` binary (no extra install). Merged edges -keep a `provenance` tag (`tsc` / `jelly`), so you can still tell the two apart. Pass `--tsc-only` to -drop Jelly and run the resolver alone, or `--call-graph-provider jelly` for Jelly alone. +The call graph is the TypeScript compiler's resolver plus a **defuse linker** — a deterministic, +per-callable pass that backfills the edges the resolver misses (alias chains, decorator +invocations, callbacks handed to library calls, parameter-flow calls) with no whole-program +fixpoint. Edges keep a `provenance` tag (`tsc` / `defuse` / `import`), so you can tell the layers +apart, and the output is byte-identical across runs. ## Table of Contents @@ -55,9 +55,13 @@ drop Jelly and run the resolver alone, or `--call-graph-provider jelly` for Jell methods, variables, decorators, and JSDoc, with precise source spans. - **Call graph** — the TypeScript compiler's resolver plus Rapid Type Analysis (RTA), with **phantom (external) nodes** for calls into imported libraries and Node builtins. -- **Pluggable call-graph backend** — the `union` of the `tsc` resolver and the embedded - [Jelly](https://github.com/cs-au-dk/jelly) flow analyzer by default (`--tsc-only` for the resolver - alone, `--call-graph-provider jelly` for Jelly alone). +- **Defuse linker** — a deterministic per-callable pass over the resolver's leftovers: alias + chains, decorator invocations, library-callback edges, and bounded interprocedural votes — + validated as a strict superset of Joern's real call pairs on the reference corpus. +- **Repository-artifact layer** — every non-code file (manifests, lockfiles, CI, containers, env) + inventoried with flat, evidence-tagged dependencies (direct and transitive) and a `ConfigKey` + family joined to the code that reads it via a level-graded `config_use` edge; see + `docs/skills/analyzing-cants-graphs/`. - **Neo4j output** — project the analysis into a labeled property graph: a self-contained `graph.cypher` snapshot, or an **incremental** push to a live database over Bolt. - **Versioned schema** — a machine-readable, version-stamped Neo4j schema contract @@ -182,11 +186,13 @@ Options: node_modules) --no-phantoms disable phantom (external) nodes for imported/required library calls - --call-graph-provider call-graph backend: union (default, tsc ∪ - jelly) | tsc | jelly | both (deprecated alias - of union) (default: "union") - --tsc-only use the tsc resolver only — opt out of Jelly - edges (overrides --call-graph-provider) + --resolve-installed probe node_modules metadata for import→package + binding (default: repo files only) + --no-artifact-text keep the artifact inventory but drop captured + raw text + --artifact-text-max-bytes per-file byte cap for captured artifact text; + larger files are truncated and flagged + (default: "262144") -c, --cache-dir cache/intermediate directory -v, --verbose increase verbosity (repeatable) -h, --help display help for command @@ -214,17 +220,12 @@ Options: cants --input ./my-ts-project --target-files src/a.ts src/b.ts ``` -4. **Resolver-only call graph (opt out of Jelly):** - ```sh - cants --input ./my-ts-project --tsc-only - ``` - -5. **Force a clean rebuild with a custom cache directory:** +4. **Force a clean rebuild with a custom cache directory:** ```sh cants --input ./my-ts-project --eager --cache-dir /path/to/custom-cache ``` -6. **Program graphs (level 3): CFG/PDG/SDG in `analysis.json`:** +5. **Program graphs (level 3): CFG/PDG/SDG in `analysis.json`:** ```sh cants --input ./my-ts-project -a 3 # full program_graphs section cants --input ./my-ts-project -a 3 --graphs cfg,pdg # scope the emitted graphs @@ -282,8 +283,8 @@ nodes all join. **Substrate (locked in [issue #2](https://github.com/codellm-devkit/codeanalyzer-typescript/issues/2)):** the CFG and reaching-definitions are hand-built from the ts-morph AST; the call-graph oracle is -the existing provenance-merged tsc ∪ Jelly graph; aliasing is a flow-insensitive copy-alias MVP -(Jelly points-to-backed propagation is a staged upgrade). Function summaries are composed +the provenance-merged tsc + defuse graph; aliasing is a flow-insensitive copy-alias MVP +(points-to-backed propagation is a staged upgrade). Function summaries are composed bottom-up over the SCC condensation of the call graph, with k-limited access paths; module globals ride the SDG as extra parameters. The analysis is deliberately sound-leaning and over-approximate; known unsoundness (dynamic `eval`, reflection/monkey-patching, npm-internal diff --git a/bun.lock b/bun.lock index 351d24f..8b6c70b 100644 --- a/bun.lock +++ b/bun.lock @@ -8,9 +8,9 @@ "commander": "^15.0.0", "neo4j-driver": "^5.28.0", "ts-morph": "^28.0.0", + "yaml": "^2.9.0", }, "devDependencies": { - "@cs-au-dk/jelly": "0.13.0", "@testcontainers/neo4j": "^12.0.3", "@types/bun": "^1.3.14", "@types/node": "^25.9.1", @@ -19,90 +19,15 @@ }, }, }, - "patchedDependencies": { - "@cs-au-dk/jelly@0.13.0": "patches/@cs-au-dk%2Fjelly@0.13.0.patch", - }, "packages": { - "@babel/code-frame": ["@babel/code-frame@7.29.7", "", { "dependencies": { "@babel/helper-validator-identifier": "^7.29.7", "js-tokens": "^4.0.0", "picocolors": "^1.1.1" } }, "sha512-Aup7aUOfpbAUg2ROOJN6Iw5f9DMBlzu0mIkm/malLQFN/YQgO48wCj0Kxa3sEHJvPVFg7siR+qRInwXd2qhQKw=="], - - "@babel/compat-data": ["@babel/compat-data@7.29.7", "", {}, "sha512-locTkQyKvwIEgBzVrn8693ebc97F2U8ZHjbXwDXJ5Fn2TCpNwTlKcaKLkdHop5c/icOFE7qt7Q9JC5hnKNa6Gg=="], - - "@babel/core": ["@babel/core@7.29.7", "", { "dependencies": { "@babel/code-frame": "^7.29.7", "@babel/generator": "^7.29.7", "@babel/helper-compilation-targets": "^7.29.7", "@babel/helper-module-transforms": "^7.29.7", "@babel/helpers": "^7.29.7", "@babel/parser": "^7.29.7", "@babel/template": "^7.29.7", "@babel/traverse": "^7.29.7", "@babel/types": "^7.29.7", "@jridgewell/remapping": "^2.3.5", "convert-source-map": "^2.0.0", "debug": "^4.1.0", "gensync": "^1.0.0-beta.2", "json5": "^2.2.3", "semver": "^6.3.1" } }, "sha512-RgHBCvtjbOK2gXSNBNIkNoEc9qoVEtau3hj8gEqKQuL3HZAibKarWFEI3Lfm6EYKkLalOh8eSrj9b+ch9H/VBA=="], - - "@babel/generator": ["@babel/generator@7.29.8", "", { "dependencies": { "@babel/parser": "^7.29.8", "@babel/types": "^7.29.8", "@jridgewell/gen-mapping": "^0.3.12", "@jridgewell/trace-mapping": "^0.3.28", "jsesc": "^3.0.2" } }, "sha512-gZbepsdh3WDtgZKWL+vTPh71LSBrm/Y4/QDZBVCcYfmeTEEuoOYwlSy+G1StfJg+/Zy550u/3TATbm7qDbbMtg=="], - - "@babel/helper-annotate-as-pure": ["@babel/helper-annotate-as-pure@7.29.7", "", { "dependencies": { "@babel/types": "^7.29.7" } }, "sha512-OoK6239jHPuSQOoS0kfTVKn0b/rVTk0seKq4Gd2UMLtmOVLjDC0ki3e+c90Trqv2gMfvJFqkiljrr568+qddiw=="], - - "@babel/helper-compilation-targets": ["@babel/helper-compilation-targets@7.29.7", "", { "dependencies": { "@babel/compat-data": "^7.29.7", "@babel/helper-validator-option": "^7.29.7", "browserslist": "^4.24.0", "lru-cache": "^5.1.1", "semver": "^6.3.1" } }, "sha512-wem6WaBj4NaVYVdNhLPPVacES6ZJ+KBBfSkTMD3YZxbP3rm3Di85tJU5ljaUNhaOynt+Aj0xruhYuzQBt8n71g=="], - - "@babel/helper-create-class-features-plugin": ["@babel/helper-create-class-features-plugin@7.29.7", "", { "dependencies": { "@babel/helper-annotate-as-pure": "^7.29.7", "@babel/helper-member-expression-to-functions": "^7.29.7", "@babel/helper-optimise-call-expression": "^7.29.7", "@babel/helper-replace-supers": "^7.29.7", "@babel/helper-skip-transparent-expression-wrappers": "^7.29.7", "@babel/traverse": "^7.29.7", "semver": "^6.3.1" }, "peerDependencies": { "@babel/core": "^7.0.0" } }, "sha512-IY3ZD9Tmooqr3TUhc3DUWxiuo8xx1DWLhd5M7hQ+ZWJamqM2BbalrBJb2MisSLoYorOj75U03qULCxQTY9r3hg=="], - - "@babel/helper-globals": ["@babel/helper-globals@7.29.7", "", {}, "sha512-3nQVUAtvkKH9zahfWgw96Jc/uFOmjACE1kQz82E2lqWmHBgjzbNlsC22nuQTfahmWeQtTq5nQ/4Nnd2A1wj4zA=="], - - "@babel/helper-member-expression-to-functions": ["@babel/helper-member-expression-to-functions@7.29.7", "", { "dependencies": { "@babel/traverse": "^7.29.7", "@babel/types": "^7.29.7" } }, "sha512-j+7JYmk1JYDtACIGj0QJqqWZjoUpMoEikQGADMaHgCMCSDqd2+P32rfcibUNrGOMWrlzK1WJBdxrB3JJQZwWtg=="], - - "@babel/helper-module-imports": ["@babel/helper-module-imports@7.29.7", "", { "dependencies": { "@babel/traverse": "^7.29.7", "@babel/types": "^7.29.7" } }, "sha512-ejHwrQQYcm9xnTivShn2IDOlIzInN34AXskvq9QicvCtEzq1Vzclu/tKF8Jq1Cg8JG2GL6/EmjgsCT7lXepE3g=="], - - "@babel/helper-module-transforms": ["@babel/helper-module-transforms@7.29.7", "", { "dependencies": { "@babel/helper-module-imports": "^7.29.7", "@babel/helper-validator-identifier": "^7.29.7", "@babel/traverse": "^7.29.7" }, "peerDependencies": { "@babel/core": "^7.0.0" } }, "sha512-UPUVSyXbOh627KiCIGQSgwWzGeBKLkaJ9PJEdrngIwMSzxLR4jS4+f1f1jb7VzBbg8nFLaYotvVPFCTqdrmTAg=="], - - "@babel/helper-optimise-call-expression": ["@babel/helper-optimise-call-expression@7.29.7", "", { "dependencies": { "@babel/types": "^7.29.7" } }, "sha512-+kmGVjcT9RGYzoDwdwEqEvGgKe3BYq+O1iGzjFubaNgZHwYHP6lsF2Yghf4kEuv9BV7tYDZ913aBW9am6YKong=="], - - "@babel/helper-plugin-utils": ["@babel/helper-plugin-utils@7.29.7", "", {}, "sha512-G7sHYigPY17oO5SYWnfD/0MTBwVR781S/JI643e/JhUYgVgWE/61SoW3NH9KWUKyKq5LVh3npif99Wkt6j86Jw=="], - - "@babel/helper-replace-supers": ["@babel/helper-replace-supers@7.29.7", "", { "dependencies": { "@babel/helper-member-expression-to-functions": "^7.29.7", "@babel/helper-optimise-call-expression": "^7.29.7", "@babel/traverse": "^7.29.7" }, "peerDependencies": { "@babel/core": "^7.0.0" } }, "sha512-atfGXWSeCiF4DnKZIfmJfQRkSw9b9gNNXR1kqKjbhG4pGYCOnkp8OcTB8E3NXjBu8NpheSnOeNKz8KT7UNFTmQ=="], - - "@babel/helper-skip-transparent-expression-wrappers": ["@babel/helper-skip-transparent-expression-wrappers@7.29.7", "", { "dependencies": { "@babel/traverse": "^7.29.7", "@babel/types": "^7.29.7" } }, "sha512-brcMGQaVzIeUb+6/bs1Av0f8YuNNjKY2JyvfRCsFuFsdKccEQ5Ges2y74D74NZ1Rz8lKJ9ksJkfqwQFJ/iNEyQ=="], - - "@babel/helper-string-parser": ["@babel/helper-string-parser@7.29.7", "", {}, "sha512-Pb5ijPrZ89GDH8223L4UP8i6QApWxs04RbPQJTeWDV0/keR2E36MeKnyr6LYmUUvqRRI+Iv87SuF1W6ErINzYw=="], - - "@babel/helper-validator-identifier": ["@babel/helper-validator-identifier@7.29.7", "", {}, "sha512-qehxGkRj55h/ff8EMaJ+cYhyaKlHIxqYDn682wQD7RNp9UujOQsHog2uS0r2vzr4pW+sXf90NeeayjcNaX3fFg=="], - - "@babel/helper-validator-option": ["@babel/helper-validator-option@7.29.7", "", {}, "sha512-N9ZErrD+yW5geCDtBqnOoxmR8+tNKiGuxKlDpuJxfsqpa2dFcexaziGAE/qoHLiDDreVNMupxGmSoNlyvsA3gw=="], - - "@babel/helpers": ["@babel/helpers@7.29.7", "", { "dependencies": { "@babel/template": "^7.29.7", "@babel/types": "^7.29.7" } }, "sha512-1k2lAGRMfHTcwuNYcCNUmaUffmQv8KWMfh2iJUUeRlwlwH4FdNG7mfPI10NPfLHJFThE4Tyr4mv7kTNZOiPuBg=="], - - "@babel/parser": ["@babel/parser@7.29.8", "", { "dependencies": { "@babel/types": "^7.29.8" }, "bin": "./bin/babel-parser.js" }, "sha512-E8lTAYNB1KW+FH+VGJuZM1ioAx2E6oVlvQFRrf5P8ZZmsiJXYAD9vTFV7yyEURNzgh1dFqMZuO6tUwcARbqFCA=="], - - "@babel/plugin-proposal-decorators": ["@babel/plugin-proposal-decorators@7.29.7", "", { "dependencies": { "@babel/helper-create-class-features-plugin": "^7.29.7", "@babel/helper-plugin-utils": "^7.29.7", "@babel/plugin-syntax-decorators": "^7.29.7" }, "peerDependencies": { "@babel/core": "^7.0.0-0" } }, "sha512-EtU0Hi3GvrTqD56xKmZvV/uCXK2ZbwVNPNLAquVItcAZpUhkXwWlo3Fmj0c2LxgSf2I8IDULeAepwNP1OefLXg=="], - - "@babel/plugin-syntax-decorators": ["@babel/plugin-syntax-decorators@7.29.7", "", { "dependencies": { "@babel/helper-plugin-utils": "^7.29.7" }, "peerDependencies": { "@babel/core": "^7.0.0-0" } }, "sha512-9MTTLbF39X6sqM92JPEsoI7++26hjZvzkxKZy64aMhWLH2mPkJ/Q3AV4QLmls3R14FpSpkOwQQfUh962JGQxxg=="], - - "@babel/plugin-syntax-typescript": ["@babel/plugin-syntax-typescript@7.29.7", "", { "dependencies": { "@babel/helper-plugin-utils": "^7.29.7" }, "peerDependencies": { "@babel/core": "^7.0.0-0" } }, "sha512-ngr+82Sh0xMz25TPCZi+nC2iTzjfCdWS2ONXTp/PtSCHCgaCNBpdMqgvJ2ccdLlClVZ7sisIgB914j/JFe+RZA=="], - - "@babel/plugin-transform-template-literals": ["@babel/plugin-transform-template-literals@7.29.7", "", { "dependencies": { "@babel/helper-plugin-utils": "^7.29.7" }, "peerDependencies": { "@babel/core": "^7.0.0-0" } }, "sha512-NCSEJ4sLFU2gqAub45HYh4fus2yQ36rr6ei6vpU7NdoJqCpxvEG8E6eJpscGyXP3VHD2Ny+fSXr04k1hoUrFqA=="], - - "@babel/plugin-transform-typescript": ["@babel/plugin-transform-typescript@7.29.7", "", { "dependencies": { "@babel/helper-annotate-as-pure": "^7.29.7", "@babel/helper-create-class-features-plugin": "^7.29.7", "@babel/helper-plugin-utils": "^7.29.7", "@babel/helper-skip-transparent-expression-wrappers": "^7.29.7", "@babel/plugin-syntax-typescript": "^7.29.7" }, "peerDependencies": { "@babel/core": "^7.0.0-0" } }, "sha512-jK52h8LaLc7JarhQV2ofeFMts4H7vnOXnqZNA6fYglBTZewRBE51KWt3BUltW1P+KoPsYkHoJeXePuz4zo2LMw=="], - - "@babel/template": ["@babel/template@7.29.7", "", { "dependencies": { "@babel/code-frame": "^7.29.7", "@babel/parser": "^7.29.7", "@babel/types": "^7.29.7" } }, "sha512-puq+Gf35oI24FeN11LkoUQFqv9uwNeWpxXZi/Ji3rRIoKAzKnxRaZ+Gkj0vKS9ZCiTESfng1N9LyOyXvo+m+Gg=="], - - "@babel/traverse": ["@babel/traverse@7.29.8", "", { "dependencies": { "@babel/code-frame": "^7.29.7", "@babel/generator": "^7.29.8", "@babel/helper-globals": "^7.29.7", "@babel/parser": "^7.29.8", "@babel/template": "^7.29.7", "@babel/types": "^7.29.8", "debug": "^4.3.1" } }, "sha512-I5z7H3bf/41ktsNVLtpN0wAa336HkqIHQ5BuPLEhTkt1jVSyZpeNKIzTgEWmlxjdg81R0IgUCcaE+Ok3NvrfZg=="], - - "@babel/types": ["@babel/types@7.29.8", "", { "dependencies": { "@babel/helper-string-parser": "^7.29.7", "@babel/helper-validator-identifier": "^7.29.7" } }, "sha512-Vj1jF3cPfxg7OAfoI7QnVKLoILlm2JF9pnVHrX8qx7AHMiYWT+NDAA7jChlNgRS4WTLc/fD1lXLmPixluj+3Gg=="], - "@balena/dockerignore": ["@balena/dockerignore@1.0.2", "", {}, "sha512-wMue2Sy4GAVTk6Ic4tJVcnfdau+gx2EnG7S+uAEe+TWJFqE4YoWN4/H8MSLj4eYJKxGg26lZwboEniNiNwZQ6Q=="], - "@colors/colors": ["@colors/colors@1.6.0", "", {}, "sha512-Ir+AOibqzrIsL6ajt3Rz3LskB7OiMVHqltZmspbW/TJuTVuyOMirVqAkjfY6JISiLHgyNqicAC8AyHHGzNd/dA=="], - - "@cs-au-dk/jelly": ["@cs-au-dk/jelly@0.13.0", "", { "dependencies": { "@babel/core": "^7.28.5", "@babel/parser": "^7.28.5", "@babel/plugin-proposal-decorators": "^7.28.0", "@babel/plugin-transform-template-literals": "^7.27.1", "@babel/plugin-transform-typescript": "^7.28.5", "@babel/traverse": "^7.28.5", "@babel/types": "^7.28.5", "commander": "^9.5.0", "micromatch": "^4.0.8", "semver": "^7.7.3", "stringify2stream": "^1.1.0", "typescript": "^5.9.3", "winston": "^3.18.3" }, "bin": { "jelly": "lib/main.js", "jelly-server": "lib/server.js" } }, "sha512-UaEL24sdiZO32ZFZ5fjAHjp6kxeTt+PODxQm5AvB7v0suWzljXusc1Oc3qUi0zb6y/xshoQj9Ee8WmvF9ic5wg=="], - - "@dabh/diagnostics": ["@dabh/diagnostics@2.0.8", "", { "dependencies": { "@so-ric/colorspace": "^1.1.6", "enabled": "2.0.x", "kuler": "^2.0.0" } }, "sha512-R4MSXTVnuMzGD7bzHdW2ZhhdPC/igELENcq5IjEverBvq5hn1SXCWcsi6eSsdWP0/Ur+SItRRjAktmdoX/8R/Q=="], - "@grpc/grpc-js": ["@grpc/grpc-js@1.14.4", "", { "dependencies": { "@grpc/proto-loader": "^0.8.0", "@js-sdsl/ordered-map": "^4.4.2" } }, "sha512-k9Dj3DV/itK9D06Y8f190Qgop7/Ui+D0njFV3LHMPwPT75DpXLQohE9Wmz0QElrJnzsjB7KPWiKJbOl7IPDArQ=="], "@grpc/proto-loader": ["@grpc/proto-loader@0.7.15", "", { "dependencies": { "lodash.camelcase": "^4.3.0", "long": "^5.0.0", "protobufjs": "^7.2.5", "yargs": "^17.7.2" }, "bin": { "proto-loader-gen-types": "build/bin/proto-loader-gen-types.js" } }, "sha512-tMXdRCfYVixjuFK+Hk0Q1s38gV9zDiDJfWL3h1rv4Qc39oILCu1TRTDt7+fGUI8K4G1Fj125Hx/ru3azECWTyQ=="], "@isaacs/cliui": ["@isaacs/cliui@8.0.2", "", { "dependencies": { "string-width": "^5.1.2", "string-width-cjs": "npm:string-width@^4.2.0", "strip-ansi": "^7.0.1", "strip-ansi-cjs": "npm:strip-ansi@^6.0.1", "wrap-ansi": "^8.1.0", "wrap-ansi-cjs": "npm:wrap-ansi@^7.0.0" } }, "sha512-O8jcjabXaleOG9DQ0+ARXWZBTfnP4WNAqzuiJK7ll44AmxGKv/J2M4TPjxjY3znBCfvBXFzucm1twdyFybFqEA=="], - "@jridgewell/gen-mapping": ["@jridgewell/gen-mapping@0.3.13", "", { "dependencies": { "@jridgewell/sourcemap-codec": "^1.5.0", "@jridgewell/trace-mapping": "^0.3.24" } }, "sha512-2kkt/7niJ6MgEPxF0bYdQ6etZaA+fQvDcLKckhy1yIQOzaoKjBBjSj63/aLVjYE3qhRt5dvM+uUyfCg6UKCBbA=="], - - "@jridgewell/remapping": ["@jridgewell/remapping@2.3.5", "", { "dependencies": { "@jridgewell/gen-mapping": "^0.3.5", "@jridgewell/trace-mapping": "^0.3.24" } }, "sha512-LI9u/+laYG4Ds1TDKSJW2YPrIlcVYOwi2fUC6xB43lueCjgxV4lffOCZCtYFiH6TNOX+tQKXx97T4IKHbhyHEQ=="], - - "@jridgewell/resolve-uri": ["@jridgewell/resolve-uri@3.1.2", "", {}, "sha512-bRISgCIjP20/tbWSPWMEi54QVPRZExkuD9lJL+UIxUKtwVJA8wW1Trb1jMs1RFXo1CBTNZ/5hpC9QvmKWdopKw=="], - - "@jridgewell/sourcemap-codec": ["@jridgewell/sourcemap-codec@1.6.0", "", {}, "sha512-T7jf+5zgsZHwNJ4lvQ7/aezbyk0nNX+zJVWpmHA7VYsEx7a7qr5Rg5IbtJFqkgze5Y2sruq1RUY8Q837Od7iFw=="], - - "@jridgewell/trace-mapping": ["@jridgewell/trace-mapping@0.3.31", "", { "dependencies": { "@jridgewell/resolve-uri": "^3.1.0", "@jridgewell/sourcemap-codec": "^1.4.14" } }, "sha512-zzNR+SdQSDJzc8joaeP8QQoCQr8NuYx2dIIytl1QeBEZHJ9uW6hebsrYgbz8hJwUQao3TWCMtmfV8Nu1twOLAw=="], - "@js-sdsl/ordered-map": ["@js-sdsl/ordered-map@4.4.2", "", {}, "sha512-iUKgm52T8HOE/makSxjqoWhe95ZJA1/G1sYsGev2JDKUSS14KAgg1LHb+Ba+IPow0xflbnSkOsZcO08C7w1gYw=="], "@kwsites/file-exists": ["@kwsites/file-exists@1.1.1", "", { "dependencies": { "debug": "^4.1.1" } }, "sha512-m9/5YGR18lIwxSFDwfE3oA7bWuq9kdau6ugN4H2rJeyhFQZcG9AgSHkQtSD15a8WvTgfz9aikZMrKPHvbpqFiw=="], @@ -127,8 +52,6 @@ "@protobufjs/utf8": ["@protobufjs/utf8@1.1.2", "", {}, "sha512-b1UQwcEZ4yCnMCD8DAL1VlbvBJE9/IX4FTIp7BG1xYpf29SLazLSrqUkj4w7Y5y7cCVP6E5tcqqcI0xemPkHug=="], - "@so-ric/colorspace": ["@so-ric/colorspace@1.1.6", "", { "dependencies": { "color": "^5.0.2", "text-hex": "1.0.x" } }, "sha512-/KiKkpHNOBgkFJwu9sh48LkHSMYGyuTcSFK/qMBdnOAlrRJzRSXAOFB5qwzaVQuDl8wAvHVMkaASQDReTahxuw=="], - "@testcontainers/neo4j": ["@testcontainers/neo4j@12.0.4", "", { "dependencies": { "testcontainers": "^12.0.4" } }, "sha512-WRQfxFilXehFqq0nkeAP7xqAZavkj1YODYPiSLHrMHDoKIUxCu5fWFdigJakmvqKk86J/pup146fqZgspsuocg=="], "@ts-morph/common": ["@ts-morph/common@0.29.0", "", { "dependencies": { "minimatch": "^10.0.1", "path-browserify": "^1.0.1", "tinyglobby": "^0.2.14" } }, "sha512-35oUmphHbJvQ/+UTwFNme/t2p3FoKiGJ5auTjjpNTop2dyREspirjMy82PLSC1pnDJ8ah1GU98hwpVt64YXQsg=="], @@ -145,8 +68,6 @@ "@types/ssh2-streams": ["@types/ssh2-streams@0.1.13", "", { "dependencies": { "@types/node": "*" } }, "sha512-faHyY3brO9oLEA0QlcO8N2wT7R0+1sHWZvQ+y3rMLwdY1ZyS1z0W3t65j9PqT4HmQ6ALzNe7RZlNuCNE0wBSWA=="], - "@types/triple-beam": ["@types/triple-beam@1.3.5", "", {}, "sha512-6WaYesThRMCl19iryMYP7/x2OVgCtbIVflDGFpWnb9irXI3UjYE4AzmYuiUKY1AJstGijoY+MgUszMgRxIYTYw=="], - "abort-controller": ["abort-controller@3.0.0", "", { "dependencies": { "event-target-shim": "^5.0.0" } }, "sha512-h8lQ8tacZYnR3vNQTgibj+tODHI5/+l06Au2Pcriv/Gmet0eaj4TwWH41sO9wnHDiQsEj19q0drzdWdeAHtweg=="], "ansi-regex": ["ansi-regex@5.0.1", "", {}, "sha512-quJQXlTSUGL2LH9SUXo8VwsY4soanhgo6LNSm84E1LBcE8s3O0wpdiRzyR9z/ZZJMlMWv37qOOb9pdJlMUEKFQ=="], @@ -179,18 +100,12 @@ "base64-js": ["base64-js@1.5.1", "", {}, "sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA=="], - "baseline-browser-mapping": ["baseline-browser-mapping@2.11.20", "", { "bin": { "baseline-browser-mapping": "dist/cli.cjs" } }, "sha512-H0ulySigv6icDJ1F7SjtdCD6PrhTpdYCmP0CactWy1+ekh0AFd0o1Wn5T8b+hnTmdBx19u9yhL6wvCylXMY7zw=="], - "bcrypt-pbkdf": ["bcrypt-pbkdf@1.0.2", "", { "dependencies": { "tweetnacl": "^0.14.3" } }, "sha512-qeFIXtP4MSoi6NLqO12WfqARWWuCKi2Rn/9hJLEmtB5yTNr9DqFWkJRCf2qShWzPeAMRnOgCrq0sg/KLv5ES9w=="], "bl": ["bl@4.1.0", "", { "dependencies": { "buffer": "^5.5.0", "inherits": "^2.0.4", "readable-stream": "^3.4.0" } }, "sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w=="], "brace-expansion": ["brace-expansion@5.0.7", "", { "dependencies": { "balanced-match": "^4.0.2" } }, "sha512-7oFy703dxfY3/NLxC1fh2SUCQ0H9rmAY+5EpDVfXjUTTs+HEwR2nYaqLv+GWcTsumwxPfiz6CzCNkwXwBUwqCA=="], - "braces": ["braces@3.0.3", "", { "dependencies": { "fill-range": "^7.1.1" } }, "sha512-yQbXgO/OSZVD2IsiLlro+7Hf6Q18EJrKSEsdoMzKePKXct3gvD8oLcOQdIzGupr5Fj+EDe8gO/lxc1BzfMpxvA=="], - - "browserslist": ["browserslist@4.28.8", "", { "dependencies": { "baseline-browser-mapping": "^2.11.12", "caniuse-lite": "^1.0.30001809", "electron-to-chromium": "^1.5.402", "node-releases": "^2.0.53", "update-browserslist-db": "^1.3.0" }, "bin": { "browserslist": "cli.js" } }, "sha512-V2NpofLblG64mfOtSgDhOJESZEGogzDMBv/q+W6oc4LXWP/q75eOXoOaaOu1EOadB9U4Bwx/e0yzbvwKH8zalA=="], - "buffer": ["buffer@6.0.3", "", { "dependencies": { "base64-js": "^1.3.1", "ieee754": "^1.2.1" } }, "sha512-FTiCpNxtwiZZHEZbcbTIcZjERVICn9yq/pDFkTl95/AxzD1naBctN7YO68riM/gLSDY7sdrMby8hofADYuuqOA=="], "buffer-crc32": ["buffer-crc32@1.0.0", "", {}, "sha512-Db1SbgBS/fg/392AblrMJk97KggmvYhr4pB5ZIMTWtaivCPMWLkmb7m21cJvpvgK+J3nsU2CmmixNBZx4vFj/w=="], @@ -201,28 +116,20 @@ "byline": ["byline@5.0.0", "", {}, "sha512-s6webAy+R4SR8XVuJWt2V2rGvhnrhxN+9S15GNuTK3wKPOXFF6RNc+8ug2XhH+2s4f+uudG4kUVYmYOQWL2g0Q=="], - "caniuse-lite": ["caniuse-lite@1.0.30001810", "", {}, "sha512-TITQPUkaz+aVk5GL6NhOdwk1aEaNTSDPsGFWrTuhKGtjTF70jL/Oht2W4c6rXUe5fu7Ie19VIahAXHIIiWWNeg=="], - "chownr": ["chownr@1.1.4", "", {}, "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg=="], "cliui": ["cliui@8.0.1", "", { "dependencies": { "string-width": "^4.2.0", "strip-ansi": "^6.0.1", "wrap-ansi": "^7.0.0" } }, "sha512-BSeNnyus75C4//NQ9gQt1/csTXyo/8Sb+afLAkzAptFuMsod9HFokGNudZpi/oQV73hnVK+sR+5PVRMd+Dr7YQ=="], "code-block-writer": ["code-block-writer@13.0.3", "", {}, "sha512-Oofo0pq3IKnsFtuHqSF7TqBfr71aeyZDVJ0HpmqB7FBM2qEigL0iPONSCZSO9pE9dZTAxANe5XHG9Uy0YMv8cg=="], - "color": ["color@5.0.3", "", { "dependencies": { "color-convert": "^3.1.3", "color-string": "^2.1.3" } }, "sha512-ezmVcLR3xAVp8kYOm4GS45ZLLgIE6SPAFoduLr6hTDajwb3KZ2F46gulK3XpcwRFb5KKGCSezCBAY4Dw4HsyXA=="], - - "color-convert": ["color-convert@3.1.3", "", { "dependencies": { "color-name": "^2.0.0" } }, "sha512-fasDH2ont2GqF5HpyO4w0+BcewlhHEZOFn9c1ckZdHpJ56Qb7MHhH/IcJZbBGgvdtwdwNbLvxiBEdg336iA9Sg=="], + "color-convert": ["color-convert@2.0.1", "", { "dependencies": { "color-name": "~1.1.4" } }, "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ=="], - "color-name": ["color-name@2.1.1", "", {}, "sha512-p2FdgwVx1a9yWBHP2wI0VgShkDpgN4kZISkxdNipGBJWpa5G6b04OINlVWCyJj0JmfvcPrgqt95E9k8yvaOJFg=="], - - "color-string": ["color-string@2.1.4", "", { "dependencies": { "color-name": "^2.0.0" } }, "sha512-Bb6Cq8oq0IjDOe8wJmi4JeNn763Xs9cfrBcaylK1tPypWzyoy2G3l90v9k64kjphl/ZJjPIShFztenRomi8WTg=="], + "color-name": ["color-name@1.1.4", "", {}, "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA=="], "commander": ["commander@15.0.0", "", {}, "sha512-z67u4ZhzCL/Tydu1lJARtEZYWbWaN7oYLHbsuzocr6y4N6WZAagG3RQ4FW61V1/0+jImpj293XfrcYnd1qxtPg=="], "compress-commons": ["compress-commons@6.0.2", "", { "dependencies": { "crc-32": "^1.2.0", "crc32-stream": "^6.0.0", "is-stream": "^2.0.1", "normalize-path": "^3.0.0", "readable-stream": "^4.0.0" } }, "sha512-6FqVXeETqWPoGcfzrXb37E50NP0LXT8kAMu5ooZayhWWdgEY4lBEEcbQNXtkuKQsGduxiIcI4gOTsxTmuq/bSg=="], - "convert-source-map": ["convert-source-map@2.0.0", "", {}, "sha512-Kvp459HrV2FEJ1CAsi1Ku+MY3kasH19TFykTz2xWmMeq6bk2NU3XXvfJ+Q61m0xktWwt+1HSYf3JZsTms3aRJg=="], - "core-util-is": ["core-util-is@1.0.3", "", {}, "sha512-ZQBvi1DcpJ4GDqanjucZ2Hj3wEO5pZDS89BWbkcrvdxksJorwUDDZamX9ldFkp9aw2lmBDLgkObEA4DWNJ9FYQ=="], "cpu-features": ["cpu-features@0.0.10", "", { "dependencies": { "buildcheck": "~0.0.6", "nan": "^2.19.0" } }, "sha512-9IkYqtX3YHPCzoVg1Py+o9057a3i0fp7S530UWokCSaFVTc7CwXPRiOjRjBQQ18ZCNafx78YfnG+HALxtVmOGA=="], @@ -243,12 +150,8 @@ "eastasianwidth": ["eastasianwidth@0.2.0", "", {}, "sha512-I88TYZWc9XiYHRQ4/3c5rjjfgkjhLyW2luGIheGERbNQ6OY7yTybanSpDXZa8y7VUP9YmDcYa+eyq4ca7iLqWA=="], - "electron-to-chromium": ["electron-to-chromium@1.5.417", "", {}, "sha512-4T+DTDWuMPM4aHlHwWdAVCVWwp7LDilnhzkj+c/Lbj91XSQrLuOmZSLtS9Q4iIqjlPUbPOnC624zDVVHCHaolQ=="], - "emoji-regex": ["emoji-regex@8.0.0", "", {}, "sha512-MSjYzcWNOA0ewAHpz0MxpYFvwg6yjy1NG3xteoqz644VCo/RPgnr1/GGt+ic3iJTzQ8Eu3TdM14SawnVUmGE6A=="], - "enabled": ["enabled@2.0.0", "", {}, "sha512-AKrN98kuwOzMIdAizXGI86UFBoo26CL21UM763y1h/GMSJ4/OHU9k2YlsmBpyScFo/wbLzWQJBMCW4+IO3/+OQ=="], - "end-of-stream": ["end-of-stream@1.4.5", "", { "dependencies": { "once": "^1.4.0" } }, "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg=="], "escalade": ["escalade@3.2.0", "", {}, "sha512-WUj2qlxaQtO4g6Pq5c29GTcWGDyd8itL8zTlipgECz3JesAiiOKotd8JU6otB3PACgG6xkJUyVhboMS+bje/jA=="], @@ -263,18 +166,10 @@ "fdir": ["fdir@6.5.0", "", { "peerDependencies": { "picomatch": "^3 || ^4" }, "optionalPeers": ["picomatch"] }, "sha512-tIbYtZbucOs0BRGqPJkshJUYdL+SDH7dVM8gjy+ERp3WAUjLEFJE+02kanyHtwjWOnwrKYBiwAmM0p4kLJAnXg=="], - "fecha": ["fecha@4.2.3", "", {}, "sha512-OP2IUU6HeYKJi3i0z4A19kHMQoLVs4Hc+DPqqxI2h/DPZHTm/vjsfC6P0b4jCMy14XizLBqvndQ+UilD7707Jw=="], - - "fill-range": ["fill-range@7.1.1", "", { "dependencies": { "to-regex-range": "^5.0.1" } }, "sha512-YsGpe3WHLK8ZYi4tWDg2Jy3ebRz2rXowDxnld4bkQB00cc/1Zw9AWnC0i9ztDJitivtQvaI9KaLyKrc+hBW0yg=="], - - "fn.name": ["fn.name@1.1.0", "", {}, "sha512-GRnmB5gPyJpAhTQdSZTSp9uaPSvl09KoYcMQtsB9rQoOmzs9dH6ffeccH+Z+cv6P68Hu5bC6JjRh4Ah/mHSNRw=="], - "foreground-child": ["foreground-child@3.3.1", "", { "dependencies": { "cross-spawn": "^7.0.6", "signal-exit": "^4.0.1" } }, "sha512-gIXjKqtFuWEgzFRJA9WCQeSJLZDjgJUOMCMzxtvFq/37KojM1BFGufqsCy0r4qSQmYLsZYMeyRqzIWOMup03sw=="], "fs-constants": ["fs-constants@1.0.0", "", {}, "sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow=="], - "gensync": ["gensync@1.0.0-beta.2", "", {}, "sha512-3hN7NaskYvMDLQY55gnW3NQ+mesEAepTqlg+VEbj7zzqEMBVNhzcGYYeqFo/TlYz6eQiFcp1HcsCZO+nGgS8zg=="], - "get-caller-file": ["get-caller-file@2.0.5", "", {}, "sha512-DyFP3BM/3YHTQOCUL/w0OZHR0lpKeGrxotcHWcqNEdnltqFwXVfhEBQ94eIo34AfQpo0rGki4cyIiftY06h2Fg=="], "get-port": ["get-port@5.1.1", "", {}, "sha512-g/Q1aTSDOxFpchXC4i8ZWvxA1lnPqx/JHqcpIw0/LX9T8x/GBbi6YnlN5nhaKIFkT8oFsscUKgDJYxfwfS6QsQ=="], @@ -289,8 +184,6 @@ "is-fullwidth-code-point": ["is-fullwidth-code-point@3.0.0", "", {}, "sha512-zymm5+u+sCsSWyD9qNaejV3DFvhCKclKdizYaJUuHA83RLjb7nSuGnddCHGv0hk+KY7BMAlsWeK4Ueg6EV6XQg=="], - "is-number": ["is-number@7.0.0", "", {}, "sha512-41Cifkg6e8TylSpdtTpeLVMqvSBEVzTttHvERD741+pnZ8ANv0004MRL43QKPDlK9cGvNp6NZWZUBlbGXYxxng=="], - "is-stream": ["is-stream@2.0.1", "", {}, "sha512-hFoiJiTl63nn+kstHGBtewWSKnQLpyb155KHheA1l39uvtO9nWIop1p3udqPcUd/xbF1VLMO4n7OI6p7RbngDg=="], "isarray": ["isarray@1.0.0", "", {}, "sha512-VLghIWNM6ELQzo7zwmcg0NmTVyWKYjvIeM83yjp0wRDTmUnrM678fQbcKBo6n2CJEF0szoG//ytg+TKla89ALQ=="], @@ -299,27 +192,15 @@ "jackspeak": ["jackspeak@3.4.3", "", { "dependencies": { "@isaacs/cliui": "^8.0.2" }, "optionalDependencies": { "@pkgjs/parseargs": "^0.11.0" } }, "sha512-OGlZQpz2yfahA/Rd1Y8Cd9SIEsqvXkLVoSw/cgwhnhFMDbsQFeZYoJJ7bIZBS9BcamUW96asq/npPWugM+RQBw=="], - "js-tokens": ["js-tokens@4.0.0", "", {}, "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ=="], - - "jsesc": ["jsesc@3.1.0", "", { "bin": { "jsesc": "bin/jsesc" } }, "sha512-/sM3dO2FOzXjKQhJuo0Q173wf2KOo8t4I8vHy6lF9poUp7bKT0/NHE8fPX23PwfhnykfqnC2xRxOnVw5XuGIaA=="], - - "json5": ["json5@2.2.3", "", { "bin": { "json5": "lib/cli.js" } }, "sha512-XmOWe7eyHYH14cLdVPoyg+GOH3rYX++KpzrylJwSW98t3Nk+U8XOl8FWKOgwtzdb8lXGf6zYwDUzeHMWfxasyg=="], - - "kuler": ["kuler@2.0.0", "", {}, "sha512-Xq9nH7KlWZmXAtodXDDRE7vs6DU1gTU8zYDHDiWLSip45Egwq3plLHzPn27NgvzL2r1LMPC1vdqh98sQxtqj4A=="], - "lazystream": ["lazystream@1.0.1", "", { "dependencies": { "readable-stream": "^2.0.5" } }, "sha512-b94GiNHQNy6JNTrt5w6zNyffMrNkXZb3KTkCZJb2V1xaEGCk093vkZ2jk3tpaeP33/OiXC+WvK9AxUebnf5nbw=="], "lodash": ["lodash@4.18.1", "", {}, "sha512-dMInicTPVE8d1e5otfwmmjlxkZoUpiVLwyeTdUsi/Caj/gfzzblBcCE5sRHV/AsjuCmxWrte2TNGSYuCeCq+0Q=="], "lodash.camelcase": ["lodash.camelcase@4.3.0", "", {}, "sha512-TwuEnCnxbc3rAvhf/LbG7tJUDzhqXyFnv3dtzLOPgCG/hODL7WFnsbwktkD7yUV0RrreP/l1PALq/YSg6VvjlA=="], - "logform": ["logform@2.7.0", "", { "dependencies": { "@colors/colors": "1.6.0", "@types/triple-beam": "^1.3.2", "fecha": "^4.2.0", "ms": "^2.1.1", "safe-stable-stringify": "^2.3.1", "triple-beam": "^1.3.0" } }, "sha512-TFYA4jnP7PVbmlBIfhlSe+WKxs9dklXMTEGcBCIvLhE/Tn3H6Gk1norupVW7m5Cnd4bLcr08AytbyV/xj7f/kQ=="], - "long": ["long@5.3.2", "", {}, "sha512-mNAgZ1GmyNhD7AuqnTG3/VQ26o760+ZYBPKjPvugO8+nLbYfX6TVpJPseBvopbdY+qpZ/lKUnmEc1LeZYS3QAA=="], - "lru-cache": ["lru-cache@5.1.1", "", { "dependencies": { "yallist": "^3.0.2" } }, "sha512-KpNARQA3Iwv+jTA0utUVVbrh+Jlrr1Fv0e56GGzAFOXN7dk/FviaDW8LHmK52DlcH4WP2n6gI8vN1aesBFgo9w=="], - - "micromatch": ["micromatch@4.0.8", "", { "dependencies": { "braces": "^3.0.3", "picomatch": "^2.3.1" } }, "sha512-PXwfBhYu0hBCPw8Dn0E+WDYb7af3dSLVWKi3HGv84IdF4TyFoC0ysxFd0Goxw7nSv4T/PzEJQxsYsEiFCKo2BA=="], + "lru-cache": ["lru-cache@10.4.3", "", {}, "sha512-JNAzZcXrCt42VGLuYz0zfAzDfAvJWW6AfYlDBQyDV5DClI2m5sAmK+OIO7s59XfsRsWHp02jAJrRadPRGTt6SQ=="], "minimatch": ["minimatch@10.2.5", "", { "dependencies": { "brace-expansion": "^5.0.5" } }, "sha512-MULkVLfKGYDFYejP07QOurDLLQpcjk7Fw+7jXS2R2czRQzR56yHRveU5NDJEOviH+hETZKSkIk5c+T23GjFUMg=="], @@ -339,14 +220,10 @@ "neo4j-driver-core": ["neo4j-driver-core@5.28.3", "", {}, "sha512-Jk+hAmjFmO5YzVH/U7FyKXigot9zmIfLz6SZQy0xfr4zfTE/S8fOYFOGqKQTHBE86HHOWH2RbTslbxIb+XtU2g=="], - "node-releases": ["node-releases@2.0.54", "", {}, "sha512-YHs7BmmcsdAI5Ozuf8JZo6PT0mv2GIWC9vMfvUC3dp65M8hn7Ux8CPL+2oBI7juNuj9d0ndhTcznq2ODBps9cQ=="], - "normalize-path": ["normalize-path@3.0.0", "", {}, "sha512-6eZs5Ls3WtCisHWp9S2GUy8dqkpGi4BVSz3GaqiE6ezub0512ESztXUwUB6C6IKbQkY2Pnb/mD4WYojCRwcwLA=="], "once": ["once@1.4.0", "", { "dependencies": { "wrappy": "1" } }, "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w=="], - "one-time": ["one-time@1.0.0", "", { "dependencies": { "fn.name": "1.x.x" } }, "sha512-5DXOiRKwuSEcQ/l0kGCF6Q3jcADFv5tSmRaJck/OqkVFcOzutB134KRSfF0xDrL39MNnqxbHBbUUcjZIhTgb2g=="], - "package-json-from-dist": ["package-json-from-dist@1.0.1", "", {}, "sha512-UEZIS3/by4OC8vL3P2dTXRETpebLI2NiI5vIrjaD/5UtrkFX/tNbwjTSRAGC/+7CAo2pIcBaRgWmcBBHcsaCIw=="], "path-browserify": ["path-browserify@1.0.1", "", {}, "sha512-b7uo2UCUOYZcnF/3ID0lulOJi/bafxa1xPe7ZPsammBSpjSWQkjNxlt635YGS2MiR9GjvuXCtz2emr3jbsz98g=="], @@ -355,9 +232,7 @@ "path-scurry": ["path-scurry@1.11.1", "", { "dependencies": { "lru-cache": "^10.2.0", "minipass": "^5.0.0 || ^6.0.2 || ^7.0.0" } }, "sha512-Xa4Nw17FS9ApQFJ9umLiJS4orGjm7ZzwUrwamcGQuHSzDyth9boKDaycYdDcZDuqYATXw4HFXgaqWTctW/v1HA=="], - "picocolors": ["picocolors@1.1.1", "", {}, "sha512-xceH2snhtb5M9liqDsmEw56le376mTZkEX/jEb/RxNFyegNul7eNslCXP9FDj/Lcu0X8KEyMceP2ntpaHrDEVA=="], - - "picomatch": ["picomatch@2.3.2", "", {}, "sha512-V7+vQEJ06Z+c5tSye8S+nHUfI51xoXIXjHQ99cQtKUkQqqO1kO/KCJUfZXuB47h/YBlDhah2H3hdUGXn8ie0oA=="], + "picomatch": ["picomatch@4.0.5", "", {}, "sha512-RvwwcruNjI1ncT5xRakeyS9Lf8lcItv34KD+aif+VH9kduAyfYBipGh12274xtenIPZ119/R9BdTBa8gAwSh0A=="], "process": ["process@0.11.10", "", {}, "sha512-cdGef/drWFoydD1JsMzuFf8100nZl+GT+yacc2bEced5f9Rjk4z+WtFUTBu9PhOi9j/jfmBPu0mMEY4wIdAF8A=="], @@ -371,7 +246,7 @@ "pump": ["pump@3.0.4", "", { "dependencies": { "end-of-stream": "^1.1.0", "once": "^1.3.1" } }, "sha512-VS7sjc6KR7e1ukRFhQSY5LM2uBWAUPiOPa/A3mkKmiMwSmRFUITt0xuj+/lesgnCv+dPIEYlkzrcyXgquIHMcA=="], - "readable-stream": ["readable-stream@3.6.2", "", { "dependencies": { "inherits": "^2.0.3", "string_decoder": "^1.1.1", "util-deprecate": "^1.0.1" } }, "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA=="], + "readable-stream": ["readable-stream@4.7.0", "", { "dependencies": { "abort-controller": "^3.0.0", "buffer": "^6.0.3", "events": "^3.3.0", "process": "^0.11.10", "string_decoder": "^1.3.0" } }, "sha512-oIGGmcpTLwPga8Bn6/Z75SVaH1z5dUut2ibSyAMVhmUggWpmDn2dapB0n7f8nwaSiRtepAsfJyfXIO5DCVAODg=="], "readdir-glob": ["readdir-glob@1.1.3", "", { "dependencies": { "minimatch": "^5.1.0" } }, "sha512-v05I2k7xN8zXvPD9N+z/uhXPaj0sUFCe2rcWZIpBsqxfP7xXFQ0tipAd/wjj1YxWyWtUS5IDJpOG82JKt2EAVA=="], @@ -383,12 +258,8 @@ "safe-buffer": ["safe-buffer@5.2.1", "", {}, "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ=="], - "safe-stable-stringify": ["safe-stable-stringify@2.5.0", "", {}, "sha512-b3rppTKm9T+PsVCBEOUR46GWI7fdOs00VKZ1+9c1EWDaDMvjQc6tUwuFyIprgGgTcWoVHSKrU8H31ZHA2e0RHA=="], - "safer-buffer": ["safer-buffer@2.1.2", "", {}, "sha512-YZo3K82SD7Riyi0E1EQPojLz7kpepnSQI9IyPbHHg1XXXevb5dJI7tpyN2ADxGcQbHG7vcyRHk0cbwqcQriUtg=="], - "semver": ["semver@7.8.5", "", { "bin": { "semver": "bin/semver.js" } }, "sha512-Y7/KDsb8LjooZpwaqGyulO6DQlksgCncchHGk+sZIY4SBvUocMBEFH5Ur1fI4dV+Jvl0w6cjvucaIi40puRioA=="], - "shebang-command": ["shebang-command@2.0.0", "", { "dependencies": { "shebang-regex": "^3.0.0" } }, "sha512-kHxr2zZpYtdmrN1qDjrrX/Z1rR1kG8Dx+gkpK1G4eXmvXswmcE1hTWBWYUzlraYw1/yZp6YuDY77YtvbN0dmDA=="], "shebang-regex": ["shebang-regex@3.0.0", "", {}, "sha512-7++dFhtcx3353uBaq8DDR4NuxBetBzC7ZQOhmTQInHEd6bSrXdiEyzCvG07Z44UYdLShWUyXt5M/yhz8ekcb1A=="], @@ -401,8 +272,6 @@ "ssh2": ["ssh2@1.17.0", "", { "dependencies": { "asn1": "^0.2.6", "bcrypt-pbkdf": "^1.0.2" }, "optionalDependencies": { "cpu-features": "~0.0.10", "nan": "^2.23.0" } }, "sha512-wPldCk3asibAjQ/kziWQQt1Wh3PgDFpC0XpwclzKcdT1vql6KeYxf5LIt4nlFkUeR8WuphYMKqUA56X4rjbfgQ=="], - "stack-trace": ["stack-trace@0.0.10", "", {}, "sha512-KGzahc7puUKkzyMt+IqAep+TVNbKP+k2Lmwhub39m1AsTSkaDutx56aDCo+HLDzf/D26BIHTJWNiTG1KAJiQCg=="], - "streamx": ["streamx@2.28.0", "", { "dependencies": { "events-universal": "^1.0.0", "fast-fifo": "^1.3.2", "text-decoder": "^1.1.0" } }, "sha512-1Yowhzjf0ivGMrTIkY9hav5TxobO9qIVqUE41fiCGMGgc3CLlf4MY+9AHmZqBWgDTue0fY9zWjYFVyf6Diuobw=="], "string-width": ["string-width@4.2.3", "", { "dependencies": { "emoji-regex": "^8.0.0", "is-fullwidth-code-point": "^3.0.0", "strip-ansi": "^6.0.1" } }, "sha512-wKyQRQpjJ0sIp62ErSZdGsjMJWsap5oRNihHhu6G7JVO/9jIB6UyevL+tXuOqrng8j/cxKTWyWUwvSTriiZz/g=="], @@ -411,8 +280,6 @@ "string_decoder": ["string_decoder@1.3.0", "", { "dependencies": { "safe-buffer": "~5.2.0" } }, "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA=="], - "stringify2stream": ["stringify2stream@1.1.0", "", {}, "sha512-69LPWdFoBFzPug93gJStxiBGaahUglPSomITXi9umiLhcyxEx1oyW27qcw5A7QVp9xwBygcAQJzkvoCZUSfrAw=="], - "strip-ansi": ["strip-ansi@6.0.1", "", { "dependencies": { "ansi-regex": "^5.0.1" } }, "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A=="], "strip-ansi-cjs": ["strip-ansi@6.0.1", "", { "dependencies": { "ansi-regex": "^5.0.1" } }, "sha512-Y38VPSHcqkFrCpFnQ9vuSXmquuv5oXOKpGeT6aGrr3o3Gc9AlVa6JBfUSOCnbxGGZF+/0ooI7KrPuUSztUdU5A=="], @@ -427,16 +294,10 @@ "text-decoder": ["text-decoder@1.2.7", "", { "dependencies": { "b4a": "^1.6.4" } }, "sha512-vlLytXkeP4xvEq2otHeJfSQIRyWxo/oZGEbXrtEEF9Hnmrdly59sUbzZ/QgyWuLYHctCHxFF4tRQZNQ9k60ExQ=="], - "text-hex": ["text-hex@1.0.0", "", {}, "sha512-uuVGNWzgJ4yhRaNSiubPY7OjISw4sw4E5Uv0wbjp+OzcbmVU/rsT8ujgcXJhn9ypzsgr5vlzpPqP+MBBKcGvbg=="], - "tinyglobby": ["tinyglobby@0.2.17", "", { "dependencies": { "fdir": "^6.5.0", "picomatch": "^4.0.4" } }, "sha512-wXR/dYpcqKmfWpEdZjiKJOwCNFndD0DMnrW/cYjVGttEkBfVgcLFHoNrlj47mjOVic9yyNu65alsgF4NQyTa2g=="], "tmp": ["tmp@0.2.7", "", {}, "sha512-e0votIpp4Uo2AJYSzVHV6xCcawuiez3DzqDAbrTc3YxBkplN6e+dM13ZeIcZnDg/QpSuU2zfZ3rzwY8ukEnaXw=="], - "to-regex-range": ["to-regex-range@5.0.1", "", { "dependencies": { "is-number": "^7.0.0" } }, "sha512-65P7iz6X5yEr1cwcgvQxbbIw7Uk3gOy5dIdtZ4rDveLqhrdJP+Li/Hx6tyK0NEb+2GCyneCMJiGqrADCSNk8sQ=="], - - "triple-beam": ["triple-beam@1.4.1", "", {}, "sha512-aZbgViZrg1QNcG+LULa7nhZpJTZSLm/mXnHXnbAbjmN5aSa0y7V+wvv6+4WaBtpISJzThKy+PIPxc1Nq1EJ9mg=="], - "ts-morph": ["ts-morph@28.0.0", "", { "dependencies": { "@ts-morph/common": "~0.29.0", "code-block-writer": "^13.0.3" } }, "sha512-Wp3tnZ2bzwxyTZMtgWVzXDfm7lB1Drz+y9DmmYH/L702PQhPyVrp3pkou3yIz4qjS14GY9kcpmLiOOMvl8oG1g=="], "tslib": ["tslib@2.8.1", "", {}, "sha512-oJFu94HQb+KVduSUQL7wnpmqnfmLsOA/nAh6b6EH0wCEoK0/mPeXU6c3wKDV83MkOuHPRHtSXKKU99IBazS/2w=="], @@ -449,16 +310,10 @@ "undici-types": ["undici-types@7.24.6", "", {}, "sha512-WRNW+sJgj5OBN4/0JpHFqtqzhpbnV0GuB+OozA9gCL7a993SmU+1JBZCzLNxYsbMfIeDL+lTsphD5jN5N+n0zg=="], - "update-browserslist-db": ["update-browserslist-db@1.3.2", "", { "dependencies": { "escalade": "^3.2.0", "picocolors": "^1.1.1" }, "peerDependencies": { "browserslist": ">= 4.21.0" }, "bin": { "update-browserslist-db": "cli.js" } }, "sha512-UQ+MSxlhRm1bzjhU+DcuXfjFO1FzNtqhK5+9Yvlp90ItDLk5vT932A0rFu619nf7RVS+Y/VeaUW1jaRDqZ8VJw=="], - "util-deprecate": ["util-deprecate@1.0.2", "", {}, "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw=="], "which": ["which@2.0.2", "", { "dependencies": { "isexe": "^2.0.0" }, "bin": { "node-which": "./bin/node-which" } }, "sha512-BLI3Tl1TW3Pvl70l3yq3Y64i+awpwXqsGBYWkkqMtnbXgrMD+yj7rhW0kuEDxzJaYXGjEW5ogapKNMEKNMjibA=="], - "winston": ["winston@3.19.0", "", { "dependencies": { "@colors/colors": "^1.6.0", "@dabh/diagnostics": "^2.0.8", "async": "^3.2.3", "is-stream": "^2.0.0", "logform": "^2.7.0", "one-time": "^1.0.0", "readable-stream": "^3.4.0", "safe-stable-stringify": "^2.3.1", "stack-trace": "0.0.x", "triple-beam": "^1.3.0", "winston-transport": "^4.9.0" } }, "sha512-LZNJgPzfKR+/J3cHkxcpHKpKKvGfDZVPS4hfJCc4cCG0CgYzvlD6yE/S3CIL/Yt91ak327YCpiF/0MyeZHEHKA=="], - - "winston-transport": ["winston-transport@4.9.0", "", { "dependencies": { "logform": "^2.7.0", "readable-stream": "^3.6.2", "triple-beam": "^1.3.0" } }, "sha512-8drMJ4rkgaPo1Me4zD/3WLfI/zPdA9o2IipKODunnGDcuqbHwjsbB79ylv04LCGGzU0xQ6vTznOMpQGaLhhm6A=="], - "wrap-ansi": ["wrap-ansi@7.0.0", "", { "dependencies": { "ansi-styles": "^4.0.0", "string-width": "^4.1.0", "strip-ansi": "^6.0.0" } }, "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q=="], "wrap-ansi-cjs": ["wrap-ansi@7.0.0", "", { "dependencies": { "ansi-styles": "^4.0.0", "string-width": "^4.1.0", "strip-ansi": "^6.0.0" } }, "sha512-YVGIj2kamLSTxw6NsZjoBxfSwsn0ycdesmc4p+Q21c5zPuZ1pl+NfxVdxPtdHvmNVOQ6XSYG4AUtyt/Fi7D16Q=="], @@ -467,8 +322,6 @@ "y18n": ["y18n@5.0.8", "", {}, "sha512-0pfFzegeDWJHJIAmTLRP2DwHjdF5s7jo9tuztdQxAhINCdvS+3nGINqPd00AphqJR/0LhANUS6/+7SCb98YOfA=="], - "yallist": ["yallist@3.1.1", "", {}, "sha512-a4UGQaWPH59mOXUYnAG2ewncQS4i4F43Tv3JoAM+s2VDAmS9NsK8GpDMLrCHPksFT7h3K6TOoUNn2pb7RoXx4g=="], - "yaml": ["yaml@2.9.0", "", { "bin": { "yaml": "bin.mjs" } }, "sha512-2AvhNX3mb8zd6Zy7INTtSpl1F15HW6Wnqj0srWlkKLcpYl/gMIMJiyuGq2KeI2YFxUPjdlB+3Lc10seMLtL4cA=="], "yargs": ["yargs@17.7.3", "", { "dependencies": { "cliui": "^8.0.1", "escalade": "^3.1.1", "get-caller-file": "^2.0.5", "require-directory": "^2.1.1", "string-width": "^4.2.3", "y18n": "^5.0.5", "yargs-parser": "^21.1.1" } }, "sha512-GZtjxm/J/4TSxuL3FNYjCmLktBTnIw/rVmKSIyKeYAZpmJB2ig9VauCC5xsa82GNKVKDAqpOn3KVzNt0zmrU0g=="], @@ -477,16 +330,6 @@ "zip-stream": ["zip-stream@6.0.1", "", { "dependencies": { "archiver-utils": "^5.0.0", "compress-commons": "^6.0.2", "readable-stream": "^4.0.0" } }, "sha512-zK7YHHz4ZXpW89AHXUPbQVGKI7uvkd3hzusTdotCg1UxyaVtg0zFJSTfW/Dq5f7OBBVnq6cZIaC8Ti4hb6dtCA=="], - "@babel/core/semver": ["semver@6.3.1", "", { "bin": { "semver": "bin/semver.js" } }, "sha512-BR7VvDCVHO+q2xBEWskxS6DJE1qRnb7DxzUrogb71CWoSficBxYsiAGd+Kl0mmq/MprG9yArRkyrQxTO6XjMzA=="], - - "@babel/helper-compilation-targets/semver": ["semver@6.3.1", "", { "bin": { "semver": "bin/semver.js" } }, "sha512-BR7VvDCVHO+q2xBEWskxS6DJE1qRnb7DxzUrogb71CWoSficBxYsiAGd+Kl0mmq/MprG9yArRkyrQxTO6XjMzA=="], - - "@babel/helper-create-class-features-plugin/semver": ["semver@6.3.1", "", { "bin": { "semver": "bin/semver.js" } }, "sha512-BR7VvDCVHO+q2xBEWskxS6DJE1qRnb7DxzUrogb71CWoSficBxYsiAGd+Kl0mmq/MprG9yArRkyrQxTO6XjMzA=="], - - "@cs-au-dk/jelly/commander": ["commander@9.5.0", "", {}, "sha512-KRs7WVDKg86PWiuAqhDrAQnTXZKraVcCc6vFdL14qrZ/DcWwuRo7VoiYXalXO7S5GKpqYiVEwCbgFDfxNHKJBQ=="], - - "@cs-au-dk/jelly/typescript": ["typescript@5.9.3", "", { "bin": { "tsc": "bin/tsc", "tsserver": "bin/tsserver" } }, "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw=="], - "@grpc/grpc-js/@grpc/proto-loader": ["@grpc/proto-loader@0.8.1", "", { "dependencies": { "lodash.camelcase": "^4.3.0", "long": "^5.0.0", "protobufjs": "^7.5.5", "yargs": "^17.7.2" }, "bin": { "proto-loader-gen-types": "build/bin/proto-loader-gen-types.js" } }, "sha512-wtF6h+DY6M3YaDBPAmvuuA6jV8Sif9MjtOI5euKFWRgCDl5PeDpPsHR9u2l6St5ceY8AZgoNDww5+HvEsXFsGg=="], "@isaacs/cliui/string-width": ["string-width@5.1.2", "", { "dependencies": { "eastasianwidth": "^0.2.0", "emoji-regex": "^9.2.2", "strip-ansi": "^7.0.1" } }, "sha512-HnLOCR3vjcY8beoNLtcjZ5/nxn2afmME6lhrDrebokqMap+XbeW8n9TXpPDOqdGK5qcI3oT0GKTW6wC7EMiVqA=="], @@ -495,44 +338,28 @@ "@isaacs/cliui/wrap-ansi": ["wrap-ansi@8.1.0", "", { "dependencies": { "ansi-styles": "^6.1.0", "string-width": "^5.0.1", "strip-ansi": "^7.0.1" } }, "sha512-si7QWI6zUMq56bESFvagtmzMdGOtoxfR+Sez11Mobfc7tm+VkUckk9bW2UeffTGVUbOksxmSw0AA2gs8g71NCQ=="], - "ansi-styles/color-convert": ["color-convert@2.0.1", "", { "dependencies": { "color-name": "~1.1.4" } }, "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ=="], - - "archiver/readable-stream": ["readable-stream@4.7.0", "", { "dependencies": { "abort-controller": "^3.0.0", "buffer": "^6.0.3", "events": "^3.3.0", "process": "^0.11.10", "string_decoder": "^1.3.0" } }, "sha512-oIGGmcpTLwPga8Bn6/Z75SVaH1z5dUut2ibSyAMVhmUggWpmDn2dapB0n7f8nwaSiRtepAsfJyfXIO5DCVAODg=="], - - "archiver-utils/readable-stream": ["readable-stream@4.7.0", "", { "dependencies": { "abort-controller": "^3.0.0", "buffer": "^6.0.3", "events": "^3.3.0", "process": "^0.11.10", "string_decoder": "^1.3.0" } }, "sha512-oIGGmcpTLwPga8Bn6/Z75SVaH1z5dUut2ibSyAMVhmUggWpmDn2dapB0n7f8nwaSiRtepAsfJyfXIO5DCVAODg=="], - "bl/buffer": ["buffer@5.7.1", "", { "dependencies": { "base64-js": "^1.3.1", "ieee754": "^1.1.13" } }, "sha512-EHcyIPBQ4BSGlvjB16k5KgAJ27CIsHY/2JBmCRReo48y9rQ3MaUzWX3KVlBa4U7MyX02HdVj0K7C3WaB3ju7FQ=="], - "compress-commons/readable-stream": ["readable-stream@4.7.0", "", { "dependencies": { "abort-controller": "^3.0.0", "buffer": "^6.0.3", "events": "^3.3.0", "process": "^0.11.10", "string_decoder": "^1.3.0" } }, "sha512-oIGGmcpTLwPga8Bn6/Z75SVaH1z5dUut2ibSyAMVhmUggWpmDn2dapB0n7f8nwaSiRtepAsfJyfXIO5DCVAODg=="], + "bl/readable-stream": ["readable-stream@3.6.2", "", { "dependencies": { "inherits": "^2.0.3", "string_decoder": "^1.1.1", "util-deprecate": "^1.0.1" } }, "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA=="], - "crc32-stream/readable-stream": ["readable-stream@4.7.0", "", { "dependencies": { "abort-controller": "^3.0.0", "buffer": "^6.0.3", "events": "^3.3.0", "process": "^0.11.10", "string_decoder": "^1.3.0" } }, "sha512-oIGGmcpTLwPga8Bn6/Z75SVaH1z5dUut2ibSyAMVhmUggWpmDn2dapB0n7f8nwaSiRtepAsfJyfXIO5DCVAODg=="], + "docker-modem/readable-stream": ["readable-stream@3.6.2", "", { "dependencies": { "inherits": "^2.0.3", "string_decoder": "^1.1.1", "util-deprecate": "^1.0.1" } }, "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA=="], "dockerode/tar-fs": ["tar-fs@2.1.5", "", { "dependencies": { "chownr": "^1.1.1", "mkdirp-classic": "^0.5.2", "pump": "^3.0.0", "tar-stream": "^2.1.4" } }, "sha512-OboTd8mmMhZDNPV+UjQcK9yKAatXu2aJ+r1w4im1Otd4M4fl2hwvdoXUxIYHFTHWK/3y3FarBP70v3vwmGlOxw=="], - "fdir/picomatch": ["picomatch@4.0.5", "", {}, "sha512-RvwwcruNjI1ncT5xRakeyS9Lf8lcItv34KD+aif+VH9kduAyfYBipGh12274xtenIPZ119/R9BdTBa8gAwSh0A=="], - "foreground-child/signal-exit": ["signal-exit@4.1.0", "", {}, "sha512-bzyZ1e88w9O1iNJbKnOlvYTrWPDl46O1bG0D3XInv+9tkPrxrN8jUUTiFlDkkmKWgn1M6CfIA13SuGqOa9Korw=="], "glob/minimatch": ["minimatch@9.0.9", "", { "dependencies": { "brace-expansion": "^2.0.2" } }, "sha512-OBwBN9AL4dqmETlpS2zasx+vTeWclWzkblfZk7KTA5j3jeOONz/tRCnZomUyvNg83wL5Zv9Ss6HMJXAgL8R2Yg=="], "lazystream/readable-stream": ["readable-stream@2.3.8", "", { "dependencies": { "core-util-is": "~1.0.0", "inherits": "~2.0.3", "isarray": "~1.0.0", "process-nextick-args": "~2.0.0", "safe-buffer": "~5.1.1", "string_decoder": "~1.1.1", "util-deprecate": "~1.0.1" } }, "sha512-8p0AUk4XODgIewSi0l8Epjs+EVnWiK7NoDIEGU0HhE7+ZyY8D1IMY7odu5lRrFXGg71L15KG8QrPmum45RTtdA=="], - "path-scurry/lru-cache": ["lru-cache@10.4.3", "", {}, "sha512-JNAzZcXrCt42VGLuYz0zfAzDfAvJWW6AfYlDBQyDV5DClI2m5sAmK+OIO7s59XfsRsWHp02jAJrRadPRGTt6SQ=="], - "readdir-glob/minimatch": ["minimatch@5.1.9", "", { "dependencies": { "brace-expansion": "^2.0.1" } }, "sha512-7o1wEA2RyMP7Iu7GNba9vc0RWWGACJOCZBJX2GJWip0ikV+wcOsgVuY9uE8CPiyQhkGFSlhuSkZPavN7u1c2Fw=="], - "tinyglobby/picomatch": ["picomatch@4.0.5", "", {}, "sha512-RvwwcruNjI1ncT5xRakeyS9Lf8lcItv34KD+aif+VH9kduAyfYBipGh12274xtenIPZ119/R9BdTBa8gAwSh0A=="], - - "zip-stream/readable-stream": ["readable-stream@4.7.0", "", { "dependencies": { "abort-controller": "^3.0.0", "buffer": "^6.0.3", "events": "^3.3.0", "process": "^0.11.10", "string_decoder": "^1.3.0" } }, "sha512-oIGGmcpTLwPga8Bn6/Z75SVaH1z5dUut2ibSyAMVhmUggWpmDn2dapB0n7f8nwaSiRtepAsfJyfXIO5DCVAODg=="], - "@isaacs/cliui/string-width/emoji-regex": ["emoji-regex@9.2.2", "", {}, "sha512-L18DaJsXSUk2+42pv8mLs5jJT2hqFkFE4j21wOmgbUqsZ2hL72NsUU785g9RXgo3s0ZNgVl42TiHp3ZtOv/Vyg=="], "@isaacs/cliui/strip-ansi/ansi-regex": ["ansi-regex@6.2.2", "", {}, "sha512-Bq3SmSpyFHaWjPk8If9yc6svM8c56dB5BAtW4Qbw5jHTwwXXcTLoRMkpDJp6VL0XzlWaCHTXrkFURMYmD0sLqg=="], "@isaacs/cliui/wrap-ansi/ansi-styles": ["ansi-styles@6.2.3", "", {}, "sha512-4Dj6M28JB+oAH8kFkTLUo+a2jwOFkuqb3yucU0CANcRRUbxS0cP0nZYCGjcc3BNXwRIsUVmDGgzawme7zvJHvg=="], - "ansi-styles/color-convert/color-name": ["color-name@1.1.4", "", {}, "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA=="], - "dockerode/tar-fs/tar-stream": ["tar-stream@2.2.0", "", { "dependencies": { "bl": "^4.0.3", "end-of-stream": "^1.4.1", "fs-constants": "^1.0.0", "inherits": "^2.0.3", "readable-stream": "^3.1.1" } }, "sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ=="], "glob/minimatch/brace-expansion": ["brace-expansion@2.1.2", "", { "dependencies": { "balanced-match": "^1.0.0" } }, "sha512-w5JZcKgdhDOgOwm8H+KgbosopHMuGcl6qbulwjtz3SM7I7P3yW1eAjzMPLrIE+NQ9vjgANKHWeMHnrT0OXW1oA=="], @@ -543,6 +370,8 @@ "readdir-glob/minimatch/brace-expansion": ["brace-expansion@2.1.2", "", { "dependencies": { "balanced-match": "^1.0.0" } }, "sha512-w5JZcKgdhDOgOwm8H+KgbosopHMuGcl6qbulwjtz3SM7I7P3yW1eAjzMPLrIE+NQ9vjgANKHWeMHnrT0OXW1oA=="], + "dockerode/tar-fs/tar-stream/readable-stream": ["readable-stream@3.6.2", "", { "dependencies": { "inherits": "^2.0.3", "string_decoder": "^1.1.1", "util-deprecate": "^1.0.1" } }, "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA=="], + "glob/minimatch/brace-expansion/balanced-match": ["balanced-match@1.0.2", "", {}, "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw=="], "readdir-glob/minimatch/brace-expansion/balanced-match": ["balanced-match@1.0.2", "", {}, "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw=="], diff --git a/docs/design/plans/2026-08-30-artifact-layer-v130-parity.md b/docs/design/plans/2026-08-30-artifact-layer-v130-parity.md new file mode 100644 index 0000000..1ba9708 --- /dev/null +++ b/docs/design/plans/2026-08-30-artifact-layer-v130-parity.md @@ -0,0 +1,1844 @@ +# Repository-artifact layer (python v1.3.0 parity) — Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Bring codeanalyzer-typescript's repository-artifact layer to codeanalyzer-python v1.3.0 parity — never-drop inventory with a text-capture policy, lockfile transitives, the ConfigKey family, deployment-env namespaces, and the level-graded `config_use` edge. + +**Architecture:** Four spec units land as staged commits on `feat/issue-101-artifacts`, each placed with the pipeline stage whose output it consumes: file-derived data in `src/artifacts/`, the literal `config_use` tier in `src/semantic_analysis/` (needs resolved callees), the dataflow tiers in `src/dataflow/` (needs def-use). Env reads mint a new L1 `config_access` body node so `config_uses.src` is a uniform ordinal id at every level. + +**Tech Stack:** TypeScript, Bun (test runner + bundler), ts-morph, `yaml` (new dependency), Neo4j projection. + +**Spec:** `docs/design/specs/2026-08-30-artifact-layer-v130-parity.md` + +## Global Constraints + +- **`SCHEMA_VERSION` in `src/build/neo4j/schema.ts` must NOT move.** PR #103's bump to `"2.2.0"` reverts to `"2.1.0"`. All analyzers re-baseline at 2.0.0 later, cross-language. +- **Schema changes are additive only** — no existing field renamed, removed, or repurposed. +- **Determinism:** every emitted list is sorted by a stable key; no `Date.now()`, no hash-order iteration. Two consecutive default runs must produce byte-identical output. +- **Extraction parses the full on-disk text**, never the (possibly truncated) `source` stored on the node. +- **`sha256` and `size_bytes` are always the full file**, regardless of capture settings. +- **Never drop a file:** unmatched-but-decodable → `roles: ["unknown"]`; undecodable → `format: "binary"`, `source: ""`. +- **Overlay posture:** a parse failure never suppresses an artifact node; it sets `extraction: "partial"`. +- **`config_uses` is superset-monotonic** across levels (L2 ⊆ L3 ⊆ L4); **`config_reads` deliberately shrinks** as levels rise — assert both. +- Language-neutral graph nouns (`Artifact`, `Package`, `ConfigKey`) stay unprefixed; this analyzer's own claims (`TS_PROVIDES`, `TS_UNRESOLVED_IMPORT`, `TS_USES_CONFIG`) keep the `TS_` prefix. +- Run `bun test` and `bun run typecheck` before every commit; both must be green. + +**Implementation order note:** the spec labels the units A–D, but D (deployment-env) *produces* config keys that C (config_use) *consumes*. Tasks below therefore run **A → B → D → C**, so every C test has real bindable keys to match against. No spec content changes. + +--- + +## File Structure + +**Create:** +- `src/artifacts/configKeys.ts` — config-key extraction (env / JSONC / TOML / INI / properties), unit B +- `src/artifacts/yamlKeys.ts` — YAML flattening with real spans, unit B +- `src/artifacts/deployEnv.ts` — Dockerfile `ENV`/`ARG`, compose, k8s → bindable env keys, unit D +- `src/semantic_analysis/configUseRules.ts` — shipped detector table, unit C +- `src/semantic_analysis/configUse.ts` — literal tier + `config_reads`, unit C +- `src/dataflow/configUse.ts` — intra + interprocedural dataflow tiers, unit C +- `docs/skills/analyzing-cants-graphs/SKILL.md` + `references/vocabulary.md` + `references/analyses.md` +- `test/config-keys.test.ts`, `test/config-use.test.ts` + +**Modify:** +- `src/schema/schema.ts` — `TSConfigKey`, `TSConfigUse`, `TSConfigRead`, `TSConfigAccess`; artifact gains `text_truncated` + `config_keys`; callable gains internal `config_accesses`; application gains `config_uses` + `config_reads` +- `src/schema/ids.ts` — `configKeyIdOf` +- `src/schema/assignIds.ts` — stamp config-key ids +- `src/schema/l1Body.ts` — emit `config_access` body nodes +- `src/schema/emit.ts` — new root sections +- `src/artifacts/index.ts` — never-drop walk, text policy, key attachment +- `src/artifacts/deps.ts` — `direct: false` transitives +- `src/syntactic_analysis/builders.ts` — record env-read accesses +- `src/options/options.ts`, `src/cli.ts` — `--artifact-text`, `--artifact-text-max-bytes` +- `src/core.ts` — pipeline placement of the tiers +- `src/build/neo4j/schema.ts`, `src/build/neo4j/project.ts` — ConfigKey vocabulary; revert the version bump +- `test/artifacts.test.ts`, `test/schema-v2.test.ts`, `test/neo4j-schema.test.ts` — gates +- `test/fixtures/artifacts-app/**` — fixture growth +- `CLAUDE.md`, `README.md`, `.claude/SCHEMA_DECISIONS.md` + +--- + +### Task 1: Text-capture policy (unit A) + +**Files:** +- Modify: `src/options/options.ts`, `src/cli.ts`, `src/artifacts/index.ts`, `src/schema/schema.ts` +- Test: `test/artifacts.test.ts` + +**Interfaces:** +- Consumes: `AnalysisOptions`, `TSArtifact` (both exist) +- Produces: `AnalysisOptions.artifactText?: boolean`, `AnalysisOptions.artifactTextMaxBytes?: number`, `DEFAULT_ARTIFACT_TEXT_MAX_BYTES = 262144`, `TSArtifact.text_truncated: boolean` + +- [ ] **Step 1: Write the failing test** + +In `test/artifacts.test.ts`, inside the `level-invariance + determinism` describe block: + +```ts + test("text capture: on by default, truncates under the cap, hash stays full-file", async () => { + const full = (await analyze(options())).application.application.artifacts["README.md"]; + expect(full?.source.length).toBeGreaterThan(0); + expect(full?.text_truncated).toBe(false); + + const capped = (await analyze(options({ artifactTextMaxBytes: 8 }))).application.application.artifacts["README.md"]; + expect(capped?.text_truncated).toBe(true); + expect(capped!.source.length).toBeLessThanOrEqual(8); + expect(capped?.sha256).toBe(full?.sha256); // hash is of the FULL file + expect(capped?.size_bytes).toBe(full?.size_bytes); + }); + + test("--no-artifact-text drops source but keeps inventory AND extraction", async () => { + const a = (await analyze(options({ artifactText: false }))).application.application; + expect(a.artifacts["package.json"]?.source).toBe(""); + expect(a.dependencies.find((d) => d.name === "express")?.locked_version).toBe("4.19.2"); + }); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `bun test test/artifacts.test.ts -t "text capture"` +Expected: FAIL — `text_truncated` is not a property; `artifactTextMaxBytes` is not on `AnalysisOptions`. + +- [ ] **Step 3: Add the options** + +In `src/options/options.ts`, above `export interface AnalysisOptions`: + +```ts +/** Default per-file byte cap for captured artifact text (256 KiB, python v1.3.0 parity). */ +export const DEFAULT_ARTIFACT_TEXT_MAX_BYTES = 256 * 1024; +``` + +and inside the interface, next to `resolveInstalled`: + +```ts + /** Capture verbatim artifact text into `source` (default true). */ + artifactText?: boolean; + /** Per-file byte cap for captured text; larger files store a flagged prefix. */ + artifactTextMaxBytes?: number; +``` + +- [ ] **Step 4: Wire the CLI flags** + +In `src/cli.ts`, add the import `import { DEFAULT_ARTIFACT_TEXT_MAX_BYTES } from "./options";`, then in `buildProgram()` before `-c, --cache-dir`: + +```ts + .option("--no-artifact-text", "keep the artifact inventory but drop captured raw text") + .option( + "--artifact-text-max-bytes ", + "per-file byte cap for captured artifact text; larger files are truncated and flagged", + String(DEFAULT_ARTIFACT_TEXT_MAX_BYTES), + ) +``` + +and in the returned options object, next to `resolveInstalled`: + +```ts + artifactText: o.artifactText !== false, + artifactTextMaxBytes: Number(o.artifactTextMaxBytes ?? DEFAULT_ARTIFACT_TEXT_MAX_BYTES), +``` + +- [ ] **Step 5: Add the schema field** + +In `src/schema/schema.ts`, in `TSArtifact`, after `source`: + +```ts + text_truncated: boolean; // true when `source` is a prefix, not the full file +``` + +- [ ] **Step 6: Apply the policy in the walk** + +In `src/artifacts/index.ts`: import the default (`import { DEFAULT_ARTIFACT_TEXT_MAX_BYTES } from "../options";`), then replace the node construction's `source` line and add the flag. The decoded `text` stays the FULL text (extraction uses it); only the stored copy is capped: + +```ts + const capture = opts.artifactText ?? true; + const cap = opts.artifactTextMaxBytes ?? DEFAULT_ARTIFACT_TEXT_MAX_BYTES; + const stored = !capture || text === undefined ? "" : text.length > cap ? text.slice(0, cap) : text; + const node: TSArtifact = { + id: "", + kind: "artifact", + path: rel, + format: format as string, + roles: roles as string[], + size_bytes: raw.length, + sha256: sha256(raw), + source: stored, + text_truncated: capture && text !== undefined && text.length > cap, + extraction: "none", + config_keys: [], + }; +``` + +(`config_keys: []` is added now so Task 4 only fills it; declare the field in Task 4's schema step — if `tsc` complains here, add `config_keys: TSConfigKey[]` to `TSArtifact` in this task and leave the type empty until Task 4.) + +- [ ] **Step 7: Run tests** + +Run: `bun test test/artifacts.test.ts && bun run typecheck` +Expected: PASS. + +- [ ] **Step 8: Commit** + +```bash +git add src/options/options.ts src/cli.ts src/artifacts/index.ts src/schema/schema.ts test/artifacts.test.ts +git commit -m "feat(artifacts): text-capture policy — flags, cap, text_truncated (#101)" +``` + +--- + +### Task 2: Never-drop inventory (unit A) + +**Files:** +- Modify: `src/artifacts/index.ts` +- Test: `test/artifacts.test.ts`, `test/fixtures/artifacts-app/` + +**Interfaces:** +- Consumes: `matchRules(relPath)` from `src/artifacts/rules.ts` (exists) +- Produces: artifacts for every non-source file; `roles: ["unknown"]` and `format: "binary"` conventions + +- [ ] **Step 1: Grow the fixture** + +```bash +cd test/fixtures/artifacts-app +printf 'plain data, no rule matches this\n' > notes.dat +printf '\x00\x01\x02\xff\xfe binary\n' > logo.bin +``` + +- [ ] **Step 2: Write the failing test** + +In `test/artifacts.test.ts`, inside the first describe block: + +```ts + test("never drops: unmatched files are unknown-role, binaries are hash-only", () => { + const unknown = arts["notes.dat"]; + expect(unknown?.roles).toEqual(["unknown"]); + expect(unknown?.format).toBe("text"); + expect(unknown?.source.length).toBeGreaterThan(0); + + const bin = arts["logo.bin"]; + expect(bin?.format).toBe("binary"); + expect(bin?.roles).toEqual(["unknown"]); + expect(bin?.source).toBe(""); + expect(bin?.sha256.length).toBe(64); + expect(bin?.size_bytes).toBeGreaterThan(0); + }); +``` + +- [ ] **Step 3: Run test to verify it fails** + +Run: `bun test test/artifacts.test.ts -t "never drops"` +Expected: FAIL — `arts["notes.dat"]` is undefined (the walk skips unmatched files). + +- [ ] **Step 4: Replace the skip with a fallback** + +In `src/artifacts/index.ts`, replace the `if (!matched) { … continue; }` block with: + +```ts + if (!matched) { + // Never drop: a file without a rule is still inventoried. Extensionless shebang files are + // scripts; everything else decodable is `unknown`; undecodable bytes are hash-only. + const probe = decodeLossy(raw); + if (path.extname(base) === "" && raw.subarray(0, 2).toString("utf-8") === "#!") { + format = "text"; + roles = ["script"]; + } else if (probe === undefined) { + format = "binary"; + roles = ["unknown"]; + } else { + format = "text"; + roles = ["unknown"]; + } + } +``` + +- [ ] **Step 5: Run tests** + +Run: `bun test && bun run typecheck` +Expected: PASS. If a count assertion in `test/schema-v2.test.ts` fails, it is counting artifact rows — update the expected number, do not weaken the assertion. + +- [ ] **Step 6: Commit** + +```bash +git add src/artifacts/index.ts test/artifacts.test.ts test/fixtures/artifacts-app +git commit -m "feat(artifacts): never-drop inventory — unknown roles, binary hash-only (#101)" +``` + +--- + +### Task 3: Lockfile transitives (unit A) + +**Files:** +- Modify: `src/artifacts/deps.ts`, `src/artifacts/index.ts` +- Test: `test/artifacts.test.ts` + +**Interfaces:** +- Consumes: `readLock(fileName, text)` → `Record`, `applyLockVersions(deps, lock)` (both exist) +- Produces: `transitiveRecords(pins, declaredNames, lockArtifactId)` → `TSDependency[]` + +- [ ] **Step 1: Write the failing test** + +In `test/artifacts.test.ts`, inside the dependencies describe block: + +```ts + test("lock-only packages become direct:false records attributed to the lock", () => { + const t = deps.find((d) => d.name === "lockonly-transitive"); + expect(t).toBeDefined(); + expect(t?.direct).toBe(false); + expect(t?.kind).toBe("runtime"); + expect(t?.prov).toEqual(["lockfile"]); + expect(t?.locked_version).toBe("1.0.0"); + expect(t?.declared_in).toBe("can://artifact/artifacts-app/package-lock.json"); + // declared packages stay direct + expect(byName.get("express")?.direct).toBe(true); + // nested shadow entries are NOT records + expect(deps.some((d) => d.name === "transitive-shadow")).toBe(false); + }); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `bun test test/artifacts.test.ts -t "lock-only"` +Expected: FAIL — no record for `lockonly-transitive`. + +- [ ] **Step 3: Add the record builder** + +Append to `src/artifacts/deps.ts`: + +```ts +/** + * Lock-only packages are TRANSITIVE: pinned with no manifest declaration. They earn records + * (`direct: false`) because the dependency SURFACE and the dependency SUPPLY CHAIN are different + * questions — a vulnerable package four levels down ships whether or not anyone named it. + * `kind` is "runtime": a lock does not record why a package is present, and inferring it would + * take a whole-graph walk this unit deliberately does not do. + */ +export function transitiveRecords( + pins: Record, + declaredNames: Set, + lockArtifactId: string, +): TSDependency[] { + const out: TSDependency[] = []; + for (const name of Object.keys(pins).sort()) { + if (declaredNames.has(name)) continue; + out.push({ + name, + spec: "", + kind: "runtime", + extras: [], + declared_in: lockArtifactId, + direct: false, + locked_version: pins[name] as string, + provides_imports: [name], + prov: ["lockfile"], + }); + } + return out; +} +``` + +- [ ] **Step 4: Add `direct` to declared records** + +In `src/artifacts/deps.ts`, in `parsePackageJson`'s pushed object, after `declared_in`: + +```ts + direct: true, +``` + +and in `src/schema/schema.ts`, in `TSDependency`, after `declared_in`: + +```ts + direct: boolean; // false = lockfile-only transitive (no manifest declares it) +``` + +- [ ] **Step 5: Emit them from the walk** + +In `src/artifacts/index.ts`, import `transitiveRecords`, keep the lock artifact id alongside the pins, and after the manifest loop: + +```ts + const declaredNames = new Set(dependencies.map((d) => d.name)); + for (const [ownerRel, pins] of Object.entries(locks).sort(([a], [b]) => a.localeCompare(b))) { + const lockRel = lockPathOf[ownerRel] as string; + const lockArtifact = artifacts[lockRel]; + if (!lockArtifact) continue; + dependencies.push(...transitiveRecords(pins, declaredNames, lockRel)); // rel path; assignIds re-stamps + } +``` + +Record `lockPathOf[ownerRel] = rel` in the same branch that fills `locks[ownerRel]`. + +- [ ] **Step 6: Extend assignIds for lock attribution** + +`declared_in` currently re-stamps only manifest paths; the same call already converts any rel path to an artifact id, so no change is needed — verify with the test rather than assuming. + +- [ ] **Step 7: Run tests** + +Run: `bun test && bun run typecheck` +Expected: PASS. + +- [ ] **Step 8: Commit** + +```bash +git add src/artifacts/deps.ts src/artifacts/index.ts src/schema/schema.ts test/artifacts.test.ts +git commit -m "feat(artifacts): lockfile transitives as direct:false records (#101)" +``` + +--- + +### Task 4: ConfigKey model + flat/JSONC extraction (unit B) + +**Files:** +- Create: `src/artifacts/configKeys.ts`, `test/config-keys.test.ts` +- Modify: `src/schema/schema.ts`, `src/schema/ids.ts`, `src/schema/assignIds.ts`, `src/artifacts/index.ts` + +**Interfaces:** +- Consumes: `TSArtifact`, `TSSpan` +- Produces: `TSConfigKey`, `configKeyIdOf(artifactId, dotted)`, `extractConfigKeys(format, roles, text): TSConfigKey[]` + +- [ ] **Step 1: Grow the fixture** + +```bash +cd test/fixtures/artifacts-app +cat > tsconfig.json <<'EOF' +{ + // JSONC: comments and trailing commas are legal in tsconfig + "compilerOptions": { "strict": true, "target": "ES2022", }, + "include": ["src"], +} +EOF +``` + +- [ ] **Step 2: Write the failing test** + +Create `test/config-keys.test.ts`: + +```ts +import { describe, expect, test } from "bun:test"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { analyze } from "../src/core"; +import type { AnalysisOptions } from "../src/options"; + +const FIXTURE = path.resolve(import.meta.dir, "fixtures/artifacts-app"); +const opts = { + input: FIXTURE, output: null, emit: "json", appName: "artifacts-app", neo4jUri: null, + neo4jUser: "neo4j", neo4jPassword: "", neo4jDatabase: null, analysisLevel: 1, graphs: [], + graphFieldDepth: 3, jobs: 1, targetFiles: null, skipTests: true, eager: true, noBuild: true, + phantoms: true, cacheDir: fs.mkdtempSync(path.join(os.tmpdir(), "cants-keys-")), verbosity: 0, +} as AnalysisOptions; + +const arts = (await analyze(opts)).application.application.artifacts; +const keysOf = (p: string): Record => + Object.fromEntries((arts[p]?.config_keys ?? []).map((k) => [k.key, k])); + +describe("config keys — flat and JSON (#101 unit B)", () => { + test(".env keys land in the env namespace with refs and stripped quotes", () => { + const k = keysOf(".env"); + expect(k["PAYMENT_HOST"]?.value).toBe("https://pay.example.com"); + expect(k["PAYMENT_HOST"]?.namespace).toBe("env"); + expect(k["DB_URL"]?.references).toEqual(["env:PAYMENT_HOST"]); + expect(k["NODE_OPTIONS"]?.value).toBe("--max-old-space-size=4096"); + }); + + test("JSONC tsconfig parses despite comments and trailing commas", () => { + const k = keysOf("tsconfig.json"); + expect(k["compilerOptions.strict"]?.value).toBe(true); + expect(k["compilerOptions.target"]?.value).toBe("ES2022"); + expect(k["include.0"]?.value).toBe("src"); // arrays get numeric segments + expect(arts["tsconfig.json"]?.extraction).toBe("full"); + }); + + test("key ids chain off the artifact id", () => { + expect(keysOf(".env")["PAYMENT_HOST"]?.id).toBe("can://artifact/artifacts-app/.env@key/PAYMENT_HOST"); + }); +}); +``` + +- [ ] **Step 3: Run test to verify it fails** + +Run: `bun test test/config-keys.test.ts` +Expected: FAIL — `config_keys` is always `[]`. + +- [ ] **Step 4: Add the schema types and id helper** + +In `src/schema/schema.ts`, before `TSArtifact`: + +```ts +/** A configuration key flattened out of a config-bearing artifact (#101 unit B). */ +export interface TSConfigKey { + id: string; // `${artifactId}@key/${dotted}` — stamped per-run by assignIds + key: string; // dotted path; numeric segments for arrays ("services.web.ports.0") + namespace: string; // env|json|yaml|toml|ini|properties|dockerfile + value?: string | number | boolean; // present by default; absent under --no-artifact-text + span?: TSSpan; // best-effort: exact for json/yaml, line-based elsewhere + references: string[]; // recognized ${VAR}/$VAR tokens, deduped, in order +} +``` + +and in `TSArtifact`, after `extraction`: + +```ts + config_keys: TSConfigKey[]; // contained children; containment mirrors DEFINES_CONFIG +``` + +In `src/schema/ids.ts`: + +```ts +/** Config-key id: the owning artifact's id, `@key/`, then the dotted path. */ +export function configKeyIdOf(artifactId: string, dotted: string): string { + return `${artifactId}@key/${dotted}`; +} +``` + +In `src/schema/assignIds.ts`, inside the artifacts loop after `art.id = …`: + +```ts + for (const ck of art.config_keys) ck.id = configKeyIdOf(art.id, ck.key); +``` + +(import `configKeyIdOf` alongside `artifactIdOf`). + +- [ ] **Step 5: Write the extractor** + +Create `src/artifacts/configKeys.ts`: + +```ts +/** + * Config-key extraction (#101 unit B): a config-bearing artifact's text → flattened dotted keys. + * Pure overlay — every parser returns [] on failure so the artifact node survives (the caller + * marks `extraction: "partial"`). Parses the FULL on-disk text, never the stored `source`. + */ +import type { TSConfigKey, TSSpan } from "../schema"; + +const PLACEHOLDER = /\$\{([A-Za-z_][A-Za-z0-9_]*)\}|\$([A-Za-z_][A-Za-z0-9_]*)/g; + +export function referencesOf(value: unknown): string[] { + if (typeof value !== "string") return []; + const out: string[] = []; + for (const m of value.matchAll(PLACEHOLDER)) { + const name = m[1] ?? m[2]; + if (name && !out.includes(`env:${name}`)) out.push(`env:${name}`); + } + return out; +} + +const scalar = (v: unknown): v is string | number | boolean => + typeof v === "string" || typeof v === "number" || typeof v === "boolean"; + +export function keyNode(key: string, namespace: string, value: unknown, span?: TSSpan): TSConfigKey { + return { + id: "", + key, + namespace, + ...(scalar(value) ? { value } : {}), + ...(span ? { span } : {}), + references: referencesOf(value), + }; +} + +/** Strip `//` and block comments and trailing commas — tsconfig/rc files are JSONC. */ +export function parseJsonc(text: string): unknown { + const stripped = text + .replace(/"(?:[^"\\]|\\.)*"|\/\*[\s\S]*?\*\/|\/\/[^\n]*/g, (m) => (m.startsWith('"') ? m : "")) + .replace(/,\s*([}\]])/g, "$1"); + return JSON.parse(stripped); +} + +function flatten(doc: unknown, prefix: string, out: TSConfigKey[], ns: string, depth: number): void { + if (depth > 24 || doc === null || typeof doc !== "object") return; + const entries: Array<[string, unknown]> = Array.isArray(doc) + ? doc.map((v, i) => [String(i), v] as [string, unknown]) + : Object.entries(doc as Record); + for (const [k, v] of entries) { + const dotted = prefix ? `${prefix}.${k}` : k; + if (scalar(v)) out.push(keyNode(dotted, ns, v)); + else flatten(v, dotted, out, ns, depth + 1); + } +} + +const ENV_LINE = /^\s*(?:export\s+)?([A-Za-z_][A-Za-z0-9_.]*)\s*=\s*(.*?)\s*$/; + +export function parseEnvKeys(text: string): TSConfigKey[] { + const out: TSConfigKey[] = []; + const lines = text.split("\n"); + lines.forEach((line, i) => { + if (!line.trim() || line.trim().startsWith("#")) return; + const m = ENV_LINE.exec(line); + if (!m) return; + let value = m[2] as string; + if ((value.startsWith('"') && value.endsWith('"')) || (value.startsWith("'") && value.endsWith("'"))) { + value = value.slice(1, -1); + } + const span: TSSpan = { start: [i + 1, 1], end: [i + 1, line.length + 1], bytes: [0, 0] }; + out.push(keyNode(m[1] as string, "env", value, span)); + }); + return out; +} + +/** INI / .properties: `[section]` prefixes a dotted key space. */ +export function parseIniKeys(text: string, namespace: string): TSConfigKey[] { + const out: TSConfigKey[] = []; + let section = ""; + text.split("\n").forEach((line, i) => { + const t = line.trim(); + if (!t || t.startsWith("#") || t.startsWith(";")) return; + const sec = /^\[([^\]]+)\]$/.exec(t); + if (sec) { + section = sec[1] as string; + return; + } + const eq = t.indexOf("="); + if (eq <= 0) return; + const key = t.slice(0, eq).trim(); + const value = t.slice(eq + 1).trim(); + const span: TSSpan = { start: [i + 1, 1], end: [i + 1, line.length + 1], bytes: [0, 0] }; + out.push(keyNode(section ? `${section}.${key}` : key, namespace, value, span)); + }); + return out; +} + +/** Dispatch by artifact format. YAML is handled by yamlKeys.ts (Task 5). */ +export function extractConfigKeys(format: string, text: string): TSConfigKey[] { + switch (format) { + case "env": + return parseEnvKeys(text); + case "json": + case "jsonc": { + const out: TSConfigKey[] = []; + flatten(parseJsonc(text), "", out, "json", 0); + return out; + } + case "ini": + return parseIniKeys(text, "ini"); + case "properties": + return parseIniKeys(text, "properties"); + default: + return []; + } +} +``` + +- [ ] **Step 6: Attach keys in the walk** + +In `src/artifacts/index.ts`, import `extractConfigKeys`, and in the extraction loop (after the `package.json` branch), replace the config branch with: + +```ts + // Config keys: attempted for every namespace-eligible format; a throw means the file is + // config-shaped but unparseable → keep the node, mark partial (overlay posture). + if (["env", "json", "jsonc", "ini", "properties", "yaml"].includes(node.format)) { + try { + const keys = extractConfigKeys(node.format, text); + if (keys.length) { + node.config_keys = keys; + node.extraction = node.extraction === "none" ? "full" : node.extraction; + } + } catch { + node.extraction = "partial"; + } + } +``` + +Guard `value` on capture: after the loop, if `opts.artifactText === false`, delete every key's `value`. + +- [ ] **Step 7: Run tests** + +Run: `bun test test/config-keys.test.ts && bun run typecheck` +Expected: PASS. + +- [ ] **Step 8: Commit** + +```bash +git add src/artifacts/configKeys.ts src/schema/schema.ts src/schema/ids.ts src/schema/assignIds.ts src/artifacts/index.ts test/config-keys.test.ts test/fixtures/artifacts-app +git commit -m "feat(artifacts): ConfigKey family — env/JSONC/ini extraction, key ids (#101)" +``` + +--- + +### Task 5: YAML config keys (unit B) + +**Files:** +- Create: `src/artifacts/yamlKeys.ts` +- Modify: `package.json`, `src/artifacts/configKeys.ts`, `test/config-keys.test.ts`, `test/fixtures/artifacts-app/` + +**Interfaces:** +- Consumes: `keyNode`, `TSConfigKey` +- Produces: `parseYamlKeys(text): TSConfigKey[]` + +- [ ] **Step 1: Add the dependency and fixture** + +```bash +bun add yaml +cd test/fixtures/artifacts-app +cat > docker-compose.yml <<'EOF' +services: + web: + image: node:22 + ports: + - "3000:3000" + environment: + PAYMENT_HOST: https://pay.example.com + FEATURE_FLAG: "on" +EOF +``` + +- [ ] **Step 2: Write the failing test** + +Append to `test/config-keys.test.ts`: + +```ts +describe("config keys — YAML (#101 unit B)", () => { + test("nested maps and sequences flatten with numeric segments and real spans", () => { + const k = keysOf("docker-compose.yml"); + expect(k["services.web.image"]?.value).toBe("node:22"); + expect(k["services.web.ports.0"]?.value).toBe("3000:3000"); + expect(k["services.web.image"]?.namespace).toBe("yaml"); + expect(k["services.web.image"]?.span?.start[0]).toBeGreaterThan(0); + }); +}); +``` + +- [ ] **Step 3: Run test to verify it fails** + +Run: `bun test test/config-keys.test.ts -t "YAML"` +Expected: FAIL — no keys for the compose file. + +- [ ] **Step 4: Write the YAML flattener** + +Create `src/artifacts/yamlKeys.ts`: + +```ts +/** + * YAML config-key flattening (#101 unit B). Uses the `yaml` package's document AST so spans are + * real offsets and anchors/flow style/multiline scalars parse correctly — a hand-rolled subset + * would silently mis-parse them. Returns [] on a parse error (overlay posture). + */ +import { LineCounter, parseDocument, isMap, isSeq, isScalar, type Node as YamlNode } from "yaml"; +import { keyNode } from "./configKeys"; +import type { TSConfigKey, TSSpan } from "../schema"; + +export function parseYamlKeys(text: string): TSConfigKey[] { + const lc = new LineCounter(); + const doc = parseDocument(text, { lineCounter: lc, keepSourceTokens: false }); + if (doc.errors.length) return []; + const out: TSConfigKey[] = []; + const spanOf = (n: YamlNode): TSSpan | undefined => { + const r = n.range; + if (!r) return undefined; + const s = lc.linePos(r[0]); + const e = lc.linePos(r[1]); + return { start: [s.line, s.col], end: [e.line, e.col], bytes: [r[0], r[1]] }; + }; + const walk = (node: unknown, prefix: string, depth: number): void => { + if (depth > 24) return; + if (isMap(node)) { + for (const item of node.items) { + const k = isScalar(item.key) ? String(item.key.value) : String(item.key); + walk(item.value, prefix ? `${prefix}.${k}` : k, depth + 1); + } + } else if (isSeq(node)) { + node.items.forEach((item, i) => walk(item, `${prefix}.${i}`, depth + 1)); + } else if (isScalar(node)) { + const v = node.value; + if (typeof v === "string" || typeof v === "number" || typeof v === "boolean") { + out.push(keyNode(prefix, "yaml", v, spanOf(node as YamlNode))); + } + } + }; + walk(doc.contents, "", 0); + return out; +} +``` + +- [ ] **Step 5: Dispatch YAML** + +In `src/artifacts/configKeys.ts`, add `import { parseYamlKeys } from "./yamlKeys";` and a case: + +```ts + case "yaml": + return parseYamlKeys(text); +``` + +- [ ] **Step 6: Run tests** + +Run: `bun test && bun run typecheck && bun run build` +Expected: PASS, and the binary still compiles with the new dependency bundled. + +- [ ] **Step 7: Commit** + +```bash +git add package.json bun.lock src/artifacts/yamlKeys.ts src/artifacts/configKeys.ts test/config-keys.test.ts test/fixtures/artifacts-app +git commit -m "feat(artifacts): YAML config keys with real spans (#101)" +``` + +--- + +### Task 6: Deployment-env namespaces (unit D) + +**Files:** +- Create: `src/artifacts/deployEnv.ts` +- Modify: `src/artifacts/index.ts`, `test/config-keys.test.ts`, `test/fixtures/artifacts-app/Dockerfile` + +**Interfaces:** +- Consumes: `keyNode`, `parseYamlKeys` +- Produces: `deploymentEnvKeys(format, roles, text): TSConfigKey[]` + +- [ ] **Step 1: Grow the fixture** + +```bash +cd test/fixtures/artifacts-app +cat > Dockerfile <<'EOF' +FROM node:22 +ARG BUILD_ID=local +ENV PAYMENT_HOST=https://pay.example.com +ENV FEATURE_FLAG "on" +COPY . . +EOF +``` + +- [ ] **Step 2: Write the failing test** + +Append to `test/config-keys.test.ts`: + +```ts +describe("deployment-env namespaces (#101 unit D)", () => { + test("Dockerfile ENV mints bindable env keys; ARG stays non-bindable dockerfile", () => { + const k = keysOf("Dockerfile"); + expect(k["PAYMENT_HOST"]?.namespace).toBe("env"); + expect(k["PAYMENT_HOST"]?.value).toBe("https://pay.example.com"); + expect(k["FEATURE_FLAG"]?.namespace).toBe("env"); + expect(k["BUILD_ID"]?.namespace).toBe("dockerfile"); // build-time only, never joins a read + }); + + test("compose environment blocks mint env keys ALONGSIDE the structural yaml keys", () => { + const k = keysOf("docker-compose.yml"); + expect(k["services.web.environment.PAYMENT_HOST"]?.namespace).toBe("yaml"); // structural + const envKeys = (arts["docker-compose.yml"]?.config_keys ?? []).filter((x) => x.namespace === "env"); + expect(envKeys.map((x) => x.key).sort()).toEqual(["FEATURE_FLAG", "PAYMENT_HOST"]); + }); +}); +``` + +- [ ] **Step 3: Run test to verify it fails** + +Run: `bun test test/config-keys.test.ts -t "deployment-env"` +Expected: FAIL — no `env`-namespace keys from Dockerfile or compose. + +- [ ] **Step 4: Write the deployment-env extractor** + +Create `src/artifacts/deployEnv.ts`: + +```ts +/** + * Deployment-env sources (#101 unit D): Dockerfile ENV, compose `environment`, and k8s container + * `env` mint BINDABLE `env`-namespace keys — the ones a `process.env.X` read joins — in addition + * to whatever structural key the file already produced. Dockerfile ARG mints a + * `dockerfile`-namespace key that is deliberately NON-bindable: build-time only. + */ +import { keyNode } from "./configKeys"; +import { parseYamlKeys } from "./yamlKeys"; +import type { TSConfigKey, TSSpan } from "../schema"; + +const DOCKER_LINE = /^\s*(ENV|ARG)\s+(.*)$/i; + +export function parseDockerfileEnv(text: string): TSConfigKey[] { + const out: TSConfigKey[] = []; + text.split("\n").forEach((line, i) => { + const m = DOCKER_LINE.exec(line); + if (!m) return; + const directive = (m[1] as string).toUpperCase(); + const rest = (m[2] as string).trim(); + const span: TSSpan = { start: [i + 1, 1], end: [i + 1, line.length + 1], bytes: [0, 0] }; + // `ENV K=V` and the legacy `ENV K V` both occur; ARG may be bare (`ARG X`). + const eq = rest.indexOf("="); + const [name, raw] = eq > 0 ? [rest.slice(0, eq), rest.slice(eq + 1)] : (() => { + const sp = rest.indexOf(" "); + return sp > 0 ? [rest.slice(0, sp), rest.slice(sp + 1)] : [rest, ""]; + })(); + let value = raw.trim(); + if ((value.startsWith('"') && value.endsWith('"')) || (value.startsWith("'") && value.endsWith("'"))) { + value = value.slice(1, -1); + } + out.push(keyNode(name.trim(), directive === "ENV" ? "env" : "dockerfile", value, span)); + }); + return out; +} + +/** + * compose `services..environment` (map or list form) and k8s + * `spec.containers[].env[].name/value` → bindable env keys. Derived from the already-flattened + * yaml keys so there is exactly one YAML parse per artifact. + */ +export function yamlEnvKeys(text: string): TSConfigKey[] { + const flat = parseYamlKeys(text); + const out: TSConfigKey[] = []; + const seen = new Set(); + const push = (name: string, value: unknown, span?: TSSpan): void => { + if (!name || seen.has(name)) return; + seen.add(name); + out.push(keyNode(name, "env", value, span)); + }; + for (const k of flat) { + // compose map form: services..environment. + const compose = /^services\.[^.]+\.environment\.([^.]+)$/.exec(k.key); + if (compose) { + push(compose[1] as string, k.value, k.span); + continue; + } + // compose list form: services..environment. = "NAME=value" + const composeList = /^services\.[^.]+\.environment\.\d+$/.exec(k.key); + if (composeList && typeof k.value === "string") { + const eq = k.value.indexOf("="); + if (eq > 0) push(k.value.slice(0, eq), k.value.slice(eq + 1), k.span); + continue; + } + // k8s: (spec|template.spec).containers..env..name = NAME (value on the sibling key) + const k8s = /^(.*\.containers\.\d+\.env\.\d+)\.name$/.exec(k.key); + if (k8s && typeof k.value === "string") { + const sibling = flat.find((x) => x.key === `${k8s[1]}.value`); + push(k.value, sibling?.value, k.span); + } + } + return out; +} + +/** Every bindable/deployment key an artifact contributes, by format. */ +export function deploymentEnvKeys(format: string, text: string): TSConfigKey[] { + if (format === "dockerfile") return parseDockerfileEnv(text); + if (format === "yaml") return yamlEnvKeys(text); + return []; +} +``` + +- [ ] **Step 5: Attach in the walk** + +In `src/artifacts/index.ts`, import `deploymentEnvKeys` and, in the same extraction block as Task 4's keys: + +```ts + const deployKeys = deploymentEnvKeys(node.format, text); + if (deployKeys.length) { + node.config_keys = [...node.config_keys, ...deployKeys]; + node.extraction = "full"; + } +``` + +- [ ] **Step 6: Run tests** + +Run: `bun test && bun run typecheck` +Expected: PASS. + +- [ ] **Step 7: Commit** + +```bash +git add src/artifacts/deployEnv.ts src/artifacts/index.ts test/config-keys.test.ts test/fixtures/artifacts-app +git commit -m "feat(artifacts): deployment-env keys — Dockerfile ENV/ARG, compose, k8s (#101)" +``` + +--- + +### Task 7: `config_access` body nodes (unit C1) + +**Files:** +- Modify: `src/schema/schema.ts`, `src/syntactic_analysis/builders.ts`, `src/schema/l1Body.ts` +- Test: `test/config-use.test.ts`, `test/fixtures/artifacts-app/src/config.ts` + +**Interfaces:** +- Consumes: `walkBody` handlers in `builders.ts` +- Produces: `TSConfigAccess` (internal, on `TSCallable.config_accesses`), `config_access` body nodes with `root`, `key?`, `span` + +- [ ] **Step 1: Grow the fixture** + +```bash +cat > test/fixtures/artifacts-app/src/config.ts <<'EOF' +export function readHost(): string | undefined { + return process.env.PAYMENT_HOST; +} +export function readFlag(): string | undefined { + return process.env["FEATURE_FLAG"]; +} +export function readDestructured(): string | undefined { + const { NODE_OPTIONS } = process.env; + return NODE_OPTIONS; +} +export function readVia(name: string): string | undefined { + return process.env[name]; +} +export function readIndirect(): string | undefined { + const key = "PAYMENT_HOST"; + return process.env[key]; +} +export function readUndeclared(): string | undefined { + return process.env.NOT_DECLARED_ANYWHERE; +} +EOF +``` + +- [ ] **Step 2: Write the failing test** + +Create `test/config-use.test.ts`: + +```ts +import { describe, expect, test } from "bun:test"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { analyze } from "../src/core"; +import type { AnalysisOptions } from "../src/options"; + +const FIXTURE = path.resolve(import.meta.dir, "fixtures/artifacts-app"); +function options(level: number): AnalysisOptions { + return { + input: FIXTURE, output: null, emit: "json", appName: "artifacts-app", neo4jUri: null, + neo4jUser: "neo4j", neo4jPassword: "", neo4jDatabase: null, analysisLevel: level, + graphs: level >= 3 ? ["cfg", "dfg", "pdg", "sdg"] : [], graphFieldDepth: 3, jobs: 1, + targetFiles: null, skipTests: true, eager: true, noBuild: true, phantoms: true, + cacheDir: fs.mkdtempSync(path.join(os.tmpdir(), "cants-cu-")), verbosity: 0, + } as AnalysisOptions; +} + +const r1 = await analyze(options(1)); +const mod = r1.application.application.symbol_table["src/config.ts"]; + +describe("config_access body nodes (#101 unit C1)", () => { + test("member, element, and destructured env reads all mint nodes with keys", () => { + const nodesOf = (fn: string) => Object.values(mod?.functions[fn]?.body ?? {}).filter((b) => b.kind === "config_access"); + expect(nodesOf("readHost").map((n) => n.key)).toEqual(["PAYMENT_HOST"]); + expect(nodesOf("readFlag").map((n) => n.key)).toEqual(["FEATURE_FLAG"]); + expect(nodesOf("readDestructured").map((n) => n.key)).toEqual(["NODE_OPTIONS"]); + expect(nodesOf("readHost")[0]?.root).toBe("process.env"); + expect(nodesOf("readHost")[0]?.callee).toBeUndefined(); // a read is not a call + }); + + test("a dynamic key mints a node with no key", () => { + const n = Object.values(mod?.functions["readVia"]?.body ?? {}).filter((b) => b.kind === "config_access"); + expect(n.length).toBe(1); + expect(n[0]?.key).toBeUndefined(); + }); +}); +``` + +- [ ] **Step 3: Run test to verify it fails** + +Run: `bun test test/config-use.test.ts -t "config_access"` +Expected: FAIL — no `config_access` nodes exist. + +- [ ] **Step 4: Add the schema shapes** + +In `src/schema/schema.ts`: + +```ts +/** INTERNAL — a recognized configuration read (env root access). Never on the wire; the wire's + * view is the `config_access` node in the owning callable's `body{}` (built by the l1Body pass). */ +export interface TSConfigAccess { + root: string; // "process.env" | "import.meta.env" | "Bun.env" + key?: string; // present when statically known + start_line: number; + start_column: number; + end_line: number; + end_column: number; + bytes: [number, number]; +} +``` + +In `TSCallable`, next to `call_sites`: + +```ts + config_accesses: TSConfigAccess[]; // INTERNAL (stripped from the wire) +``` + +In `TSBodyNode`, after the call-node attributes: + +```ts + // config_access attributes (copied from the recorded access by the l1Body pass) + root?: string; + key?: string; +``` + +Add `"config_accesses"` to the structural strip in `src/schema/emit.ts`'s `stripCallable`. + +- [ ] **Step 5: Record accesses in the builder** + +In `src/syntactic_analysis/builders.ts`, add to `BodyHandlers`: + +```ts + onConfigAccess: (n: Node, root: string, key?: string) => void; +``` + +and inside `walkBody`'s `visit`, before the call check: + +```ts + const access = envRootAccess(node); + if (access) h.onConfigAccess(node, access.root, access.key); +``` + +Add the recognizer (module scope): + +```ts +const ENV_ROOTS = new Set(["process.env", "import.meta.env", "Bun.env"]); + +/** `process.env.X` / `process.env["X"]` / `import.meta.env.X` — a read, not a call. */ +function envRootAccess(node: Node): { root: string; key?: string } | null { + if (Node.isPropertyAccessExpression(node)) { + const root = node.getExpression().getText(); + if (!ENV_ROOTS.has(root)) return null; + return { root, key: node.getName() }; + } + if (Node.isElementAccessExpression(node)) { + const root = node.getExpression().getText(); + if (!ENV_ROOTS.has(root)) return null; + const arg = node.getArgumentExpression(); + return { root, ...(arg && Node.isStringLiteral(arg) ? { key: arg.getLiteralValue() } : {}) }; + } + return null; +} +``` + +Destructuring (`const { X } = process.env`) is a VariableDeclaration whose initializer is an env root: handle it in the same `visit`, emitting one access per binding element: + +```ts + if (Node.isVariableDeclaration(node)) { + const init = node.getInitializer(); + const name = node.getNameNode(); + if (init && ENV_ROOTS.has(init.getText()) && Node.isObjectBindingPattern(name)) { + for (const el of name.getElements()) { + h.onConfigAccess(el, init.getText(), el.getPropertyNameNode()?.getText() ?? el.getName()); + } + } + } +``` + +In `buildCallable`'s handlers object: + +```ts + onConfigAccess: (n, root, key) => + config_accesses.push({ + root, + ...(key !== undefined ? { key } : {}), + ...span(n), + bytes: [n.getStart(), n.getEnd()], + }), +``` + +declaring `const config_accesses: TSConfigAccess[] = [];` beside `call_sites`, adding it to the returned callable, and adding a no-op `onConfigAccess: () => {}` to the two other `walkBody` call sites (`buildStatemented`'s bare-anon sweep and any other handler literal) so module-scope accesses are not attributed to a callable. + +- [ ] **Step 6: Emit the body nodes** + +In `src/schema/l1Body.ts`, after the call-node loop inside `resetCallable`: + +```ts + // config_access nodes share the body key space with calls: allocate AFTER them, disambiguating + // against keys already present so a read and a call on one line never collide. + for (const ca of c.config_accesses) { + const base = `${ca.start_line}:${ca.start_column}`; + let key = base; + for (let k = 2; key in body; k++) key = `${base}/${k}`; + body[key] = { + kind: "config_access", + span: { start: [ca.start_line, ca.start_column], end: [ca.end_line, ca.end_column], bytes: ca.bytes }, + root: ca.root, + ...(ca.key !== undefined ? { key: ca.key } : {}), + }; + } +``` + +- [ ] **Step 7: Run tests** + +Run: `bun test && bun run typecheck` +Expected: PASS. Monotonicity and count-parity gates in `test/schema-v2.test.ts` cover body nodes generically and should stay green; if a hard-coded count fails, update the number. + +- [ ] **Step 8: Commit** + +```bash +git add src/schema/schema.ts src/schema/l1Body.ts src/schema/emit.ts src/syntactic_analysis/builders.ts test/config-use.test.ts test/fixtures/artifacts-app/src/config.ts +git commit -m "feat(schema): config_access body nodes for env reads (#101)" +``` + +--- + +### Task 8: Literal tier + first-class unresolved reads (unit C2/C3) + +**Files:** +- Create: `src/semantic_analysis/configUseRules.ts`, `src/semantic_analysis/configUse.ts` +- Modify: `src/schema/schema.ts`, `src/schema/emit.ts`, `src/core.ts` +- Test: `test/config-use.test.ts` + +**Interfaces:** +- Consumes: `AnalysisInternal.artifacts` (config keys), `TSModule`, `forEachCallable` +- Produces: `TSConfigUse`, `TSConfigRead`, `resolveLiteralConfigUses(app, appId): { uses: TSConfigUse[]; reads: TSConfigRead[] }` + +- [ ] **Step 1: Write the failing test** + +Append to `test/config-use.test.ts`: + +```ts +const r2 = await analyze(options(2)); +const app2 = r2.application.application; +const useDsts = (fnFragment: string): string[] => + app2.config_uses.filter((u) => u.src.includes(fnFragment)).map((u) => u.dst).sort(); + +describe("config_use literal tier (#101 unit C3)", () => { + test("a literal env read joins every declaring ConfigKey", () => { + const dsts = useDsts("readHost"); + expect(dsts).toContain("can://artifact/artifacts-app/.env@key/PAYMENT_HOST"); + expect(dsts).toContain("can://artifact/artifacts-app/Dockerfile@key/PAYMENT_HOST"); + expect(app2.config_uses.every((u) => u.prov.includes("literal"))).toBe(true); + }); + + test("src is a global ordinal body-node id", () => { + const u = app2.config_uses.find((x) => x.src.includes("readHost")); + expect(u?.src).toMatch(/^can:\/\/typescript\/artifacts-app\/src\/config\.ts\/readHost@\d+:\d+$/); + }); + + test("a literal with no declared key is an undefined-key read, not an edge", () => { + const read = app2.config_reads.find((r) => r.key === "NOT_DECLARED_ANYWHERE"); + expect(read?.reason).toBe("undefined-key"); + expect(read?.prov).toEqual(["literal"]); + expect(app2.config_uses.some((u) => u.src.includes("readUndeclared"))).toBe(false); + }); + + test("a dynamic key is a non-literal read at L2", () => { + const read = app2.config_reads.find((r) => r.site.includes("readVia")); + expect(read?.reason).toBe("non-literal"); + expect(read?.key).toBeUndefined(); + }); + + test("Dockerfile ARG is never bindable", () => { + expect(app2.config_uses.some((u) => u.dst.endsWith("@key/BUILD_ID"))).toBe(false); + }); +}); +``` + +- [ ] **Step 1b: Add the call-rule fixture and test** + +```bash +cat >> test/fixtures/artifacts-app/src/config.ts <<'EOF' +import nconf from "nconf"; +export function readViaLibrary(): string | undefined { + return nconf.get("PAYMENT_HOST"); +} +EOF +``` + +Append to `test/config-use.test.ts`'s literal-tier describe block: + +```ts + test("a CALL rule resolves through the resolved external callee", () => { + const dsts = useDsts("readViaLibrary"); + expect(dsts.some((d) => d.endsWith("@key/PAYMENT_HOST"))).toBe(true); + const u = app2.config_uses.find((x) => x.src.includes("readViaLibrary")); + expect(u?.src).toMatch(/@\d+:\d+$/); // the CALL node's ordinal id + }); +``` + +(`nconf` need not be installed: with `phantoms: true` the resolver homes the callee as an +external symbol keyed by the import specifier, which is exactly what the rule matches on.) + +- [ ] **Step 2: Run test to verify it fails** + +Run: `bun test test/config-use.test.ts -t "literal tier"` +Expected: FAIL — `config_uses` is not a property of the application. + +- [ ] **Step 3: Add the wire shapes** + +In `src/schema/schema.ts`: + +```ts +/** One resolved config read: a recognized read whose key closed on exactly one literal that + * matches a declared ConfigKey. `src` is the read's GLOBAL ordinal id; `dst` the key's id. */ +export interface TSConfigUse { + src: string; + dst: string; + prov: Array<"literal" | "dataflow">; +} + +/** A recognized read that resolved to no declared key — first class, so an untraceable read is + * as visible as a traced one. `config_reads` SHRINKS as levels rise (higher tiers resolve some); + * that is deliberate and is the layer's one non-monotonic section. */ +export interface TSConfigRead { + site: string; // GLOBAL ordinal id + callee: string; // the read root ("process.env") or the resolved callee id for call rules + key?: string; // set only for reason "undefined-key" + reason: "non-literal" | "undefined-key"; + prov: Array<"literal" | "dataflow">; +} +``` + +Add to `AnalysisInternal` and to `TSApplication`: + +```ts + config_uses: TSConfigUse[]; + config_reads: TSConfigRead[]; +``` + +(optional on `AnalysisInternal`, required on `TSApplication`), and in `src/schema/emit.ts`'s root literal: + +```ts + config_uses: app.config_uses ?? [], + config_reads: app.config_reads ?? [], +``` + +- [ ] **Step 4: Ship the detector table** + +Create `src/semantic_analysis/configUseRules.ts`: + +```ts +/** + * Shipped config-use detector table (#101 unit C2). Two rule kinds: ACCESS rules name env roots + * whose member/element reads are configuration reads (recognized in builders.ts, which mints the + * `config_access` body node); CALL rules name a module+callable whose argument at `key_arg` + * carries the key. No user-extension flag — same posture as the artifact rules table. + */ +export interface AccessRule { + root: string; + namespaces: string[]; +} +export interface CallRule { + id: string; + module: string; // matched prefix-aware against the resolved callee's external module + callable: string; + key_arg: number; + namespaces: string[]; +} + +export const ACCESS_RULES: AccessRule[] = [ + { root: "process.env", namespaces: ["env"] }, + { root: "import.meta.env", namespaces: ["env"] }, + { root: "Bun.env", namespaces: ["env"] }, +]; + +export const CALL_RULES: CallRule[] = [ + { id: "deno.env.get", module: "Deno.env", callable: "get", key_arg: 0, namespaces: ["env"] }, + { id: "config.get", module: "config", callable: "get", key_arg: 0, namespaces: ["json", "yaml"] }, + { id: "config.has", module: "config", callable: "has", key_arg: 0, namespaces: ["json", "yaml"] }, + { id: "nconf.get", module: "nconf", callable: "get", key_arg: 0, namespaces: ["json", "yaml", "env"] }, + { id: "dotenv.parse", module: "dotenv", callable: "parse", key_arg: 0, namespaces: ["env"] }, +]; +``` + +- [ ] **Step 5: Write the literal tier** + +Create `src/semantic_analysis/configUse.ts`: + +```ts +/** + * config_use literal tier (#101 unit C3). Runs with the call graph — the L2 stage — because call + * rules need resolved callees. Joins a read's statically-known key to declared ConfigKeys on + * (namespace, key); a read that resolves to nothing becomes a first-class `config_reads` record. + * Deterministic: every output list is sorted. + */ +import type { AnalysisInternal, TSConfigRead, TSConfigUse } from "../schema"; +import { forEachCallable, type TSCallable } from "../schema"; +import { callBodyKeys } from "../schema/l1Body"; +import { ACCESS_RULES, CALL_RULES, type CallRule } from "./configUseRules"; + +/** (namespace, key) → declared ConfigKey ids, sorted. */ +export function keyIndex(app: AnalysisInternal): Map { + const idx = new Map(); + for (const art of Object.values(app.artifacts ?? {})) { + for (const ck of art.config_keys) { + const k = `${ck.namespace}${ck.key}`; + const arr = idx.get(k) ?? []; + arr.push(ck.id); + idx.set(k, arr); + } + } + for (const arr of idx.values()) arr.sort(); + return idx; +} + +export interface LiteralTierResult { + uses: TSConfigUse[]; + reads: TSConfigRead[]; +} + +export function resolveLiteralConfigUses(app: AnalysisInternal): LiteralTierResult { + const idx = keyIndex(app); + const uses: TSConfigUse[] = []; + const reads: TSConfigRead[] = []; + const rootNamespaces = new Map(ACCESS_RULES.map((r) => [r.root, r.namespaces])); + + for (const mod of Object.values(app.symbol_table)) { + forEachCallable(mod, (c) => { + for (const [local, node] of Object.entries(c.body)) { + // CALL rules: a `call` node whose resolved callee matches module+callable, with the key + // at `key_arg`. The key literal comes from the recorded call site's arguments; a call + // whose key argument is not a literal is a non-literal read, same as a dynamic access. + if (node.kind === "call") { + const rule = matchCallRule(node.callee, externalIndex); + if (!rule) continue; + const site = `${c.id}@${local}`; + const key = literalArgumentAt(c, local, rule.key_arg); + if (key === undefined) { + reads.push({ site, callee: String(node.callee), reason: "non-literal", prov: ["literal"] }); + continue; + } + const dsts = rule.namespaces.flatMap((ns) => idx.get(`${ns} ${key}`) ?? []); + if (!dsts.length) { + reads.push({ site, callee: String(node.callee), key, reason: "undefined-key", prov: ["literal"] }); + continue; + } + for (const dst of [...new Set(dsts)].sort()) uses.push({ src: site, dst, prov: ["literal"] }); + continue; + } + if (node.kind !== "config_access") continue; + const site = `${c.id}@${local}`; + const root = String(node.root ?? ""); + const namespaces = rootNamespaces.get(root) ?? ["env"]; + if (node.key === undefined) { + reads.push({ site, callee: root, reason: "non-literal", prov: ["literal"] }); + continue; + } + const dsts = namespaces.flatMap((ns) => idx.get(`${ns}${node.key}`) ?? []); + if (!dsts.length) { + reads.push({ site, callee: root, key: node.key as string, reason: "undefined-key", prov: ["literal"] }); + continue; + } + for (const dst of [...new Set(dsts)].sort()) uses.push({ src: site, dst, prov: ["literal"] }); + } + }); + } + uses.sort((a, b) => a.src.localeCompare(b.src) || a.dst.localeCompare(b.dst)); + reads.sort((a, b) => a.site.localeCompare(b.site) || (a.key ?? "").localeCompare(b.key ?? "")); + return { uses, reads }; +} + +/** + * A call node's resolved `callee` id names an external as + * `can://…/@external//`. Match prefix-aware on module (python's rule: a rule for + * `config` matches `config` and `config/lib/x`) and exactly on the member. + */ +export function matchCallRule(callee: unknown, externals: Map): CallRule | null { + if (typeof callee !== "string") return null; + const ext = externals.get(callee); + if (!ext) return null; + for (const rule of CALL_RULES) { + if (rule.callable !== ext.name) continue; + if (ext.module === rule.module || ext.module.startsWith(`${rule.module}/`)) return rule; + } + return null; +} + +/** The string literal at `argIndex` of the call site backing this body key, or undefined. */ +export function literalArgumentAt(c: TSCallable, bodyKey: string, argIndex: number): string | undefined { + for (const [key, cs] of callBodyKeys(c.call_sites)) { + if (key !== bodyKey) continue; + const raw = cs.arguments?.[argIndex]; + if (raw === undefined) return undefined; + const m = /^["'`](.*)["'`]$/.exec(raw.trim()); + return m ? (m[1] as string) : undefined; + } + return undefined; +} +``` + +`literalArgumentAt` needs the call site's argument TEXTS, which `TSCallsite` does not record today +(it stores `argument_types`). Add them in this task: in `src/schema/schema.ts` give `TSCallsite` + +```ts + arguments: string[]; // raw source text per argument — INTERNAL, feeds the config-use key match +``` + +and in `src/syntactic_analysis/builders.ts`'s `buildCallsite`, alongside `argument_types`: + +```ts + arguments: args.map((a) => a.getText()), +``` + +It is INTERNAL (never on the wire): `l1Body` does not copy it onto the body node, and python +records the same datum on its call sites for the same reason. + +The external index is built from the application: `new Map(Object.entries(app.external_symbols ?? {}).map(([sig, e]) => [extIdOf(sig), e]))` — but the resolved `callee` is already the can:// external id, so pass `root.external_symbols` keyed by id directly (they are keyed by id after homing). + +- [ ] **Step 6: Wire the pipeline** + +In `src/core.ts`, after the call-graph block and before the program graphs, and only at `analysisLevel >= 2`: + +```ts + // config_use literal tier (#101): needs the artifact layer's keys and, for call rules, the + // resolved call graph — so it runs with the L2 stage. Ids inside `src`/`dst` are stamped by + // assignIds during finalize; the tier records them against the same per-run ids. + let configUses: TSConfigUse[] = []; + let configReads: TSConfigRead[] = []; +``` + +Because `src`/`dst` reference `can://` ids that `assignIds` stamps during `finalizeAnalysis`, run the tier **inside** `finalizeAnalysis` instead, immediately after `backfillCallees` at `level >= 2`: + +```ts + const literal = resolveLiteralConfigUses(app); + root.config_uses = literal.uses; + root.config_reads = literal.reads; +``` + +(import from `../semantic_analysis/configUse`). Delete the two `let` declarations from `core.ts` if you added them — the tier lives in `emit.ts` where the ids exist. + +- [ ] **Step 7: Run tests** + +Run: `bun test && bun run typecheck` +Expected: PASS. + +- [ ] **Step 8: Commit** + +```bash +git add src/semantic_analysis/configUseRules.ts src/semantic_analysis/configUse.ts src/schema/schema.ts src/schema/emit.ts test/config-use.test.ts +git commit -m "feat(callgraph): config_use literal tier + first-class unresolved reads (#101)" +``` + +--- + +### Task 9: Dataflow tiers (unit C3) + +**Files:** +- Create: `src/dataflow/configUse.ts` +- Modify: `src/schema/emit.ts` +- Test: `test/config-use.test.ts` + +**Interfaces:** +- Consumes: `LiteralTierResult`, `ProgramGraphs`, `AnalysisInternal` +- Produces: `widenConfigUsesWithDataflow(app, pg, literal, level)` → `LiteralTierResult` + +- [ ] **Step 1: Write the failing test** + +Append to `test/config-use.test.ts`: + +```ts +const r3 = await analyze(options(3)); +const app3 = r3.application.application; + +describe("config_use dataflow tiers (#101 unit C3)", () => { + test("an indirect key resolves at -a 3 and carries prov dataflow", () => { + const u = app3.config_uses.find((x) => x.src.includes("readIndirect")); + expect(u?.dst).toContain("@key/PAYMENT_HOST"); + expect(u?.prov).toContain("dataflow"); + }); + + test("config_uses is superset-monotonic L2 ⊆ L3", () => { + const key = (u: { src: string; dst: string }): string => `${u.src}${u.dst}`; + const l3 = new Set(app3.config_uses.map(key)); + for (const u of app2.config_uses) expect(l3.has(key(u))).toBe(true); + }); + + test("config_reads shrinks as levels rise (the deliberate non-monotonic section)", () => { + expect(app3.config_reads.length).toBeLessThan(app2.config_reads.length); + // a read that never closes on a literal stays unresolved at every level + expect(app3.config_reads.some((r) => r.site.includes("readVia"))).toBe(true); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `bun test test/config-use.test.ts -t "dataflow tiers"` +Expected: FAIL — `readIndirect` has no edge at L3. + +- [ ] **Step 3: Write the widening pass** + +Create `src/dataflow/configUse.ts`: + +```ts +/** + * config_use dataflow tiers (#101 unit C3). Widens the literal tier over the def-use substrate: + * + * - INTRA (-a 3): the read's key expression is a local identifier whose reaching definitions in + * this callable all agree on ONE string literal. + * - INTERPROC (-a 4): the key is a parameter, and every resolved internal call site passes the + * same string literal at that position (one call boundary, no fixpoint). + * + * Superset-monotonic: it only ADDS uses, and only REMOVES the corresponding `config_reads`. + */ +import { Node, type Project } from "ts-morph"; +import type { AnalysisInternal, TSConfigRead, TSConfigUse } from "../schema"; +import { keyIndex } from "../semantic_analysis/configUse"; + +export interface ConfigUseSets { + uses: TSConfigUse[]; + reads: TSConfigRead[]; +} + +/** + * `project` gives the AST the tiers read; `astKeyOf` returns the literal a read's key expression + * closes on, or null. Intra: an identifier with a single string-literal initializer whose binding + * is never reassigned. Interproc (level >= 4): a parameter whose every resolved caller argument + * is the same literal. + */ +export function widenConfigUses( + app: AnalysisInternal, + project: Project, + literal: ConfigUseSets, + level: number, +): ConfigUseSets { + if (level < 3) return literal; + const idx = keyIndex(app); + const uses = [...literal.uses]; + const resolvedSites = new Set(); + + for (const read of literal.reads) { + if (read.reason !== "non-literal") continue; + const key = resolveKeyThroughDataflow(read, project, app, level); + if (key === null) continue; + const dsts = idx.get(`env${key}`) ?? []; + if (!dsts.length) continue; + for (const dst of [...new Set(dsts)].sort()) { + uses.push({ src: read.site, dst, prov: ["dataflow"] }); + } + resolvedSites.add(read.site); + } + + const reads = literal.reads + .filter((r) => !resolvedSites.has(r.site)) + .map((r) => (resolvedSites.size ? { ...r, prov: [...new Set([...r.prov, "dataflow" as const])] } : r)); + uses.sort((a, b) => a.src.localeCompare(b.src) || a.dst.localeCompare(b.dst)); + return { uses, reads }; +} +``` + +Implement `resolveKeyThroughDataflow` in the same file: locate the read's AST node by span (the `site` id's `@line:col` suffix plus the callable's file), take its key expression, and + +```ts +function resolveKeyThroughDataflow( + read: TSConfigRead, + project: Project, + app: AnalysisInternal, + level: number, +): string | null { + const node = accessNodeFor(read.site, project, app); + if (!node || !Node.isElementAccessExpression(node)) return null; + const arg = node.getArgumentExpression(); + if (!arg || !Node.isIdentifier(arg)) return null; + const decl = arg.getSymbol()?.getDeclarations()?.[0]; + if (!decl) return null; + // INTRA: `const key = "LITERAL"` in the same callable, never reassigned. + if (Node.isVariableDeclaration(decl)) { + const init = decl.getInitializer(); + if (init && Node.isStringLiteral(init) && !isReassigned(decl)) return init.getLiteralValue(); + return null; + } + // INTERPROC (-a 4): a parameter whose every resolved caller passes one identical literal. + if (level >= 4 && Node.isParameterDeclaration(decl)) return uniqueLiteralArgument(decl, app, project); + return null; +} +``` + +with `accessNodeFor` (span lookup, mirroring `indexCallExpressions`'s keying), `isReassigned` (any assignment whose left side resolves to the same declaration), and `uniqueLiteralArgument` (walk `app.call_graph` edges whose `target` is the enclosing callable's signature, fetch each call AST node, read the argument at the parameter's index, return the literal when all agree). Each helper stays under 30 lines; write them alongside the tests below. + +- [ ] **Step 4: Wire it into finalize** + +In `src/schema/emit.ts`, `finalizeAnalysis` gains an optional `project` parameter (the root ts-morph `Project`, already available in `core.ts`), and after the literal tier: + +```ts + if (level >= 3 && project) { + const widened = widenConfigUses(app, project, { uses: root.config_uses, reads: root.config_reads }, level); + root.config_uses = widened.uses; + root.config_reads = widened.reads; + } +``` + +`core.ts` passes `project` through. + +- [ ] **Step 5: Run tests** + +Run: `bun test && bun run typecheck` +Expected: PASS. + +- [ ] **Step 6: Commit** + +```bash +git add src/dataflow/configUse.ts src/schema/emit.ts src/core.ts test/config-use.test.ts +git commit -m "feat(dataflow): config_use intra + interprocedural tiers (#101)" +``` + +--- + +### Task 10: Neo4j projection + version revert + +**Files:** +- Modify: `src/build/neo4j/schema.ts`, `src/build/neo4j/project.ts`, `schema.neo4j.json` +- Test: `test/config-use.test.ts`, `test/neo4j-schema.test.ts`, `test/schema-v2.test.ts` + +**Interfaces:** +- Consumes: `TSApplication.artifacts[].config_keys`, `config_uses` +- Produces: `ConfigKey` node label, `DEFINES_CONFIG` and `TS_USES_CONFIG` relationship types + +- [ ] **Step 1: Write the failing test** + +Append to `test/config-use.test.ts`: + +```ts +import { project as neoProject } from "../src/build/neo4j"; + +describe("Neo4j projection of the config layer (#101)", () => { + const rows = neoProject(r2.application); + + test("ConfigKey nodes are neutral and hang off their artifact", () => { + const id = "can://artifact/artifacts-app/.env@key/PAYMENT_HOST"; + const n = rows.nodes.find((x) => x.value === id); + expect(n?.labels).toContain("ConfigKey"); + expect(n?.labels).not.toContain("TSConfigKey"); + expect(rows.edges.some((e) => e.type === "DEFINES_CONFIG" && e.to.value === id)).toBe(true); + }); + + test("TS_USES_CONFIG carries prov and points at a ConfigKey", () => { + const e = rows.edges.find((x) => x.type === "TS_USES_CONFIG"); + expect(e?.props["prov"]).toEqual(["literal"]); + expect(String(e?.to.value)).toContain("@key/"); + }); +}); +``` + +- [ ] **Step 2: Run test to verify it fails** + +Run: `bun test test/config-use.test.ts -t "Neo4j projection of the config"` +Expected: FAIL — no `ConfigKey` rows. + +- [ ] **Step 3: Revert the version and declare the vocabulary** + +In `src/build/neo4j/schema.ts`: set `export const SCHEMA_VERSION = "2.1.0";` (reverting #103's bump), add the label beside `Package`: + +```ts + { + label: "ConfigKey", + mergeLabel: "ConfigKey", + key: "id", + properties: { id: "string", key: "string", namespace: "string", value: "string", references: "string[]" }, + }, +``` + +and the relationships beside the artifact-layer block: + +```ts + { type: "DEFINES_CONFIG", from: ["Artifact"], to: ["ConfigKey"], properties: {} }, + { type: "TS_USES_CONFIG", from: ["TSBodyNode"], to: ["ConfigKey"], properties: { prov: "string[]" } }, +``` + +- [ ] **Step 4: Project the rows** + +In `src/build/neo4j/project.ts`, inside the artifact loop after `b.edge("HAS_ARTIFACT", …)`: + +```ts + for (const ck of art.config_keys) { + const kRef = b.node(["ConfigKey"], "id", ck.id, prune({ + id: ck.id, key: ck.key, namespace: ck.namespace, + value: ck.value !== undefined ? String(ck.value) : null, + references: ck.references.length ? ck.references : null, + })); + b.edge("DEFINES_CONFIG", aRef, kRef); + } +``` + +and after the dependency/unresolved block: + +```ts + for (const u of root.config_uses ?? []) { + b.edge("TS_USES_CONFIG", ref(u.src), { label: "ConfigKey", keyProp: "id", value: u.dst }, prune({ prov: u.prov })); + } +``` + +- [ ] **Step 5: Teach the gates** + +In `test/neo4j-schema.test.ts`, add `"ConfigKey"` to `NEUTRAL_LABELS` and `"DEFINES_CONFIG"` to `NEUTRAL_RELS`. In `test/schema-v2.test.ts`, add config-key rows to the node-count expectation and `"DEFINES_CONFIG"`/`"TS_USES_CONFIG"` to the artifact-layer edge sum. + +- [ ] **Step 6: Regenerate and run** + +Run: `bun run gen:schema && bun test && bun run typecheck` +Expected: PASS; `schema.neo4j.json` shows the new labels with `schema_version` still `2.1.0`. + +- [ ] **Step 7: Commit** + +```bash +git add src/build/neo4j/schema.ts src/build/neo4j/project.ts schema.neo4j.json test/config-use.test.ts test/neo4j-schema.test.ts test/schema-v2.test.ts +git commit -m "feat(neo4j): ConfigKey/DEFINES_CONFIG/TS_USES_CONFIG; revert the version bump (#101)" +``` + +--- + +### Task 11: Consumer skill, docs, and the payload measurement + +**Files:** +- Create: `docs/skills/analyzing-cants-graphs/SKILL.md`, `references/vocabulary.md`, `references/analyses.md` +- Modify: `CLAUDE.md`, `README.md`, `.claude/SCHEMA_DECISIONS.md`, `docs/design/specs/artifacts-and-dependencies.md` + +**Interfaces:** +- Consumes: the shipped vocabulary from Tasks 1–10 +- Produces: consumer documentation; no code + +- [ ] **Step 1: Measure the transitive payload** + +```bash +S=/private/tmp/claude-501/-Users-rkrsn-workspace-codellm-devkit-codeanalyzer-typescript/aa8a01c3-0b52-4850-81f9-c37f12e945a4/scratchpad +bun run src/index.ts -i "$S/vscode" -a 1 -o /tmp/vscode-artifacts >/dev/null +python3 - <<'PY' +import json +d = json.load(open("/tmp/vscode-artifacts/analysis.json"))["application"] +deps = d["dependencies"] +print("artifacts:", len(d["artifacts"])) +print("dependencies:", len(deps), "direct:", sum(1 for x in deps if x["direct"]), "transitive:", sum(1 for x in deps if not x["direct"])) +print("payload MB:", round(len(json.dumps(d)) / 1e6, 1)) +PY +``` + +Record the numbers — they go in the PR body and in the skill's dependency section. + +- [ ] **Step 2: Write the skill** + +Create `docs/skills/analyzing-cants-graphs/SKILL.md` with YAML frontmatter (`name: analyzing-cants-graphs`, a `description` naming the query surface), a level table (which nodes/edges exist at `-a 1|2|3|4`), an identity section (`can://` code ids, `can://artifact/...`, purl, `@key/`), and a **standing traps** section carrying the guidance verbatim from the spec's consumer-documentation section: + +> Your dependency *surface* and your dependency *supply chain* are different questions. +> "What does this app declare?" filters `direct: true`. "What actually ships / where does +> CVE-XXXX live?" needs the transitives — a vulnerable package four levels down is in your +> bundle whether or not you named it. + +plus the `config_reads` shrink, the `--app-name` join precondition, `sha256`-vs-truncated-`source`, `value` absent under `--no-artifact-text`, `ARG` non-bindable, and `config_access` carrying no `callee`. + +- [ ] **Step 3: Write the references** + +`references/vocabulary.md`: every label, relationship, and property this analyzer emits, in tables — including `direct: false` = lockfile-only transitive on `DECLARES_DEPENDENCY`. +`references/analyses.md`: runnable Cypher recipes, each stating its minimum `-a` level, including both dependency queries side by side: + +```cypher +// declared surface only +MATCH (:Artifact)-[d:DECLARES_DEPENDENCY {direct: true}]->(p:Package) RETURN p.name, d.kind, d.spec; +// full shipped set, transitives included (supply chain / CVE questions) +MATCH (:Artifact)-[d:DECLARES_DEPENDENCY]->(p:Package) +OPTIONAL MATCH (:Artifact)-[l:LOCKS]->(p) RETURN p.name, d.direct, coalesce(l.version, d.spec); +// which code reads a config key +MATCH (b:TSBodyNode)-[u:TS_USES_CONFIG]->(k:ConfigKey) RETURN b.id, k.key, k.namespace, u.prov; +// config reads nobody can trace (JSON only — not in the graph) +``` + +- [ ] **Step 4: Update repo docs** + +`CLAUDE.md`: replace the artifact-layer paragraph with the v1.3.0 contract (three sections, config keys, config_use tiers, the skill's location). +`README.md`: run `bun run gen:readme` for the `--help` block; add a short artifact-layer bullet to the feature list. +`.claude/SCHEMA_DECISIONS.md`: append the decisions — `config_access` as new L1 vocabulary, the deliberate `config_reads` shrink, `direct: false` transitives, `SCHEMA_VERSION` held. +`docs/design/specs/artifacts-and-dependencies.md`: add a `> Superseded by 2026-08-30-artifact-layer-v130-parity.md` line at the top. + +- [ ] **Step 5: Full verification** + +```bash +bun test && bun run typecheck && bun run build && bun run gen:schema && git diff --stat schema.neo4j.json +``` + +Expected: all green; `schema.neo4j.json` unchanged by the regen (Task 10 already committed it). + +- [ ] **Step 6: Commit** + +```bash +git add docs/skills CLAUDE.md README.md .claude/SCHEMA_DECISIONS.md docs/design/specs +git commit -m "docs: consumer query skill + artifact-layer docs (#101)" +``` + +- [ ] **Step 7: Update the PR** + +```bash +git push +gh pr edit 103 --body-file <(cat <<'EOF' +Closes #101. Stacked on #102. Spec: docs/design/specs/2026-08-30-artifact-layer-v130-parity.md +Plan: docs/design/plans/2026-08-30-artifact-layer-v130-parity.md +EOF +) +``` + +Then extend the body with the measured payload numbers from Step 1 and the unit-by-unit summary. + +--- + +## Self-Review + +**Spec coverage (re-checked after fixing the call-rule gap):** unit A → Tasks 1–3; unit B → Tasks 4–5; unit D → Task 6; unit C → Tasks 7–9; Neo4j section → Task 10; consumer-documentation section → Task 11; gates section → assertions distributed across Tasks 1–10 plus Task 11 Step 5; the payload measurement → Task 11 Step 1. + +**Type consistency:** `TSConfigKey` (Task 4) is consumed by name in Tasks 5, 6, 10; `TSConfigUse`/`TSConfigRead` (Task 8) by Tasks 9, 10; `TSConfigAccess` (Task 7) by Task 8's tier via `body[].root`/`key`; `keyIndex`/`resolveLiteralConfigUses` (Task 8) by Task 9; `deploymentEnvKeys` (Task 6) and `extractConfigKeys` (Task 4) by `src/artifacts/index.ts` in their own tasks. + +**Known follow-ups (not in this plan, by decision):** `env_file:` indirection in compose (spec §D); extending `CALL_RULES` beyond the shipped five entries as fixtures demand — the matching machinery itself ships in Task 8. diff --git a/docs/design/specs/2026-08-30-artifact-layer-v130-parity.md b/docs/design/specs/2026-08-30-artifact-layer-v130-parity.md new file mode 100644 index 0000000..c4d6491 --- /dev/null +++ b/docs/design/specs/2026-08-30-artifact-layer-v130-parity.md @@ -0,0 +1,276 @@ +# Repository-artifact layer — parity with codeanalyzer-python v1.3.0 + +- **Date:** 2026-08-30 +- **Status:** approved (brainstorming dialogue in-session); supersedes + `artifacts-and-dependencies.md`, which anchored on the pre-1.3.0 shape +- **Scope:** `codeanalyzer-typescript`; schema v2 **additive** +- **Parity anchor:** codeanalyzer-python **v1.3.0** (tag, released 2026-08-29) — artifacts + + dependencies (#157/#160), the ConfigKey family (#152/#163), the level-graded `config_use` edge + (#162/#164), deployment-env namespaces (#165/#168). Copy-from: that repo's + `docs/design/specs/2026-08-27-artifacts-and-dependencies-design.md`, + `2026-08-28-config-key-family-design.md`, `2026-08-28-config-use-edge-design.md` +- **Tracking:** org epic `codellm-devkit/.github#45`; this repo's work item #101 (PR #103), + branch `feat/issue-101-artifacts` +- **Decomposition:** one spec, four staged units (A→D), each independently green + +## Problem + +Schema v2 is code-only. The org epic's repository-artifact layer — the producer-side evidence a +cross-service analysis needs — shipped in python at v1.3.0; TypeScript emits a subset of its +pre-release shape (PR #103) and none of the ConfigKey / `config_use` / deployment-env work. This +spec brings TypeScript to the v1.3.0 contract, with the TS-native decisions the epic delegates +to each analyzer. + +## Units + +| Unit | Content | +| --- | --- | +| **A** | Inventory reconcile: never-drop walk, text-capture policy, `direct: false` transitives | +| **B** | ConfigKey family: per-artifact `config_keys[]`, namespaces, spans, references | +| **C** | `config_use`: `config_access` body nodes, detector table, level-graded tiers, first-class unresolved reads | +| **D** | Deployment-env: Dockerfile `ENV`/`ARG`, compose and k8s `env` | + +## Wire model (python v1.3.0 field-for-field; TS-native where the ecosystem differs) + +```ts +application.artifacts: Record +application.dependencies: TSDependency[] +application.unresolved_imports: TSImportBinding[] +application.config_uses: TSConfigUse[] // C +application.config_reads: TSConfigRead[] // C + +TSArtifact { id: `can://artifact//`, kind: "artifact", path, format, roles[], + size_bytes, sha256, source, text_truncated, extraction: none|partial|full, + config_keys: TSConfigKey[] } +TSConfigKey { id: `${artifactId}@key/${dotted}`, key, namespace, value?, span?, references[] } +TSDependency { name, ecosystem: "npm", spec, kind: runtime|dev|optional|peer|build, extras[], + declared_in, direct, locked_version?, provides_imports[], prov[] } +TSConfigUse { src /* body-node ordinal id */, dst /* ConfigKey id */, prov: ("literal"|"dataflow")[] } +TSConfigRead { site, callee, key?, reason: "non-literal"|"undefined-key", prov[] } +``` + +Identity: artifact ids are language-neutral (`can://artifact//`) so sibling analyzers +over one repo emit the same id for the same file; `--app-name` agreement is the join +precondition. Packages are purl-keyed (`pkg:npm/`, scoped `pkg:npm/%40scope/`). +Config keys hang off their artifact — containment mirrors `DEFINES_CONFIG`. The code tree stays +code-only. + +`kind: "peer"` remains this analyzer's one coined additive token against the shared +`runtime|dev|optional|build` enum (npm's contract-with-host has no analogue in the ratified set). + +## Unit A — inventory reconcile + +1. **Never drop.** Every non-source file is inventoried: rules-matched → its `roles`; unmatched + but decodable → `roles: ["unknown"]`; undecodable → `format: "binary"`, `source: ""`, hash and + size only. (Today's branch skips unmatched files — the largest divergence from shipped python.) +2. **Text policy.** Capture on by default; `--artifact-text` / `--no-artifact-text`; + `--artifact-text-max-bytes` (default 256 KiB). `text_truncated` marks a stored prefix. + `sha256` and `size_bytes` are always the **full file**. Extraction (dependencies, config keys) + parses the **full on-disk text**, never the stored copy — truncation can never change + extracted meaning. +3. **Transitives.** Lock-only packages become records with `direct: false`, `prov: ["lockfile"]`. + Payload growth is measured on a real repository and reported, not assumed. + +Unchanged from the current branch: neutral ids, purl packages, `provides_imports` (`@types/x` +also provides `x`), `unresolved_imports` with the type-only rule, `--resolve-installed`, +`TS_PROVIDES` / `TS_UNRESOLVED_IMPORT`. + +## Unit B — ConfigKey family + +`src/artifacts/configKeys.ts`. L1 data, identical at every level. Keys are dotted paths with +numeric segments for arrays (`services.web.ports.0`). + +| Artifact | namespace | notes | +| --- | --- | --- | +| `.env` family | `env` | flat keys, quotes stripped | +| JSON configs | `json` | **JSONC-tolerant** for `tsconfig*` / rc files (comments, trailing commas) | +| YAML (compose, k8s, workflows) | `yaml` | new `yaml` dependency; node positions give real spans | +| TOML | `toml` | | +| INI / `.properties` | `ini` / `properties` | | +| Dockerfile | `dockerfile` | refined by unit D | + +- `value` is present **by default** (capture is on); absent under `--no-artifact-text`. +- `span` is best-effort into the artifact source: exact for JSON/YAML, line-based for + env/ini/dockerfile. +- `references[]` records recognized `${VAR}` / `$VAR` tokens, deduplicated, in order of appearance. +- **Overlay posture:** a parse failure never suppresses the artifact node — the node stays and + `extraction` becomes `"partial"`. + +`extraction` across the layer: `"full"` when the artifact's meaning was extracted (dependency +records and/or config keys), `"partial"` when extraction was attempted and failed or completed +only in part, `"none"` when the artifact's roles call for no extraction (docs, legal, binary, +`unknown`). + +## Unit C — the `config_use` edge + +### C1. `config_access` body nodes (new L1 vocabulary) + +Python's detector table is call-based, and they dropped `os.environ["X"]` after verifying a +subscript never lowers to a call body node. In TypeScript that shape *is* the dominant idiom, so +recognized non-call reads mint a `body{}` node of kind **`config_access`** during the L1 walk +(`builders.ts`), keeping `config_uses.src` a uniform ordinal id at every level: + +- `process.env.X`, `process.env["X"]` +- `import.meta.env.X`, `Bun.env.X` +- destructuring — `const { PORT, HOST } = process.env` mints one node per bound element + +The node carries `span`, `root` (e.g. `"process.env"`), and `key` when statically known. It has +no `callee` (it is not a call). Additive at L1; recorded in `.claude/SCHEMA_DECISIONS.md`. + +### C2. Detector table + +`src/semantic_analysis/configUseRules.ts`, shipped in code (the `rules.ts` precedent; no +user-extension flag, matching python's posture). Two rule kinds: + +- **access rules** — env roots above → namespaces `[env]` +- **call rules** — `{ module, callable, key_arg, namespaces }`: `Deno.env.get`, node-config + `get`, `nconf.get`, and the equivalents; matched prefix-aware on module, as python does + +### C3. Tiers + +Level-graded, never guessing: + +- **literal** (`-a 2`+, `prov: ["literal"]`) — the key is statically known; join a declared + `TSConfigKey` on `(namespace, key)`. Several artifacts may declare one key (`.env` *and* + Dockerfile `ENV`): one edge per match, emitted in sorted order. +- **dataflow-intra** (`-a 3`+, `prov: ["dataflow"]`) — a non-literal key resolved through + reaching-definitions (`src/dataflow/defuse.ts`) to a unique string literal in the callable. +- **dataflow-interproc** (`-a 4`, same `prov`) — the chain crosses one call boundary via the SDG + param/summary edges. + +Superset-monotonic: literal ⊆ +intra ⊆ +interproc. + +**Unresolved reads are first class.** A key that never closes on exactly one literal → +`config_reads` with `reason: "non-literal"`; a literal matching no declared key → +`reason: "undefined-key"`. `prov` lists every tier attempted. `config_reads` deliberately +**shrinks** as levels rise — the one non-monotonic section, documented here and in the decision +log (python carries the same caveat). + +## Unit D — deployment-env namespaces + +Three sources mint **bindable `env`-namespace keys in addition to** the structural key their file +already produces in unit B: + +- Dockerfile `ENV FOO=bar` → `env:FOO` +- compose `services..environment` — map **and** list forms → `env:*` +- k8s `spec.containers[].env[]` (`name` / `value`) → `env:*` + +Dockerfile `ARG` stays namespace `dockerfile`: **non-bindable**, build-time only, never joins an +env read. `env_file:` indirection is out of scope for this unit. + +## Neo4j projection + +- Neutral `ConfigKey` node (id-keyed) and `DEFINES_CONFIG` (Artifact→ConfigKey) — neutral because + sibling analyzers MERGE onto the same nodes. +- `TS_USES_CONFIG` (TSBodyNode→ConfigKey, property `prov`) — language-prefixed because the claim + is this analyzer's. +- `config_reads` stay JSON-only: records of absence, not edges. +- The conformance gate's neutral allowlist grows by `ConfigKey` / `DEFINES_CONFIG`. +- **`SCHEMA_VERSION` is untouched** by this work (PR #103's 2.2.0 bump reverts). All analyzers + re-baseline at 2.0.0 when the layer settles across languages — the maintainer's call, recorded + in the epic. + +## Pipeline placement + +`analyze()`, in order: + +1. `buildSymbolTable` — builders mint `config_access` nodes during the L1 walk +2. artifact inventory → dependencies → config keys (after the symbol table: import binding needs + module imports), level-ungated, not cached +3. call graph (L2) → **literal tier** (`src/semantic_analysis/configUse.ts`, needs resolved callees) +4. program graphs (L3/L4) → **dataflow tiers** (`src/dataflow/`, over the def-use substrate) +5. `finalizeAnalysis` — `assignIds` stamps artifact, config-key, and `declared_in` ids per run + +This is pipeline-shaped placement rather than python's single `artifacts/` package: each pass +sits with the stage whose output it consumes, matching how this repo already places +`defuseLinker` in `semantic_analysis`. + +## Testing and gates + +Fixture `artifacts-app` grows: an unmatched file, a binary, a JSONC `tsconfig`, compose + k8s + +Dockerfile (`ENV` **and** `ARG`), `.env`, and a source file exercising every detector shape and +every tier — literal, intra-dataflow, interprocedural, plus one `undefined-key` and one +`non-literal` read. + +- artifacts / dependencies / config keys identical at `-a 1|2|3|4` +- `config_uses` superset-monotonic L2 ⊆ L3 ⊆ L4; `config_reads` shrink asserted, not merely allowed +- two consecutive default runs byte-identical +- Neo4j rows for `ConfigKey` / `DEFINES_CONFIG` / `TS_USES_CONFIG`; count-parity gate taught the + new families; `schema.neo4j.json` regenerated (version unmoved) +- measured payload delta from `direct: false` transitives on a real repository, reported in the PR +- full suite + typecheck green at every unit boundary + +## Consumer documentation — the query skill + +The layer ships with a consumer-facing skill, mirroring python's +`docs/skills/analyzing-canpy-graphs/` (released as `analyzing-canpy-graphs-SKILL.md` + +`…-skill.tar.gz` alongside `schema.json` / `schema.cypher`): + +``` +docs/skills/analyzing-cants-graphs/ + SKILL.md # level table, identity, standing traps + references/vocabulary.md # labels, edges, properties — never guess one + references/analyses.md # recipe catalogue, each stating its minimum -a level +``` + +Python's version documents the *mechanics* of these fields. Ours must also carry the +**decision guidance** — why a consumer picks one query over another — because every trap below is +a place where a mechanically-correct query answers the wrong question. Required content: + +**`direct` — dependency surface vs. supply chain.** State it plainly, in these terms: + +> Your dependency *surface* and your dependency *supply chain* are different questions. +> "What does this app declare?" filters `direct: true`. "What actually ships / where does +> CVE-XXXX live?" needs the transitives — a vulnerable package four levels down is in your +> bundle whether or not you named it. + +with both recipes side by side (declared-only, and the full pinned set), and the scoping note +that transitives are top-level lock entries carrying `kind: "runtime"` because a lock does not +record why a package is present. + +**The other traps, each with its "why":** + +- **`config_reads` shrinks as `-a` rises** — it is the one deliberately non-monotonic section. A + consumer diffing two levels must read a vanished record as *resolved at the higher tier*, not + as *fixed in the code*. Pair it with `config_uses`, which is superset-monotonic. +- **`--app-name` is the cross-analyzer join precondition** — artifact ids are language-neutral so + a TS and a python analysis of one monorepo MERGE onto the same `:Artifact`; they only do so if + both runs pinned the same app name. Analyzers pointed at different subdirectories disagree. +- **`sha256` is always the full file; `source` may not be.** Under `--artifact-text-max-bytes`, + `text_truncated: true` means the stored text is a prefix — hash-compare on `sha256`, never on + `source`, and never re-derive meaning from a truncated copy (the analyzer itself parses the + full on-disk text). +- **`value` is present by default** and absent under `--no-artifact-text` — an absent `value` is a + capture setting, not an empty config key. +- **Dockerfile `ARG` is not bindable.** `ENV` mints an `env`-namespace key a `process.env` read + joins; `ARG` mints a `dockerfile`-namespace key that deliberately never joins one — it exists + at build time only. A query that unions the two namespaces will report build-time values as + runtime configuration. +- **`config_access` nodes are reads, not calls** — they carry no `callee`; joining them through + `TS_RESOLVES_TO` finds nothing. Follow `TS_USES_CONFIG` to the `ConfigKey`. + +## Caveats and risks + +- **Transitive payload.** npm lock trees are large; unit A's `direct: false` records are the + growth point. Measured, reported, and revisited only with numbers. +- **New runtime dependency** (`yaml`) enters the compiled binary. Accepted for correct YAML + (anchors, flow style, multiline) and real spans; a hand-rolled subset would silently mis-parse. +- **`config_access` moves `body{}`** at L1. Additive, but it is new vocabulary in the wire's most + load-bearing map; the schema decision log records it. +- **`config_reads` is non-monotonic** by design — inherited from python, documented in both + places rather than silently absorbed. +- **SDK lockstep.** `python-sdk`'s TypeScript models are `extra="forbid"`; six new application + keys must land there before its analyzer pin moves. Same obligation python's own release carries. +- **Cross-analyzer joins need `--app-name` agreement**; analyzers pointed at different + subdirectories of one monorepo will disagree on artifact ids. + +## Definition of done + +- Units A–D land as staged commits on `feat/issue-101-artifacts`, each green, closing #101 via + PR #103. +- Every gate above passes; the spec's caveats are reflected in `.claude/SCHEMA_DECISIONS.md`. +- `CLAUDE.md`, `README.md` (`--help` block), and the epic comment trail record the shipped + contract; `artifacts-and-dependencies.md` is marked superseded by this spec. +- `docs/skills/analyzing-cants-graphs/` exists with the guidance above (not just field + mechanics), every recipe states its minimum `-a` level, and it ships as a release asset + alongside the machine-readable contract — python v1.3.0's asset set is the parity bar. diff --git a/docs/design/specs/artifacts-and-dependencies.md b/docs/design/specs/artifacts-and-dependencies.md new file mode 100644 index 0000000..659f5f4 --- /dev/null +++ b/docs/design/specs/artifacts-and-dependencies.md @@ -0,0 +1,98 @@ +# Artifacts and dependencies — the repository-artifact layer for TypeScript + +> Superseded by 2026-08-30-artifact-layer-v130-parity.md + +- **Status:** implemented (branch `feat/issue-101-artifacts`); **recalibrated 2026-08-27** to the + ratified python contract after the first cut anchored on an orphaned branch +- **Scope:** `codeanalyzer-typescript`; schema v2 **additive** (no level, id-tier, or existing-field movement) +- **Parity anchor:** codeanalyzer-python **PR #160** (implementation of the approved spec + `2026-08-27-artifacts-and-dependencies-design.md`, PR #158). NOT the `51ee29e` + `feat/configuration-files` branch — that shape (language-namespaced `@artifact/` ids, contained + dependency/config-key children, `artifact_kind` enum, text-capture caps) was never merged; this + spec's first revision mirrored it and has been rebuilt. +- **Tracking:** one work item (#101), one PR (#103), branch stacked on `feat/issue-100-linker-propagation` + +## Contract-impact triage + +| Question | Answer | +| --- | --- | +| Schema v2 shape | **additive**: `application.artifacts{}` (flat nodes), `application.dependencies[]` (flat evidence rows), `application.unresolved_imports[]` | +| Identity | artifact ids are **language-NEUTRAL**: `can://artifact//` — the first `can://` segment is a namespace (a language for code, the literal `artifact` for files), so sibling analyzers over one repo emit the SAME id for the same file. `` agreement is the precondition for cross-analyzer joins | +| Levels / monotonicity | ungated, identical at `-a 1..4`; monotonicity holds trivially | +| schema_version | unchanged; Neo4j contract 2.1.0 → **2.2.0** (additive) | +| Repos | `codeanalyzer-typescript` now; **python-sdk** must gain the three families before its analyzer pin moves (its models are `extra="forbid"` — verified; python's own PR #160 carries the same obligation); repo docs | +| Shared vocabulary movement | ONE additive token: dependency `kind: "peer"` (npm's contract-with-host) against the ratified enum `runtime\|dev\|optional\|build`. Recorded like the `"reaching-defs"` precedent | + +## The mirrored model (python PR #160 shapes) + +``` +application.artifacts: Record +TSArtifact id = can://artifact// (per-run), kind "artifact", path, + format (json|jsonc|yaml|toml|ini|dockerfile|yarnlock|env|text), + roles[] (dependency-manifest|tool-config|container-image|service-topology| + ci|env|packaging|legal|docs|script|unknown), + size_bytes, sha256, source (verbatim, UNBOUNDED by decision — spec §3), + extraction (none|partial|full) + +application.dependencies: TSDependency[] # flat, no node ids — :Package (purl) is the node +TSDependency name (@scope kept), spec, kind (runtime|dev|optional|peer|build), extras[] (npm: []), + declared_in (artifact id), locked_version?, provides_imports[], + prov[] (declared|lockfile|installed-metadata|heuristic) + +application.unresolved_imports: TSImportBinding[] +TSImportBinding module (specifier root), bound_to?, prov[] +``` + +## Locked TS decisions (recalibration session 2026-08-27) + +1. **config_keys dropped** — python's unit 4 owns config extraction; config-role artifacts get + node + roles + source only. The earlier TS config-key parser is parked (this file is its record). +2. **`peer` kind coined** (additive). npm mapping: `dependencies→runtime`, + `devDependencies→dev`, `optionalDependencies→optional`, `peerDependencies→peer`. +3. **Capture is rules-matched only** (python's posture): the shipped table in + `src/artifacts/rules.ts` (glob → format/roles, roles union across matches; basename patterns + match at any depth, `/`-patterns anchor at root) + extensionless shebang files as `script`. + An unmatched file is NOT an artifact. `source` is verbatim and unbounded (python's decision; + revisit only with measured payload numbers); binary probe failures carry `source: ""`. +4. **Dependencies are declared-only and flat**: every `package.json` (workspace members included) + emits records with `declared_in` = its artifact id; the JSON lock family + (`package-lock.json`/`npm-shrinkwrap.json`/`bun.lock` JSONC) backfills `locked_version` on the + OWNING (sibling) manifest's records and appends `"lockfile"` to `prov`; locks never create + records; `yarn.lock`/`pnpm-lock.yaml` are inventory-only artifacts. +5. **provides_imports**: the package name itself; `@types/x` also provides `x` + (DefinitelyTyped `scope__pkg` unmangled to `@scope/pkg`). +6. **Import binding / unresolved_imports**: every non-relative, non-builtin specifier ROOT from + the symbol table's imports. A VALUE import needs the runtime package; an `import type` is + satisfiable by `@types/x` alone. Only-@types-for-a-value-import → partially bound + (`bound_to: "@types/x"`, `prov: ["heuristic"]`). `--resolve-installed` (opt-in, default off) + probes `node_modules//package.json` (`prov: ["installed-metadata"]`); default runs read + only repo files and stay byte-identical. +7. **Neo4j (contract 2.2.0)**: language-NEUTRAL `:Artifact` and `:Package` (purl ids, + `pkg:npm/`, scoped `pkg:npm/%40scope/`) — the deliberate, sanctioned exception to + TS-prefixing so sibling analyzers MERGE onto the same nodes (the conformance gate allowlists + exactly these). Edges: `HAS_ARTIFACT`, `DECLARES_DEPENDENCY` (props spec/kind/extras/prov, + `_k` = kind), `LOCKS` (version; fans from every lock artifact present — python's documented + coarse fan), and the analyzer's own claims `TS_PROVIDES` (Package→minted module-level + `:TSExternal` ghost) and `TS_UNRESOLVED_IMPORT` (application→ghost, prov). `source` stays off + the graph. +8. **Pipeline**: `src/artifacts/` (rules, deps, binding, index) runs in `analyze()` after the + symbol table (binding needs module imports), level-ungated, not cached; `assignIds` stamps + artifact ids and re-stamps `declared_in` per run (`--app-name` rule). + +## Definition of done + +- Three sections emitted identically at every `-a`; monotonicity + conformance + count-parity + gates green (parity gate counts neutral Artifact/Package rows + minted ghosts explicitly). +- Fixture app: root+workspace manifests, both JSON locks, `yarn.lock` inventory-only, `.env`, + tsconfig, Dockerfile, CI workflow, LICENSE, an undeclared VALUE import, an `import type` + satisfied by `@types` — every kind token incl. `peer`, prov chains, purl ids (scoped included), + `--resolve-installed` exercised. +- Determinism: two consecutive default runs byte-identical. +- `schema.neo4j.json` regenerated at 2.2.0; CLAUDE.md + SCHEMA_DECISIONS + README/--help updated. + +## Release plan + +Ships in the minor after the linker train (#97 → #99 → #102 → #103); schema_version unmoved; +Neo4j 2.2.0 in release notes. **SDK lockstep required** (`extra="forbid"`): python-sdk gains the +three families before its pin moves. Cross-analyzer id joins additionally require pinned +`--app-name` agreement between analyzers (spec §2 precondition). diff --git a/docs/design/specs/defuse-linker-call-graph.md b/docs/design/specs/defuse-linker-call-graph.md new file mode 100644 index 0000000..6bf38c6 --- /dev/null +++ b/docs/design/specs/defuse-linker-call-graph.md @@ -0,0 +1,144 @@ +# tsc + defuse linker call graph (Jelly removal) + +- **Status:** accepted, not yet implemented +- **Scope:** `codeanalyzer-typescript` only; one PR, tracked in #98 +- **Tracking:** #98 (work item); branch `feat/issue-098-defuse-linker`, stacked on + `refactor/issue-096-native-v2-model` (#97) — lands after it merges +- **Parity precedent:** codeanalyzer-python 1.2.0, which replaced PyCG's global fixpoint with + Jedi + a per-callable defuse linker + (`codeanalyzer-python/docs/design/specs/2026-08-25-defuse-linker-call-graph-design.md`, #148 + there). This spec is the TypeScript instantiation of that architecture; divergences are called + out explicitly. + +## Motivation + +Jelly is the analyzer's scale ceiling and its heaviest dependency. It is a whole-program flow +analysis — the same cost class as python's removed PyCG (3h19m on odoo without convergence; +Fraunhofer CPG OOM at 44GB on the same corpus) — and it is bundled INTO the shipped binary +(`src/main.ts` `__jelly` dispatch, `CANTS_SELF_JELLY`, `patches/`, the `@cs-au-dk/jelly` +dependency). On the vscode-class targets we want to analyze, the Jelly leg is unusable; the tsc +leg alone loses exactly the edges Jelly recovers. + +Measured on the repo fixtures (union provider, L2): 79 edges total, of which **6 are +jelly-only**, in four sharp classes: + +| Class | Fixture evidence | +| --- | --- | +| Decorator invocations | `UserController.show → Get`, `→ Param`, `list → Get` (sample-app) | +| Library-mediated callback edges | `UserService.describeAll → ` (lambda passed to `.map`) | +| Receiver typing inside anons | ` → User.describe` (element type of the mapped array) | +| Param-flow calls | ` → named` via a function-valued parameter (anon-app) | + +Small counts, but the classes are the point: each is a bounded, per-callable resolution problem. +The expensive substrate already exists in this repo as the L3 kernels +(`src/dataflow/defuse.ts` — k-limited access-path def-use with the flow-insensitive alias +substrate — plus the CFG machinery). The replacement follows the same Joern/Fraunhofer CPG +architecture python adopted: a fast base graph from the type-checker, then a **local** linker +pass that backfills what the resolver missed. No global fixpoint anywhere. + +## Contract-impact triage + +| Question | Answer | +| --- | --- | +| Schema v2 shape (node/edge kinds, fields, ids, levels) | unchanged; schema_version stays **2.1.0** | +| `prov` vocabulary | `"jelly"` disappears; **`"defuse"`** coined for linker-derived edges (technique-named, matching python's `"defuse"` and the DDG's `"reaching-defs"`/`"points-to"`). `"tsc"` and `"import"` unchanged. Both-found edges merge to `["defuse", "tsc"]` via the existing provenance union | +| Refinement contract | unchanged — the linker runs inside the L2 build; `callee: null→id` stays the single sanctioned refinement | +| Monotonicity gate | unaffected (edges only added at L2, as today) | +| `synthesized_callables` | shape + 2.1.0 compat index unchanged; the residual-fallback path stays, but provider-reported unknowns effectively vanish (tsc + linker only name tree signatures) | +| Repos | `codeanalyzer-typescript` now. **python-sdk follow-up** (separate PR, next SDK minor): the `tsc_only` kwarg threads `cldk/core.py` → `backend_config.py` → `typescript/codeanalyzer.py` → `typescript_analysis.py` and passes `--tsc-only`; it must be removed once this releases. SDK `prov` is passthrough (`List[str]` — verified), no model change | +| CLI (**BREAKING**) | `--call-graph-provider` and `--tsc-only` removed; one code path, no backend flag (python: "the linker is cheap and deterministic; nothing to opt out of") | + +## Locked decisions (design session 2026-08-26) + +1. **tsc resolver is the base call graph, always.** Its edges keep `prov: ["tsc"]`; RTA + expansion and the phantom/import leg (`prov: ["import"]`) are untouched. +2. **Jelly is removed wholesale**: `src/semantic_analysis/jellyProvider.ts`, the union provider + and `selectProvider`, `options.callGraphProvider`, both CLI flags, the `__jelly` argv mode in + `src/main.ts`, `CANTS_SELF_JELLY`, the `@cs-au-dk/jelly` dependency, its `patches/`, and + `union-provider.test.ts` (superseded by the linker suite + reference validation). +3. **The linker runs at L2 with targeted kernels**: def-use state is built only for callables + that still contain unresolved call sites after the tsc leg. Per-callable, no fixpoint, + **sorted iteration mandated** — deterministic by construction. +4. **Linker edges carry `prov: ["defuse"]`** and merge with tsc edges through the existing + `mergeCallGraphs` provenance union. Resolutions reach the L1 `call` body nodes through the + same channel the tsc leg uses today, with python's cache rule preserved: linker resolutions + are **never persisted into `callee_signature`** (the symbol table round-trips the analysis + cache; a persisted resolution would resurface on a warm run with the wrong provenance). +5. **External-callback edges are kept** — a deliberate, documented **divergence from python**: + when a function value (anonymous or named) is passed as an argument to an external or + unresolved callee, the linker emits `enclosing-callable → function-value`, + `prov: ["defuse"]`, **edge-only** (no body call node — matching Jelly's observed behavior; + there is no real call site in first-party code). Rationale: JS/TS is callback-central, and + 2.1.0 materialized anonymous callables as tree nodes precisely so they can be addressed — + they must stay reachable by edge, not only by containment. The parity clause covers shared + vocabulary, not per-language recall. +6. **Decorator invocations become linker edges**, same edge-only rule: decorators are captured + in the model (`decorators[]` with `qualified_name`) but their factory calls are outside + `walkBody`'s reach, so no body node exists today and none is added (adding one would move the + wire). The linker resolves `qualified_name`/`name` against the symbol table and emits + `decorated-owner → decorator`, `prov: ["defuse"]`. +7. **No backend flag.** One code path. + +## Tier ladder + +Tiers land in order; the Joern ledger (below) decides how far down the ladder the +implementation must go before the gate is clean. Each tier is per-callable or bounded-round — +never a fixpoint. + +- **T1 — local value chase.** For an unresolved call site whose callee expression is a local + binding: chase the def-use chain (existing `defuse.ts` kernels, k-limited access paths) + through alias assignments (`const f = handler; f()`) to a function literal / declaration / + import binding. Imports resolve cross-module through the symbol table (the checker already + did most of this; the chase covers what it declared as "a variable", not "a function"). +- **T2 — decorator edges** (decision 6). +- **T3 — external-callback rule** (decision 5). +- **T4 — interprocedural votes, bounded.** A type-oracle round in python's style, narrowed by + what tsc already proves: (a) function values passed at **resolved internal** call sites vote + for the callee's parameter — a parameter-invoking site (`cb()`) resolves to the voted + functions; (b) return summaries (`return inner` / unique ctor returns) let + `const f = factory(); f()` resolve; (c) `this.x = fn` property assignments type + `this.x()` sites. Two bounded rounds (round one's resolutions vote before round two), + internal-target votes only. +- **T5 — CHA-by-name fallback.** Receiver call sites that survive every typed tier resolve to + every internal callable of that method name (bounded per site) — the over-approximation Joern + itself emits for untyped receivers. Applied last so precise resolutions are never widened. + +## Reference validation (the enforced gate) + +Mirrors python's method, per the maintainer's mandate: iterate edge-for-edge against **Joern +`jssrc2cpg`** (available in `~/workspace/codellm-devkit/joern-dist`) until our call graph is a +**strict superset of every real edge** Joern produces on the validation corpus: + +- **Corpus:** `test/fixtures/sample-app`, `dataflow-app`, `anon-app`, plus **one real-world + express/nest application** vendored or pinned at implementation time (the toy fixtures alone + are too small to trust a superset claim). +- **"Real edge"** = both endpoints exist in source and are nameable in this schema; Joern's + synthetic families (`N` internals where we hold the positional anon node, ``, + ``, fabricated members) are excluded through a **committed exception ledger, audited + per class** — python's discipline, not a waiver. +- **Scale benchmark:** microsoft/vscode at L1/L2 — wall-clock, peak RSS, edge counts by `prov` + — reported in the PR next to Joern `jssrc2cpg` on the same tree (or its failure mode). The + giant-JSON emission ceiling is out of scope here (separate issue); the benchmark measures the + analyze/compute phase and the Bolt projection path. + +## Acceptance + +- Joern superset ledger committed: 100% real-edge coverage per corpus app, every residual + classified. +- **Jelly-recovery spike metric** (python's PyCG analog): of today's jelly-only edges on the + fixtures, the % the linker recovers — reported in the PR, no hard gate (some jelly edges may + be judged junk by the ledger; the report says which and why). +- **A/B determinism:** paired runs byte-identical on the corpus `call_graph` (the linker adds + no nondeterminism; tsc inference is deterministic — stronger than python's Jedi caveat). +- Full suite, typecheck, monotonicity and Neo4j conformance gates green; `git grep -li jelly` + over `src`/`packaging`/`patches`/`package.json` returns nothing; the binary loses its + `__jelly` mode and shrinks. + +## Release plan + +- Ships in the analyzer's next MINOR (with the #96 native-model rewrite already queued for it); + release notes carry **BREAKING** lines for the removed `--call-graph-provider`/`--tsc-only` + flags — python 1.2.0 precedent for a flag removal in a minor. schema_version untouched. +- **python-sdk follow-up (tracked in this spec; file the issue when picked up):** remove the + `tsc_only` kwarg chain and its `--tsc-only` pass-through, then bump the SDK's pinned analyzer + version. Until it lands, `tsc_only=True` against the new binary is the one known break. diff --git a/docs/design/specs/defuse-linker-joern-ledger.md b/docs/design/specs/defuse-linker-joern-ledger.md new file mode 100644 index 0000000..8503d1d --- /dev/null +++ b/docs/design/specs/defuse-linker-joern-ledger.md @@ -0,0 +1,98 @@ +# Joern superset ledger — tsc + defuse call graph (#98) + +The enforced acceptance gate of `defuse-linker-call-graph.md`: the analyzer's L2 call graph must +be a **strict superset of every real call pair** Joern `jssrc2cpg` produces on the validation +corpus, plus a scale audit on microsoft/vscode. "Real" = a SINGLE-candidate Joern resolution +whose endpoints exist in source and are nameable in this schema; everything excluded is +classified below and audited per family, never waved through. Reproduce with `scripts/joern/` +(dump-calls.sc → compare_joern.py; corpus RESIDUAL must be 0). + +- **Joern:** v4 distribution, `jssrc2cpg` frontend +- **Analyzer:** branch `feat/issue-098-defuse-linker` (tsc resolver + defuse linker, no Jelly) + +## Corpus gate (enforced: residual 0) + +| App | Joern real pairs | Covered | Residual | +| --- | --- | --- | --- | +| `test/fixtures/sample-app` | 23 | 23 | **0** | +| `test/fixtures/dataflow-app` | 19 | 19 | **0** | +| `test/fixtures/anon-app` | 2 | 2 | **0** | +| nestjs-realworld-example-app @ `c1c2cc4` (35 files) | 30 | 30 | **0** (ours: 126 internal edges, 4.2× Joern's 30) | + +A/B determinism: paired analyzer runs byte-identical on every corpus app and on vscode's edge +dump (hash-compared). The linker adds no nondeterminism; the tsc checker is deterministic. + +## vscode scale audit (microsoft/vscode @ a3c9dc6, `src/`, 8,735 TS files / 1.15M LOC) + +64GB M-series (10 cores). Analyzer single-threaded (`-j 1`), eager, no deps materialized; +Joern on all 10 cores at `-Xmx48g`. + +| Run | Wall | Max RSS | Output | +| --- | --- | --- | --- | +| cants L1 | **4m15s** | 18.9GB | 136,973 callables | +| cants L2 | **5m52s** | 24.4GB | **1,024,232 edges** — tsc 970,334 (324,525 resolved + 778,070 RTA + 89,344 phantom), defuse 54,170 (430 decorator / 26,466 callback / 1,797 votes / 31,581 CHA / rest chase) | +| cants **L4** (full SDG + artifact layer) | **10m42s** | 28.6GB | 1,028,736 call edges; CFG/CDG/DDG attached for 123,221 callables; **param_in 722,820 / param_out 200,775**; finalize survives via the structural (structuredClone) strip — the prior stringify-roundtrip clone OOM'd at exactly this scale, measured | +| Joern jssrc2cpg parse | **9m06s** | 30.3GB | CPG; 941,132 call rows + 768,350 parameter rows dumped (streamed writer — the single-StringBuilder dump crossed the JVM's 2GB array cap) | + +Superset audit against Joern's single-candidate real pairs, after nine ledger-driven fix +rounds: **54,918 / 55,074 covered (99.72%), residual 135** — past python's odoo bar (99.0%, +final residual 243). Round 9/10 (#100): property-initializer attribution closed the whole +Registry-as-field family; the T4a property votes, T4b chained returns, and T4c ctor-field chain +landed; and Joern's parameter tables now PROVE the Promise-executor shadows (21 classified +`joern-param-shadow` by their own dump). Reference: the engines this architecture replaced DNF'd at this scale +class (PyCG 3h19m without convergence; Fraunhofer CPG OOM at 44GB). + +### Analyzer fixes the ledger forced (python's reference-validation experience, repeated) + +1. **Concise-arrow call sites** — `u => u.describe()` recorded no call (children-only body walk); + Jelly's approximated edge had masked the L1 gap. Now checker-typed. +2. **Module-scope callers** — top-level `main()`, class decorators, the top-level express idiom + had no caller. Attributed to the MODULE (python #131 parity), module prefix id-homed so the + edges land on the module node. +3. **Tagged template calls** — `` inline`url(...)` `` was invisible to L1/L2 end to end (walkBody, + resolver, call index), while L3's exception model already treated it as a call. vscode's + cssValue idiom found it; regression-tested. +4. **Parameter-default initializer calls** — `f(sel, style = getSharedStyleSheet())` executes in + the callee's activation but lived outside `getBody()`. vscode's domStylesheets family found + it; regression-tested. +5. **JS sources were never discovered** — `SOURCE_EXTS` was ts-only, so vscode's vendored + `marked.js` was "analyzed" through its bodiless `marked.d.ts` (zero edges from every body). + `.js`/`.jsx`/`.mjs`/`.cjs` are first-class now, with two-way sibling rules: a `.js` beside a + real `.ts` source is build output (skipped); a `.d.ts` beside an analyzed `.js` is its + declaration file (skipped as a module — the checker still reads it from disk). +6. **T4c — the ctor-field callback chain** — `this.migrate(...)` where the field arrives through + the constructor (parameter property / `this.f = param`) resolves to the function values passed + at the class's `new` sites, with ONE bounded parameter hop (`register(key, cb)` → + `new Migration(key, cb)`), and module-scope resolved calls feed the vote rounds — so the + registered callbacks' own `write()` sites resolve too. vscode's migrateOptions family + (22 pairs) closed end-to-end, no fixpoint anywhere. + +## Exception classes (audited) + +| Class | vscode count | What it is / verdict | +| --- | --- | --- | +| `joern-synthetic-helper` | 169,861 | Their TS-lowering machinery (`__decorate`, `__param`, `__metadata`, `__ecma.*`, `require`/`import` plumbing) — desugaring artifacts, not source calls | +| `joern-name-fanout` | 131,710 | Multi-candidate `callee` lists (one `.toString()` row links a 131KB candidate string) — candidate enumeration, not resolution; python's "speculative typed-attribute fan-out". Informational: 5,151 of these have ≥1 candidate covered by our graph | +| `external` | 173,092 | Callee homes outside the project — outside the internal gate; we carry these as phantom edges with id-homed external nodes | +| `notin` / fabricated stubs | 61,883 | Parameters-as-callees (`next()` → fabricated `::program:next`), decorator-value targets (`@User(...)` where `User = createParamDecorator(...)`), import-stubs — targets that do not exist in source as callables (python's identical families) | +| `odd-chain` / `lambda-unmapped` | 5,455 / 1,842 | Their fullName grammar edge cases and lambdas our line-matcher cannot uniquely map — mapping losses, counted, not silently dropped | +| `joern-this-misresolution` (covered, listed) | 1,469 | Their single "resolution" names a same-file free function while the receiver in source is `this.` — we hold the typed method edge (e.g. `setZoomLevel → WindowManager.getZoomLevel` vs their `→ browser.getZoomLevel`) | +| `joern-name-misresolution` (covered, listed) | 338 | Same shape across files — they name-linked `ActionBar.dispose` where the call is the imported free `dispose` from lifecycle.ts; we hold the typed import edge | +| `joern-unresolved` | 72 | ``, no linked callee — their unresolved set | + +## The audited residual (189, classified) + +| Family | ≈count | Nature | +| --- | --- | --- | +| Residual Promise-executor shadows | ~13 | Deeper lambda callers whose parameter tables Joern itself under-reports — same fabrication family as the 21 their tables DO prove | +| **Static/instance same-name collision** | 11 | `Range.isEmpty` (instance) calls `Range.isEmpty` (static): the signature grammar cannot mark static, both collapse to ONE signature — the pair is unrepresentable and the collision gate flags it. A REAL schema-grammar limitation surfaced by this audit → design-mode follow-up | +| Closure-local callables through deep value flow | ~55 | Functions escaping via event emitters/registries beyond T4/T4a/T4b/T4c's bounded hops (settingsTree `onChange`, event utilities, `registerAction` registries) — python zeroed its analog only with whole-program propagation (#150), the staged next step | +| Accessor/duck-typed and misc tails | ~56 | Getter-vs-method naming (`EventMultiplexer.event`), interface duck-typing (`ISearchTreeFolderMatch.id`), terminalTaskSystem/resources dynamic patterns | + +## Known non-goals (recorded, deliberate) + +- Whole-program propagation for escaped closure-locals (python #150's tier) — staged follow-up, + not this issue. +- Static/instance signature discrimination — schema id-grammar change, design mode. +- Property-arrow class members as tree callables — future schema work; T5's bounded CHA covers + the call sites meanwhile. diff --git a/docs/skills/analyzing-cants-graphs/SKILL.md b/docs/skills/analyzing-cants-graphs/SKILL.md new file mode 100644 index 0000000..16f5e9b --- /dev/null +++ b/docs/skills/analyzing-cants-graphs/SKILL.md @@ -0,0 +1,129 @@ +--- +name: analyzing-cants-graphs +description: Use when querying the cants (codeanalyzer-typescript) Neo4j graph — call-graph, structure, inheritance, control/data-flow, dependency/SBOM, or configuration-use questions, or when writing any Cypher over schema v2's projection. +--- + +# Analyzing cants graphs (schema v2, Neo4j projection) + +One additive tree + typed edge overlays, projected as a property graph (`--emit neo4j` → +`graph.cypher` snapshot or live Bolt push). Vocabulary is fixed — **never guess a label, property, +or key** — it is all in [references/vocabulary.md](references/vocabulary.md), generated from +`src/build/neo4j/schema.ts` and enforced by `test/neo4j-schema.test.ts`. The recipe catalogue +(structure, calls, inheritance, control/data-flow, slicing, dependencies/artifacts/SBOM, +configuration) is [references/analyses.md](references/analyses.md); every recipe states its +minimum `-a` level. + +This skill mirrors codeanalyzer-python's `docs/skills/analyzing-canpy-graphs/`, which documents +field *mechanics*. The traps below go further: each one is a place where a mechanically-correct +query answers the wrong question. + +## What exists at which level + +| `-a` | tree | edges | +| --- | --- | --- | +| 1 | callables + `call`/`config_access` body nodes (`callee` null) | `TS_DECLARES`, `TS_HAS_METHOD`, `TS_HAS_FIELD`, `TS_HAS_BODY_NODE`, `TS_EXTENDS`, `TS_IMPLEMENTS` | +| 2 | `callee` resolved | `TS_CALLS` (prov tsc/defuse/import), `TS_RESOLVES_TO`, `TS_USES_CONFIG` (literal tier) | +| 3 | full statement `body`, `@entry`/`@exit` | `TS_CFG_NEXT`, `TS_CDG`, `TS_DDG` (prov `reaching-defs`); `TS_USES_CONFIG` widens (+dataflow, intra) | +| 4 | `@formal_in:N`/`@formal_out`/`actual_in:N`/`actual_out` vertices | `TS_PARAM_IN`, `TS_PARAM_OUT`, `TS_SUMMARY`, `TS_DDG` widened (+`points-to`); `TS_USES_CONFIG` widens further (+dataflow, interprocedural — same `prov` tag as intra) | + +The repository-artifact layer — `Artifact`/`Package`/`ConfigKey` and every edge among them +(`HAS_ARTIFACT`, `DECLARES_DEPENDENCY`, `LOCKS`, `DEFINES_CONFIG`, `TS_PROVIDES`, +`TS_UNRESOLVED_IMPORT`) — is **L1 data, level-free**: identical at every `-a`. Neo4j is always +projected **full-depth** for the level actually analyzed (`--emit neo4j` + `-a`/`--graphs` +together is a CLI error, not a partial graph). + +## Identity in 20 seconds + +- **`can://` ids are opaque** — match on properties, never delimiter-split an id. +- Code ids are two-tier: **durable** at callable depth and above — + `can://///`, e.g. + `can://typescript/artifacts-app/src/config.ts/readHost` — and **ordinal** below it, appended + with `@`: `line:col` for statements/calls/`config_access`, `@entry`/`@exit`/`@formal_in:N`/ + `@formal_out` for synthetic vertices, `/actual_in:N`/`/actual_out` + for actuals. Example: `can://typescript/artifacts-app/src/config.ts/readHost@12:3`. +- **`Artifact.id` is language-neutral**: `can://artifact//` — no `typescript` segment, + so a TS and a Python analysis of one monorepo mint the *same* id for the same file (see the + `--app-name` trap below). +- **`Package.id` is a purl**: `pkg:npm/`, scoped `pkg:npm/%40scope/` — the + cross-language SBOM join key. +- **`ConfigKey.id`** is `@key/` (numeric segments for array indices, e.g. + `services.web.ports.0`). The `key` *property* is always the bare dotted name; the *id* alone can + carry an internal `arg.`/`env.` disambiguation prefix (see the `ARG` trap below) — never match on + a `key` substring expecting to see that prefix. + +## Standing traps + +**`direct` — dependency surface vs. supply chain.** + +> Your dependency *surface* and your dependency *supply chain* are different questions. +> "What does this app declare?" filters `direct: true`. "What actually ships / where does +> CVE-XXXX live?" needs the transitives — a vulnerable package four levels down is in your +> bundle whether or not you named it. + +Both recipes live side by side in `references/analyses.md` §6. Scoping note: every +`direct: false` record is a top-level lock entry carrying `kind: "runtime"` — a lock file does not +record *why* a package is present, so this layer asserts the safe default instead of inferring one. + +**Measured, so the magnitude is concrete, not asserted** (this repo, codeanalyzer-typescript +itself, `-a 1` — an illustrative sample, not a universal constant): 185 dependency records, 16 +`direct: true` / 169 `direct: false` — **91% of records are transitive**, invisible to the +declared-surface query. That is why the two recipes above return wildly different answers on the +same repo. Keeping them is cheap: `dependencies[]` is ~41 KB of a 2.67 MB payload (~1.5%), so the +transitive records are not what drives payload growth here — verbatim artifact `source` text is. + +**`config_reads` shrinks as `-a` rises — deliberately.** It is the layer's one non-monotonic +section (`config_uses` is the opposite: asserted superset-monotonic, L2 ⊆ L3 ⊆ L4 — verified on +this branch's fixture at 21/25/29 uses and 10/9/8 reads across L2/L3/L4). A read unresolved at the +literal tier can close at a higher dataflow tier, so it *moves* from `config_reads` into +`config_uses` as the level climbs. Diffing two levels and seeing a `config_reads` record vanish +means "resolved at the higher tier," never "fixed in the code." Pair the two: `config_uses` is the +graph edge (`TS_USES_CONFIG`); `config_reads` never became an edge — it is JSON-only, a record of +absence, not a graph fact (analyses.md §7). + +**`--app-name` is the cross-analyzer join precondition.** Artifact ids are language-neutral +specifically so a TS and a Python analysis of one repository MERGE onto the same `:Artifact` node +— but only if both runs pinned the *same* `--app-name`. Two analyzers pointed at different +subdirectories of one monorepo, or run with mismatched app names, will silently disagree on every +artifact and package id and never merge. + +**`sha256` is always the full file; `source` may not be — and in Neo4j, `Artifact` has no +`source` at all.** Under `--artifact-text-max-bytes`, `text_truncated: true` in `analysis.json` +means the stored `source` is a prefix — hash-compare on `sha256`, never on `source`, and never +re-derive meaning from a truncated copy (the analyzer itself always parses the full on-disk text +for extraction). In the graph, this is moot for a different reason: `Artifact` carries `sha256` + +`size_bytes` but **no `source` property at all** — verbatim text lives only in `analysis.json`, by +design (`src/build/neo4j/project.ts`: "`source` text stays off the graph — hash and size +dereference to it"). A query like `WHERE f.source CONTAINS "..."` will not error; it will silently +match nothing. + +**`value` is present by default and absent under `--no-artifact-text`** — an absent `ConfigKey.value` +is a capture setting, not an empty or unset config key. Check `references/vocabulary.md`'s +`extraction` field before treating a missing `value` as "this key has no value." + +**Dockerfile `ARG` is not bindable.** `ENV` mints an `env`-namespace key that a `process.env` read +can join; `ARG` mints a `dockerfile`-namespace key that deliberately never joins one — it exists at +build time only. A query that unions the two namespaces (or filters `ConfigKey` without checking +`namespace`) will report build-time values as runtime configuration. This is also why the config-key +*id* sometimes carries an `arg.`/`env.` prefix (see Identity above): the same bare name can mint +twice on one artifact — a Dockerfile's `ARG VERSION` beside its own `ENV VERSION=$VERSION` — and +only the id, never the `key` property, disambiguates them. + +**`config_access` nodes are reads, not calls.** They carry `root`/`key?` (JSON only — not +projected to Neo4j) but no `callee` at any level, so joining them through `TS_RESOLVES_TO` finds +nothing — that relationship only ever targets a `call` node's resolution. Follow `TS_USES_CONFIG` +from the body node straight to the `ConfigKey` instead. + +**`TSExternal` ghosts come in two grains that do not share an id.** Call-graph targets +(`.../@external//`) and dependency/import-hygiene ghosts (`.../@external/`, +no name segment) sit on the same label but different ids — join them by the `module` property when +a query needs both (`references/vocabulary.md`, "External ghosts"). + +**No entrypoint or import-graph vocabulary exists yet.** `TSCallable` carries no `is_entrypoint`; a +"reachable from the entrypoints" query needs a root set you supply yourself (analyses.md §2). A +module's `imports[]`/`exports[]` (specifiers, aliases, type-only flags) live only in +`analysis.json`'s `TSModule` — `TS_UNRESOLVED_IMPORT`/`TS_PROVIDES` cover the dependency-hygiene +case only, not a general per-module import graph. + +Every trap above is a silent-empty-result failure, not an error: Cypher does not reject a query +naming a nonexistent label, property, or relationship — it just returns nothing. Cross-check +`references/vocabulary.md` whenever a query returns an empty result you did not expect. diff --git a/docs/skills/analyzing-cants-graphs/references/analyses.md b/docs/skills/analyzing-cants-graphs/references/analyses.md new file mode 100644 index 0000000..c742ead --- /dev/null +++ b/docs/skills/analyzing-cants-graphs/references/analyses.md @@ -0,0 +1,246 @@ +# The analysis catalogue (Cypher) + +Every recipe names its minimum `-a`. Bound every transitive walk — unbounded `*1..` enumerates +paths (exponential on real corpora); use `*1..N` with `DISTINCT`, or `shortestPath` for existence +questions. Every property referenced below is declared in `src/build/neo4j/schema.ts` — cross-check +`references/vocabulary.md` before trusting a query that isn't on this page. + +## 1. Structure & inventory (L1) + +```cypher +// modules with their classes and free functions +MATCH (m:TSModule) +OPTIONAL MATCH (m)-[:TS_DECLARES]->(k:TSClass) +OPTIONAL MATCH (m)-[:TS_DECLARES]->(f:TSCallable) +RETURN m.name, count(DISTINCT k) AS classes, count(DISTINCT f) AS functions +ORDER BY m.name LIMIT 25 + +// locate a callable — the graph has no source text, only file + line span (see vocabulary.md) +MATCH (c:TSCallable {name: "analyze"}) +RETURN c.signature, c._module, c.start_line, c.end_line, c.cyclomatic_complexity +``` + +## 2. Call graph (L2) + +```cypher +// direct callers / callees +MATCH (caller:TSCallable)-[:TS_CALLS]->(t:TSCallable {name: "authorize"}) RETURN caller.id +MATCH (t:TSCallable {name: "authorize"})-[:TS_CALLS]->(callee) RETURN labels(callee), callee.id + +// bounded transitive reachability (who can reach the dataflow layer?) +MATCH (c:TSCallable)-[:TS_CALLS*1..8]->(t:TSCallable) +WHERE t._module STARTS WITH "src/dataflow" +RETURN DISTINCT c.id + +// fan-in / fan-out hotspots +MATCH (c:TSCallable) +OPTIONAL MATCH (c)<-[i:TS_CALLS]-() WITH c, count(i) AS fan_in +OPTIONAL MATCH (c)-[o:TS_CALLS]->() RETURN c.id, fan_in, count(o) AS fan_out +ORDER BY fan_in + count(o) DESC LIMIT 20 + +// provenance split: edges only one resolver found (prov ⊆ {tsc, defuse, import}) +MATCH ()-[e:TS_CALLS]->() WHERE e.prov = ["import"] RETURN count(e) + +// per-callsite resolution (which statement calls what) +MATCH (c:TSCallable)-[:TS_HAS_BODY_NODE]->(s:TSBodyNode {kind: "call"})-[:TS_RESOLVES_TO]->(t) +RETURN c.id, s.start_line, labels(t), t.id LIMIT 50 +``` + +**Reachability from a root set** (L2) — this schema has **no `is_entrypoint` flag**; supply the +root callable ids yourself (an HTTP handler, a CLI command, whatever your own convention is): + +```cypher +MATCH (c:TSCallable) WHERE NOT c.id IN $roots + AND NOT EXISTS { MATCH (r:TSCallable)-[:TS_CALLS*1..12]->(c) WHERE r.id IN $roots } +RETURN c.id, c._module, c.start_line +``` + +**Recursion cycles** (L2): + +```cypher +// self-recursion +MATCH (c:TSCallable)-[:TS_CALLS]->(c) RETURN c.id +// mutual recursion up to length 6, one row per cycle instance +MATCH p = (c:TSCallable)-[:TS_CALLS*2..6]->(c) +WHERE ALL(n IN nodes(p)[1..] WHERE n.id >= c.id) // canonical start, dedups rotations +RETURN [n IN nodes(p) | n.id] AS cycle LIMIT 50 +``` + +## 3. Inheritance (L1) + +`TS_EXTENDS`/`TS_IMPLEMENTS` are **resolved-only**: an external/library supertype never appears as +an edge target (no `TSExternal` endpoint exists for either relationship — check before writing a +query that assumes one). + +```cypher +// hierarchy under a base (bounded) +MATCH (base:TSClass {name: "Disposable"})<-[:TS_EXTENDS*1..6]-(sub:TSClass) RETURN sub.signature + +// overrides: subclass redefines a superclass method +MATCH (sub:TSClass)-[:TS_EXTENDS]->(sup:TSClass), + (sub)-[:TS_HAS_METHOD]->(m:TSCallable), + (sup)-[:TS_HAS_METHOD]->(base:TSCallable {name: m.name}) +RETURN sub.signature, m.name, base.id AS overrides +``` + +## 4. Control flow & data dependence (L3; alias-widened at L4) + +```cypher +// a callable's CFG in order +MATCH (c:TSCallable {name: "reconcile"})-[:TS_HAS_BODY_NODE]->(s:TSBodyNode) +OPTIONAL MATCH (s)-[n:TS_CFG_NEXT]->(t:TSBodyNode) +RETURN s.id, s.kind, s.start_line, collect({to: t.id, kind: n.kind}) ORDER BY s.start_line + +// unreachable statements (no CFG path from @entry) +MATCH (c:TSCallable)-[:TS_HAS_BODY_NODE]->(entry:TSBodyNode {kind: "entry"}) +MATCH (c)-[:TS_HAS_BODY_NODE]->(s:TSBodyNode) +WHERE s.kind IN ["statement", "call", "config_access"] + AND NOT EXISTS { MATCH (entry)-[:TS_CFG_NEXT*1..64]->(s) } +RETURN c.id, s.id, s.start_line + +// which condition guards this statement (control dependence) +MATCH (s:TSBodyNode {id: $stmt})<-[:TS_CDG]-(guard:TSBodyNode) RETURN guard.id, guard.kind, guard.start_line + +// complexity hotspots (precomputed) +MATCH (c:TSCallable) RETURN c.id, c.cyclomatic_complexity ORDER BY c.cyclomatic_complexity DESC LIMIT 20 + +// def-use chain of one variable inside a callable +MATCH (c:TSCallable {name: "applyDiscount"})-[:TS_HAS_BODY_NODE]->(a:TSBodyNode) +MATCH (a)-[d:TS_DDG {var: "total"}]->(b:TSBodyNode) +RETURN a.start_line, b.start_line, d.prov + +// syntactic-only view (drop the L4 alias-derived edges) +MATCH (a)-[d:TS_DDG]->(b) WHERE "reaching-defs" IN d.prov RETURN count(d) +``` + +## 5. Slicing & interprocedural reachability (L3 intra; L4 interproc) + +```cypher +// backward slice from a statement (intra) +MATCH (s:TSBodyNode {id: $global_id})<-[:TS_DDG|TS_CDG*1..10]-(dep:TSBodyNode) +RETURN DISTINCT dep.id, dep.start_line +// forward slice: reverse the arrow +// interprocedural: add TS_PARAM_IN|TS_PARAM_OUT|TS_SUMMARY to the union (L4), keep the bound + +// everything a chosen callable's parameters can influence (L4; pick $callable_sig yourself) +MATCH (e:TSCallable {signature: $callable_sig})-[:TS_HAS_BODY_NODE]->(src:TSBodyNode {kind: "formal_in"}) +MATCH (src)-[:TS_DDG|TS_PARAM_IN|TS_PARAM_OUT|TS_SUMMARY*1..12]->(s:TSBodyNode) +WHERE s.kind IN ["statement", "call", "config_access"] +RETURN DISTINCT src.of, s.id + +// source -> sink existence with witness path (shortestPath terminates where enumeration cannot) +MATCH (src:TSBodyNode {kind: "formal_in"})<-[:TS_HAS_BODY_NODE]-(e:TSCallable {signature: $callable_sig}) +MATCH (sink:TSBodyNode {kind: "call"})-[:TS_RESOLVES_TO]->(x:TSExternal) WHERE x.module = "child_process" +MATCH p = shortestPath((src)-[:TS_DDG|TS_PARAM_IN|TS_PARAM_OUT|TS_SUMMARY*..40]->(sink)) +RETURN e.id, x.name, [n IN nodes(p) | n.id] AS witness +``` + +This is graph substrate, not a taint product: the analyzer stops at the edges above and never +stores a `taint_flows` list. Composing source/sink packs over this reachability is the consuming +SDK's job. + +## 6. Dependencies, SBOM & artifacts (L1) + +Two different questions, two different filters — see SKILL.md's standing traps for the "why": + +```cypher +// declared surface only — "what does this app declare?" +MATCH (:Artifact)-[d:DECLARES_DEPENDENCY {direct: true}]->(p:Package) RETURN p.name, d.kind, d.spec; +// full shipped set, transitives included — "what actually ships / where does CVE-XXXX live?" +MATCH (:Artifact)-[d:DECLARES_DEPENDENCY]->(p:Package) +OPTIONAL MATCH (:Artifact)-[l:LOCKS]->(p) RETURN p.name, d.direct, coalesce(l.version, d.spec); +``` + +Measured on this repo (codeanalyzer-typescript itself, `-a 1` — illustrative, not a universal +constant): 185 records, 16 direct / 169 transitive (91% transitive). The first query alone would +have missed 91% of the dependency records the second one returns. `dependencies[]` payload is +~41 KB of a 2.67 MB total (~1.5%) — the transitive records are cheap to keep relative to total +payload. + + +More of the same layer: + +```cypher +// full SBOM row: declaring manifest, spec, pin, evidence +MATCH (f:Artifact)-[d:DECLARES_DEPENDENCY]->(p:Package) +OPTIONAL MATCH (lf:Artifact)-[l:LOCKS]->(p) +RETURN p.name, d.kind, d.spec, d.direct, l.version AS locked, f.path AS declared_in, d.prov +ORDER BY p.name + +// undeclared imports (dependency hygiene) +MATCH (a:TSApplication)-[u:TS_UNRESOLVED_IMPORT]->(x:TSExternal) +RETURN x.module, u.prov + +// which callables reach code from a declared package (blast radius) — TS_PROVIDES and TS_CALLS +// ghosts are DIFFERENT ids on the shared :TSExternal label; join on `module` (vocabulary.md) +MATCH (p:Package {id: "pkg:npm/commander"})-[:TS_PROVIDES]->(g:TSExternal) +MATCH (x:TSExternal) WHERE x.module = g.module +MATCH (c:TSCallable)-[:TS_CALLS*1..6]->(x) +RETURN DISTINCT c.id + +// declared but never imported (candidate dead dependency; heuristic — dynamic requires invisible) +MATCH (:Artifact)-[:DECLARES_DEPENDENCY]->(p:Package) +WHERE NOT (p)-[:TS_PROVIDES]->() RETURN p.name + +// spec-vs-lock drift +MATCH (f:Artifact)-[d:DECLARES_DEPENDENCY]->(p:Package)<-[l:LOCKS]-(:Artifact) +WHERE d.spec <> "" AND NOT l.version STARTS WITH replace(split(d.spec, ",")[0], "^", "") +RETURN p.name, d.spec, l.version + +// topology/container inventory — roles and hashes, NOT raw text (Artifact carries no `source` +// in Neo4j; read analysis.json for that, or re-open the file at `path`) +MATCH (:TSApplication)-[:HAS_ARTIFACT]->(f:Artifact) +WHERE any(r IN f.roles WHERE r IN ["service-topology", "container-image"]) +RETURN f.path, f.format, f.roles, f.sha256 + +// cross-language SBOM join point: purl ids are shared across sibling analyzers +MATCH (p:Package) WHERE p.id STARTS WITH "pkg:" RETURN p.ecosystem, count(*) +``` + +## 7. Config-use bridge (L2 literal; L3/L4 widen) + +```cypher +// which code reads a config key +MATCH (b:TSBodyNode)-[u:TS_USES_CONFIG]->(k:ConfigKey) RETURN b.id, k.key, k.namespace, u.prov; +// config reads nobody can trace (JSON only — not in the graph): analysis.json's +// application.config_reads[], { site, callee, key?, reason: "non-literal"|"undefined-key", prov } +``` + +```cypher +// blast radius of renaming a key: every reading body node + its owning callable +MATCH (k:ConfigKey {key: "PAYMENT_HOST"})<-[:TS_USES_CONFIG]-(b:TSBodyNode) +MATCH (c:TSCallable)-[:TS_HAS_BODY_NODE]->(b) +RETURN c.id, b.start_line + +// declaration <-> use: who defines a key, who reads it, does anyone +MATCH (a:Artifact)-[:DEFINES_CONFIG]->(k:ConfigKey {key: "PAYMENT_HOST"}) +OPTIONAL MATCH (k)<-[:TS_USES_CONFIG]-(b:TSBodyNode) +RETURN a.path, k.namespace, k.value, count(b) AS reads + +// Dockerfile ARG vs ENV: which of this image's keys are build-time only (never bindable) +MATCH (a:Artifact {format: "dockerfile"})-[:DEFINES_CONFIG]->(k:ConfigKey) +RETURN k.key, + CASE k.namespace WHEN "dockerfile" THEN "build-time only (ARG)" ELSE "runtime-bindable (ENV)" END AS binding + +// literal-tier-only view (drop the dataflow-widened edges) +MATCH ()-[u:TS_USES_CONFIG]->() WHERE u.prov = ["literal"] RETURN count(u) +``` + +Edges never guess: the literal tier (`-a 2`) needs a statically-known key at a recognized env root +or detector-listed call; the dataflow tiers (`-a 3` intra, `-a 4` interprocedural) resolve only +chains that close over exactly one string literal. Everything else lands in `config_reads` with a +reason — see SKILL.md's standing traps for why that list shrinks as `-a` rises. + +## 8. Health metrics (any level) + +```cypher +// external call surface per module (free functions only — methods hang off TSClass, not TSModule) +MATCH (m:TSModule)-[:TS_DECLARES]->(:TSCallable)-[:TS_CALLS]->(x:TSExternal) +RETURN m.name, count(DISTINCT x.module) AS external_modules ORDER BY external_modules DESC LIMIT 20 + +// orphan ghosts (referenced by nothing after filtering) +MATCH (x:TSExternal) WHERE NOT ()-[]->(x) RETURN count(x) + +// biggest callables by line span +MATCH (c:TSCallable) RETURN c.id, c.end_line - c.start_line AS lines ORDER BY lines DESC LIMIT 20 +``` diff --git a/docs/skills/analyzing-cants-graphs/references/vocabulary.md b/docs/skills/analyzing-cants-graphs/references/vocabulary.md new file mode 100644 index 0000000..e417973 --- /dev/null +++ b/docs/skills/analyzing-cants-graphs/references/vocabulary.md @@ -0,0 +1,104 @@ +# Neo4j vocabulary (authoritative: `schema.neo4j.json`; regenerate with `bun run gen:schema`) + +Source of truth: `src/build/neo4j/schema.ts` (`NODE_LABELS`/`REL_TYPES`), enforced by +`test/neo4j-schema.test.ts` — the emitter can never write an undeclared label, relationship, or +property. If a query below returns nothing, check the property is actually declared before +assuming the graph is empty. + +## Node labels + +| label | merge key | properties | notes | +| --- | --- | --- | --- | +| `TSApplication` | `id` | id, schema_version, language, max_level, k_limit, analyzer_name, analyzer_version | one per run; the `:Application` anchor | +| `Artifact` | `id` (`can://artifact//`) | id, kind, path, format, roles[], size_bytes, sha256, extraction | **language-neutral, no TS prefix by design** — sibling analyzers MERGE onto the same node. No `source`/`text_truncated`/`config_keys` here (see "No verbatim text in the graph" below) | +| `Package` | `id` (purl `pkg:npm/`, scoped `pkg:npm/%40scope/`) | id, ecosystem, name | language-neutral | +| `ConfigKey` | `id` (`@key/`) | id, key, namespace, value, references[] | language-neutral; `key` is always the bare dotted name even when `id` carries an internal `arg.`/`env.` disambiguation prefix (see SKILL.md's identity section) | +| `TSModule` | `id` | _module, content_hash, id, is_declaration_file, is_tsx, kind, name, start_line, end_line | `name` is the file key (e.g. `"src/config.ts"`, WITH extension) — same value as `_module` | +| `TSClass` | `id` | _module, base_classes[], id, implements_types[], is_abstract, is_ambient, is_exported, kind, name, signature, start_line, end_line | | +| `TSInterface` | `id` | _module, base_classes[], id, is_ambient, is_exported, kind, name, signature, start_line, end_line | | +| `TSEnum` | `id` | _module, id, is_ambient, is_const, is_exported, kind, name, signature, start_line, end_line | | +| `TSTypeAlias` | `id` | _module, aliased_type, id, is_ambient, is_exported, kind, name, signature, start_line, end_line | | +| `TSNamespace` | `id` | _module, id, is_ambient, is_exported, kind, name, signature, start_line, end_line | | +| `TSCallable` | `id` | _module, accessibility, accessor_kind, cyclomatic_complexity, id, is_abstract, is_ambient, is_async, is_exported, is_generator, is_implicit, is_static, kind, name, return_type, signature, start_line, end_line | function / method / constructor / getter / setter / arrow / function_expression — one label for all `TSCallableKind` values | +| `TSField` | `id` | _module, id, kind, name, type, start_line, end_line | module var, class/interface property, or enum member | +| `TSBodyNode` | `id` (GLOBAL ordinal) | _module, callee, id, kind, of, parent, start_line, end_line | see `TSBodyNode.kind` below; no `method_name`/`receiver_expr`/`argument_types`/`root`/`key` in Neo4j — those exist only in `analysis.json` | +| `TSExternal` | `id` | _module, id, kind, module, name | ghosts; **two grains**, same label — see "External ghosts" below | +| `TSAnonymousCallable` | `id` | (same property set as `TSCallable`, plus `path`, `start_column`) | **not a separate node** — a second label co-carried on the real `:TSCallable` tree node of an unnamed arrow/function expression (schema 2.1.0). `MATCH (c:TSAnonymousCallable)` finds exactly the anonymous ones; every other query against `:TSCallable` already includes them | + +`TSBodyNode.kind`: `entry`, `exit`, `statement`, `call`, `config_access`, `formal_in`, +`formal_out`, `actual_in`, `actual_out`. `call` nodes carry `callee` (null until `-a 2`); +`config_access` nodes never carry `callee` — they are reads, not calls (see SKILL.md's traps). + +`Artifact.format`: `json` \| `jsonc` \| `yaml` \| `toml` \| `ini` \| `dockerfile` \| `yarnlock` \| +`env` \| `text` \| `binary` (`src/artifacts/rules.ts`). `Artifact.roles` (list, unioned across every +matching rule): `dependency-manifest`, `tool-config`, `container-image`, `service-topology`, `ci`, +`env`, `packaging`, `legal`, `docs`, `script`, `unknown`. `Artifact.extraction`: `none` \| `partial` +\| `full`. `ConfigKey.namespace`: `env` \| `json` \| `yaml` \| `toml` \| `ini` \| `properties` \| +`dockerfile`. + +### No verbatim text in the graph + +Neither `TSModule` nor `TSCallable` nor `Artifact` carries source text, a file path, or column +positions in Neo4j — only `_module`/`path` (the file key) and `start_line`/`end_line`. This is a +deliberate design line (`src/build/neo4j/project.ts`: "`source` text stays off the graph — hash +and size dereference to it"), not an omission. To read exact text: re-open the file at +`start_line`/`end_line`, or read `analysis.json`, where every module's `source` is stored once and +every node's exact text is `source.slice(...span.bytes)`. + +### External ghosts (`TSExternal`) — two grains, one label + +- **Call-graph targets** (`src/schema/homing.ts::homeExternals`): `/@external//` + — one ghost per (module, member), the endpoint of `TS_CALLS`/`TS_RESOLVES_TO` for a call into a + library or Node builtin. +- **Dependency/import-hygiene ghosts** (`src/build/neo4j/project.ts::importGhost`): + `/@external/` — one ghost per import-specifier ROOT, the endpoint of `TS_PROVIDES` + and `TS_UNRESOLVED_IMPORT`. No `name` segment: `module` alone. + +These are **different ids on the same label** — a call-graph ghost for `express`'s `Router` member +does not share an id with the dependency ghost for the `express` package. Join them by the shared +`module` property when a query needs both (e.g. "which callables reach code from this declared +package" — `references/analyses.md` §6). + +## Relationships + +| type | from → to | properties | notes | +| --- | --- | --- | --- | +| `TS_HAS_MODULE` | TSApplication → TSModule | — | | +| `TS_DECLARES` | TSModule/TSNamespace/TSCallable → TSClass/TSInterface/TSEnum/TSTypeAlias/TSNamespace/TSCallable | — | containment; also how a nested/anonymous callable is reached | +| `TS_HAS_METHOD` | TSClass/TSInterface → TSCallable | — | | +| `TS_HAS_FIELD` | TSModule/TSClass/TSInterface/TSEnum/TSNamespace → TSField | — | | +| `TS_HAS_BODY_NODE` | TSCallable/TSAnonymousCallable → TSBodyNode | — | | +| `TS_RESOLVES_TO` | TSBodyNode → TSCallable/TSExternal/TSAnonymousCallable | — | per-callsite resolution (L2); only `call` nodes have an outgoing edge — `config_access` never does | +| `TS_CALLS` | TSCallable/TSAnonymousCallable → TSCallable/TSExternal/TSAnonymousCallable | weight, prov[] | condensed call graph; prov ⊆ {tsc, defuse, import} | +| `TS_EXTENDS` | TSClass/TSInterface → TSClass/TSInterface | — | resolved-only (external/library supertypes never reach here) | +| `TS_IMPLEMENTS` | TSClass → TSInterface/TSClass | — | | +| `TS_CFG_NEXT` | TSBodyNode → TSBodyNode | kind, `_k` | control flow; `_k` = `kind` (a conditional's true/false pair needs both edges to coexist) | +| `TS_CDG` | TSBodyNode → TSBodyNode | — | control dependence | +| `TS_DDG` | TSBodyNode → TSBodyNode | var, prov[], `_k` | one edge per (var, prov); prov ⊆ {reaching-defs (L3), points-to (L4)}; `_k` = `"\|"` | +| `TS_PARAM_IN` | caller `actual_in` → callee `formal_in` | var | L4 | +| `TS_PARAM_OUT` | callee `formal_out` → caller `actual_out` | var | L4 | +| `TS_SUMMARY` | `actual_in` → `actual_out` (same call site) | var | L4 transitive shortcut | +| `HAS_ARTIFACT` | TSApplication → Artifact | — | L1, level-free | +| `DECLARES_DEPENDENCY` | Artifact → Package | spec, kind, direct, extras[], prov[] | one edge per (artifact, package, kind); `direct: false` = lockfile-only transitive, never manifest-declared | +| `LOCKS` | Artifact (a lock file) → Package | version | fans from every lock artifact present (coarse fan — python's documented posture) | +| `DEFINES_CONFIG` | Artifact → ConfigKey | — | containment; level-free | +| `TS_PROVIDES` | Package → TSExternal (module-level ghost) | — | correlatable with the call-graph grain via `module` only (see "External ghosts" above), not the same id; `TS_`-prefixed because the claim is this analyzer's own | +| `TS_UNRESOLVED_IMPORT` | TSApplication → TSExternal (module-level ghost) | prov[] | undeclared-import hygiene signal | +| `TS_USES_CONFIG` | TSBodyNode → ConfigKey | prov[] | which read joins which key; prov ⊆ {literal (L2+), dataflow (L3 intra, L4 interproc — same tag both tiers)}; superset-monotonic `-a 2 ⊆ 3 ⊆ 4` | + +There is **no relationship for unresolved config reads** — `config_reads` (JSON: `site`, `callee`, +`key?`, `reason`, `prov[]`) is not projected; it records an absence, not a graph fact. There is also +**no import-graph relationship** — a module's `imports[]`/`exports[]` (with specifiers, aliases, +type-only flags) exist only in `analysis.json`'s `TSModule`; `TS_UNRESOLVED_IMPORT`/`TS_PROVIDES` +cover the dependency-hygiene case only, not a general per-module import graph. There is also **no +entrypoint vocabulary** — `TSCallable` carries no `is_entrypoint`/`entrypoint_frameworks`; this +analyzer does not (yet) detect framework entrypoints. + +All dataflow relationships are stored src→dst in the forward direction. + +Dependency `prov` vocabulary: `declared`, `lockfile`, `installed-metadata` (only with +`--resolve-installed`), `heuristic`. `TSDependency.kind` (`DECLARES_DEPENDENCY.kind`): `runtime` \| +`dev` \| `optional` \| `peer` \| `build` — `peer` is this analyzer's one coined additive token +against the shared cross-language enum (npm's contract-with-host has no analogue in it). A +`direct: false` record is always `kind: "runtime"`: a lock file does not record *why* a package is +present, and inferring one would need a whole-graph walk this layer deliberately does not do. diff --git a/graph.cypher b/graph.cypher new file mode 100644 index 0000000..8356717 --- /dev/null +++ b/graph.cypher @@ -0,0 +1,267 @@ +// ── constraints & indexes ── +CREATE CONSTRAINT application_id IF NOT EXISTS FOR (x:Application) REQUIRE x.id IS UNIQUE; +CREATE CONSTRAINT cannode_id IF NOT EXISTS FOR (x:CanNode) REQUIRE x.id IS UNIQUE; +CREATE INDEX callable_name IF NOT EXISTS FOR (c:TSCallable) ON (c.name); +CREATE INDEX cannode_kind IF NOT EXISTS FOR (n:CanNode) ON (n.kind); +CREATE INDEX cannode_module IF NOT EXISTS FOR (n:CanNode) ON (n._module); + +// ── wipe this project's prior subgraph (external targets are shared) ── +MATCH (a:Application {id: 'can://typescript/anon-app'}) +OPTIONAL MATCH (a)-[:TS_HAS_MODULE]->(m:TSModule) +OPTIONAL MATCH (m)-[:TS_DECLARES|TS_HAS_METHOD|TS_HAS_FIELD|TS_HAS_BODY_NODE*1..]->(x) +DETACH DELETE x, m, a; + +// ── nodes ── +UNWIND [ + {k: 'can://typescript/anon-app', p: {id: 'can://typescript/anon-app', schema_version: '2.1.0', language: 'typescript', max_level: 4, k_limit: 3, analyzer_name: 'codeanalyzer-typescript', analyzer_version: '1.0.0'}} +] AS row +MERGE (n:Application {id: row.k}) +SET n += row.p, n:TSApplication; +UNWIND [ + {k: 'can://typescript/anon-app/src/routes.ts', p: {id: 'can://typescript/anon-app/src/routes.ts', kind: 'module', name: 'src/routes.ts', is_tsx: false, is_declaration_file: false, start_line: 1, end_line: 22, _module: 'src/routes.ts'}} +] AS row +MERGE (n:CanNode {id: row.k}) +SET n += row.p, n:TSModule; +UNWIND [ + {k: 'can://typescript/anon-app/src/routes.ts/', p: {id: 'can://typescript/anon-app/src/routes.ts/', kind: 'arrow', signature: 'src/routes.', name: '(anonymous)', return_type: 'void', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: false, is_ambient: false, is_implicit: false, start_line: 13, end_line: 15, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/', p: {id: 'can://typescript/anon-app/src/routes.ts/login/', kind: 'arrow', signature: 'src/routes.login.', name: '(anonymous)', return_type: 'void', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: false, is_ambient: false, is_implicit: false, start_line: 2, end_line: 5, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer/', p: {id: 'can://typescript/anon-app/src/routes.ts/outer/', kind: 'arrow', signature: 'src/routes.outer.', name: '(anonymous)', return_type: '() => number', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: false, is_ambient: false, is_implicit: false, start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer//', p: {id: 'can://typescript/anon-app/src/routes.ts/outer//', kind: 'arrow', signature: 'src/routes.outer..', name: '(anonymous)', return_type: 'number', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: false, is_ambient: false, is_implicit: false, start_line: 20, end_line: 20, _module: 'src/routes.ts'}} +] AS row +MERGE (n:CanNode {id: row.k}) +SET n += row.p, n:TSCallable:TSAnonymousCallable; +UNWIND [ + {k: 'can://typescript/anon-app/src/routes.ts/@14:3', p: {id: 'can://typescript/anon-app/src/routes.ts/@14:3', kind: 'call', start_line: 14, end_line: 14, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_in:0', p: {id: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_in:0', kind: 'actual_in', of: 'arg0', parent: '14:3', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_out', p: {id: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_out', kind: 'actual_out', of: '$ret', parent: '14:3', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/@entry', kind: 'entry', start_line: 13, end_line: 15, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/@exit', kind: 'exit', start_line: 13, end_line: 15, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/@formal_in:0', p: {id: 'can://typescript/anon-app/src/routes.ts/@formal_in:0', kind: 'formal_in', of: 'req', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/@formal_in:1', p: {id: 'can://typescript/anon-app/src/routes.ts/@formal_in:1', kind: 'formal_in', of: 'res', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login@2:3', p: {id: 'can://typescript/anon-app/src/routes.ts/login@2:3', kind: 'statement', start_line: 2, end_line: 5, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/login@entry', kind: 'entry', start_line: 1, end_line: 6, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/login@exit', kind: 'exit', start_line: 1, end_line: 6, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/login@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@3:5', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@3:5', kind: 'statement', start_line: 3, end_line: 3, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@4:5', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@4:5', kind: 'call', callee: 'can://typescript/anon-app/src/routes.ts/query', start_line: 4, end_line: 4, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_in:0', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_in:0', kind: 'actual_in', of: 'arg0', parent: '4:5', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_out', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_out', kind: 'actual_out', of: '$ret', parent: '4:5', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@entry', kind: 'entry', start_line: 2, end_line: 5, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@exit', kind: 'exit', start_line: 2, end_line: 5, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@formal_in:0', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@formal_in:0', kind: 'formal_in', of: 'req', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@formal_in:1', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@formal_in:1', kind: 'formal_in', of: 'res', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/login/@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/login/@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/named@17:21', p: {id: 'can://typescript/anon-app/src/routes.ts/named@17:21', kind: 'statement', start_line: 17, end_line: 17, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/named@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/named@entry', kind: 'entry', start_line: 17, end_line: 17, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/named@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/named@exit', kind: 'exit', start_line: 17, end_line: 17, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/named@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/named@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer@20:3', p: {id: 'can://typescript/anon-app/src/routes.ts/outer@20:3', kind: 'statement', start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/outer@entry', kind: 'entry', start_line: 19, end_line: 21, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/outer@exit', kind: 'exit', start_line: 19, end_line: 21, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/outer@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', p: {id: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', kind: 'statement', start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer/@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/outer/@entry', kind: 'entry', start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer/@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/outer/@exit', kind: 'exit', start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer/@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/outer/@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', p: {id: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', kind: 'statement', start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer//@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/outer//@entry', kind: 'entry', start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer//@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/outer//@exit', kind: 'exit', start_line: 20, end_line: 20, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer//@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/outer//@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/query@9:3', p: {id: 'can://typescript/anon-app/src/routes.ts/query@9:3', kind: 'statement', start_line: 9, end_line: 9, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/query@entry', p: {id: 'can://typescript/anon-app/src/routes.ts/query@entry', kind: 'entry', start_line: 8, end_line: 10, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/query@exit', p: {id: 'can://typescript/anon-app/src/routes.ts/query@exit', kind: 'exit', start_line: 8, end_line: 10, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/query@formal_in:0', p: {id: 'can://typescript/anon-app/src/routes.ts/query@formal_in:0', kind: 'formal_in', of: 'sql', _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/query@formal_out', p: {id: 'can://typescript/anon-app/src/routes.ts/query@formal_out', kind: 'formal_out', of: '$ret', _module: 'src/routes.ts'}} +] AS row +MERGE (n:CanNode {id: row.k}) +SET n += row.p, n:TSBodyNode; +UNWIND [ + {k: 'can://typescript/anon-app/src/routes.ts/app', p: {id: 'can://typescript/anon-app/src/routes.ts/app', kind: 'field', name: 'app', type: 'any', start_line: 12, end_line: 12, _module: 'src/routes.ts'}} +] AS row +MERGE (n:CanNode {id: row.k}) +SET n += row.p, n:TSField; +UNWIND [ + {k: 'can://typescript/anon-app/src/routes.ts/login', p: {id: 'can://typescript/anon-app/src/routes.ts/login', kind: 'function', signature: 'src/routes.login', name: 'login', return_type: '(req: any, res: any) => void', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: true, is_ambient: false, is_implicit: false, start_line: 1, end_line: 6, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/named', p: {id: 'can://typescript/anon-app/src/routes.ts/named', kind: 'arrow', signature: 'src/routes.named', name: 'named', return_type: 'number', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: false, is_ambient: false, is_implicit: false, start_line: 17, end_line: 17, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/outer', p: {id: 'can://typescript/anon-app/src/routes.ts/outer', kind: 'function', signature: 'src/routes.outer', name: 'outer', return_type: '() => () => number', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: true, is_ambient: false, is_implicit: false, start_line: 19, end_line: 21, _module: 'src/routes.ts'}}, + {k: 'can://typescript/anon-app/src/routes.ts/query', p: {id: 'can://typescript/anon-app/src/routes.ts/query', kind: 'function', signature: 'src/routes.query', name: 'query', return_type: 'string', cyclomatic_complexity: 1, is_static: false, is_abstract: false, is_async: false, is_generator: false, is_exported: true, is_ambient: false, is_implicit: false, start_line: 8, end_line: 10, _module: 'src/routes.ts'}} +] AS row +MERGE (n:CanNode {id: row.k}) +SET n += row.p, n:TSCallable; + +// ── relationships ── +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/query', p: {weight: 1, prov: ['tsc']}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_CALLS]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/@entry', t: 'can://typescript/anon-app/src/routes.ts/@14:3', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login@entry', t: 'can://typescript/anon-app/src/routes.ts/login@2:3', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@entry', t: 'can://typescript/anon-app/src/routes.ts/login/@3:5', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@entry', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/named@entry', t: 'can://typescript/anon-app/src/routes.ts/named@17:21', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer@entry', t: 'can://typescript/anon-app/src/routes.ts/outer@20:3', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/@entry', t: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//@entry', t: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/query@entry', t: 'can://typescript/anon-app/src/routes.ts/query@9:3', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_CDG]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/@14:3', t: 'can://typescript/anon-app/src/routes.ts/@exit', k: 'exception', p: {kind: 'exception'}}, + {f: 'can://typescript/anon-app/src/routes.ts/@14:3', t: 'can://typescript/anon-app/src/routes.ts/@exit', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/@entry', t: 'can://typescript/anon-app/src/routes.ts/@14:3', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/login@2:3', t: 'can://typescript/anon-app/src/routes.ts/login@exit', k: 'return', p: {kind: 'return'}}, + {f: 'can://typescript/anon-app/src/routes.ts/login@entry', t: 'can://typescript/anon-app/src/routes.ts/login@2:3', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@3:5', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@4:5', t: 'can://typescript/anon-app/src/routes.ts/login/@exit', k: 'exception', p: {kind: 'exception'}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@4:5', t: 'can://typescript/anon-app/src/routes.ts/login/@exit', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@entry', t: 'can://typescript/anon-app/src/routes.ts/login/@3:5', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/named@17:21', t: 'can://typescript/anon-app/src/routes.ts/named@exit', k: 'return', p: {kind: 'return'}}, + {f: 'can://typescript/anon-app/src/routes.ts/named@entry', t: 'can://typescript/anon-app/src/routes.ts/named@17:21', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer@20:3', t: 'can://typescript/anon-app/src/routes.ts/outer@exit', k: 'return', p: {kind: 'return'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer@entry', t: 'can://typescript/anon-app/src/routes.ts/outer@20:3', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', t: 'can://typescript/anon-app/src/routes.ts/outer/@exit', k: 'exception', p: {kind: 'exception'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', t: 'can://typescript/anon-app/src/routes.ts/outer/@exit', k: 'return', p: {kind: 'return'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/@entry', t: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', t: 'can://typescript/anon-app/src/routes.ts/outer//@exit', k: 'exception', p: {kind: 'exception'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', t: 'can://typescript/anon-app/src/routes.ts/outer//@exit', k: 'return', p: {kind: 'return'}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//@entry', t: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', k: 'fallthrough', p: {kind: 'fallthrough'}}, + {f: 'can://typescript/anon-app/src/routes.ts/query@9:3', t: 'can://typescript/anon-app/src/routes.ts/query@exit', k: 'return', p: {kind: 'return'}}, + {f: 'can://typescript/anon-app/src/routes.ts/query@entry', t: 'can://typescript/anon-app/src/routes.ts/query@9:3', k: 'fallthrough', p: {kind: 'fallthrough'}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_CFG_NEXT {_k: row.k}]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/@entry', t: 'can://typescript/anon-app/src/routes.ts/@14:3', k: 'req.query.probe|reaching-defs', p: {var: 'req.query.probe', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/@entry', t: 'can://typescript/anon-app/src/routes.ts/@14:3', k: 'res.send|reaching-defs', p: {var: 'res.send', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/login@2:3', t: 'can://typescript/anon-app/src/routes.ts/login@formal_out', k: 'return|reaching-defs', p: {var: 'return', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@3:5', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5', k: 'email|reaching-defs', p: {var: 'email', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@entry', t: 'can://typescript/anon-app/src/routes.ts/login/@3:5', k: 'req.body.email|reaching-defs', p: {var: 'req.body.email', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/named@17:21', t: 'can://typescript/anon-app/src/routes.ts/named@formal_out', k: 'return|reaching-defs', p: {var: 'return', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer@20:3', t: 'can://typescript/anon-app/src/routes.ts/outer@formal_out', k: 'return|reaching-defs', p: {var: 'return', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer@entry', t: 'can://typescript/anon-app/src/routes.ts/outer@20:3', k: 'src/routes.named|reaching-defs', p: {var: 'src/routes.named', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', t: 'can://typescript/anon-app/src/routes.ts/outer/@formal_out', k: 'return|reaching-defs', p: {var: 'return', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/@entry', t: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', k: 'src/routes.named|reaching-defs', p: {var: 'src/routes.named', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', t: 'can://typescript/anon-app/src/routes.ts/outer//@formal_out', k: 'return|reaching-defs', p: {var: 'return', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//@entry', t: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', k: 'src/routes.named|reaching-defs', p: {var: 'src/routes.named', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/query@9:3', t: 'can://typescript/anon-app/src/routes.ts/query@formal_out', k: 'return|reaching-defs', p: {var: 'return', prov: ['reaching-defs']}}, + {f: 'can://typescript/anon-app/src/routes.ts/query@entry', t: 'can://typescript/anon-app/src/routes.ts/query@9:3', k: 'sql|reaching-defs', p: {var: 'sql', prov: ['reaching-defs']}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_DDG {_k: row.k}]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/login', t: 'can://typescript/anon-app/src/routes.ts/login/', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/', t: 'can://typescript/anon-app/src/routes.ts/outer//', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer', t: 'can://typescript/anon-app/src/routes.ts/outer/', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts', t: 'can://typescript/anon-app/src/routes.ts/', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts', t: 'can://typescript/anon-app/src/routes.ts/login', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts', t: 'can://typescript/anon-app/src/routes.ts/named', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts', t: 'can://typescript/anon-app/src/routes.ts/outer', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts', t: 'can://typescript/anon-app/src/routes.ts/query', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_DECLARES]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@14:3', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_in:0', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@formal_in:0', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@formal_in:1', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/', t: 'can://typescript/anon-app/src/routes.ts/@formal_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@3:5', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_in:0', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@formal_in:0', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@formal_in:1', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/', t: 'can://typescript/anon-app/src/routes.ts/login/@formal_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login', t: 'can://typescript/anon-app/src/routes.ts/login@2:3', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login', t: 'can://typescript/anon-app/src/routes.ts/login@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login', t: 'can://typescript/anon-app/src/routes.ts/login@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login', t: 'can://typescript/anon-app/src/routes.ts/login@formal_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/named', t: 'can://typescript/anon-app/src/routes.ts/named@17:21', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/named', t: 'can://typescript/anon-app/src/routes.ts/named@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/named', t: 'can://typescript/anon-app/src/routes.ts/named@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/named', t: 'can://typescript/anon-app/src/routes.ts/named@formal_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//', t: 'can://typescript/anon-app/src/routes.ts/outer//@20:22', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//', t: 'can://typescript/anon-app/src/routes.ts/outer//@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//', t: 'can://typescript/anon-app/src/routes.ts/outer//@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer//', t: 'can://typescript/anon-app/src/routes.ts/outer//@formal_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/', t: 'can://typescript/anon-app/src/routes.ts/outer/@20:16', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/', t: 'can://typescript/anon-app/src/routes.ts/outer/@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/', t: 'can://typescript/anon-app/src/routes.ts/outer/@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer/', t: 'can://typescript/anon-app/src/routes.ts/outer/@formal_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer', t: 'can://typescript/anon-app/src/routes.ts/outer@20:3', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer', t: 'can://typescript/anon-app/src/routes.ts/outer@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer', t: 'can://typescript/anon-app/src/routes.ts/outer@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/outer', t: 'can://typescript/anon-app/src/routes.ts/outer@formal_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/query', t: 'can://typescript/anon-app/src/routes.ts/query@9:3', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/query', t: 'can://typescript/anon-app/src/routes.ts/query@entry', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/query', t: 'can://typescript/anon-app/src/routes.ts/query@exit', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/query', t: 'can://typescript/anon-app/src/routes.ts/query@formal_in:0', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/query', t: 'can://typescript/anon-app/src/routes.ts/query@formal_out', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_HAS_BODY_NODE]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts', t: 'can://typescript/anon-app/src/routes.ts/app', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_HAS_FIELD]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app', t: 'can://typescript/anon-app/src/routes.ts', p: {}} +] AS row +MATCH (a:Application {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_HAS_MODULE]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_in:0', t: 'can://typescript/anon-app/src/routes.ts/query@formal_in:0', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_PARAM_IN]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/query@formal_out', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_out', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_PARAM_OUT]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/login/@4:5', t: 'can://typescript/anon-app/src/routes.ts/query', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_RESOLVES_TO]->(b) +SET r += row.p; +UNWIND [ + {f: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_in:0', t: 'can://typescript/anon-app/src/routes.ts/@14:3/actual_out', p: {}}, + {f: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_in:0', t: 'can://typescript/anon-app/src/routes.ts/login/@4:5/actual_out', p: {}} +] AS row +MATCH (a:CanNode {id: row.f}) +MATCH (b:CanNode {id: row.t}) +MERGE (a)-[r:TS_SUMMARY]->(b) +SET r += row.p; diff --git a/package.json b/package.json index 02c960b..300dd43 100644 --- a/package.json +++ b/package.json @@ -9,7 +9,7 @@ }, "scripts": { "start": "bun run src/index.ts", - "build": "bun build ./src/main.ts ./src/dataflow/worker.ts --compile --external @babel/preset-typescript --outfile dist/cants", + "build": "bun build ./src/main.ts ./src/dataflow/worker.ts --compile --outfile dist/cants", "gen:schema": "bun run src/index.ts --emit schema > schema.neo4j.json", "gen:readme": "bun run scripts/update-readme.ts", "test:container": "RUN_CONTAINER_TESTS=1 bun test test/neo4j-bolt.test.ts", @@ -19,17 +19,14 @@ "dependencies": { "commander": "^15.0.0", "neo4j-driver": "^5.28.0", - "ts-morph": "^28.0.0" + "ts-morph": "^28.0.0", + "yaml": "^2.9.0" }, "devDependencies": { - "@cs-au-dk/jelly": "0.13.0", "@testcontainers/neo4j": "^12.0.3", "@types/bun": "^1.3.14", "@types/node": "^25.9.1", "testcontainers": "^12.0.3", "typescript": "^6.0.3" - }, - "patchedDependencies": { - "@cs-au-dk/jelly@0.13.0": "patches/@cs-au-dk%2Fjelly@0.13.0.patch" } } diff --git a/packaging/python/build_wheels.sh b/packaging/python/build_wheels.sh index 061b3b8..65bbac6 100755 --- a/packaging/python/build_wheels.sh +++ b/packaging/python/build_wheels.sh @@ -77,12 +77,8 @@ for entry in "${TARGETS[@]}"; do clean_bin - # Entry is src/main.ts (the multi-call dispatcher that also embeds the Jelly CLI), NOT src/index.ts. - # --external @babel/preset-typescript: Jelly's Babel core dynamically require()s that preset; it is - # never loaded at runtime (Jelly sets babelrc/configFile false), so excluding it is safe and avoids - # a bundle-time resolution error. - ( cd "$REPO_ROOT" && bun build ./src/main.ts --compile --target="$target" \ - --external @babel/preset-typescript --outfile "$BIN_DIR/cants$ext" ) + # Entry is src/main.ts — NOT src/index.ts. + ( cd "$REPO_ROOT" && bun build ./src/main.ts --compile --target="$target" --outfile "$BIN_DIR/cants$ext" ) # Ship the Neo4j schema contract (platform-independent) next to the binary, so consumers can # read the version-locked schema.json without invoking the binary. See codeanalyzer_typescript.schema_path(). diff --git a/patches/@cs-au-dk%2Fjelly@0.13.0.patch b/patches/@cs-au-dk%2Fjelly@0.13.0.patch deleted file mode 100644 index 0a88e7b..0000000 --- a/patches/@cs-au-dk%2Fjelly@0.13.0.patch +++ /dev/null @@ -1,18 +0,0 @@ -diff --git a/lib/parsing/parser.js b/lib/parsing/parser.js -index de36b2fcf1d06cfa8703b0d2a3f0f2cd11c43557..eed24065f2a859b539bc974d6607bf9196d30e4d 100644 ---- a/lib/parsing/parser.js -+++ b/lib/parsing/parser.js -@@ -12,11 +12,11 @@ const transformOptions = [false, true].map((fragmentStateDefined) => (0, core_1. - cloneInputAst: false, - plugins: [ - extras_1.replaceTypeScriptImportExportAssignmentsAndAddConstructors, -- ['@babel/plugin-transform-typescript', { -+ [require('@babel/plugin-transform-typescript').default, { - onlyRemoveTypeImports: fragmentStateDefined, - allowDeclareFields: fragmentStateDefined, - }], -- ['@babel/plugin-transform-template-literals', { loose: true }] -+ [require('@babel/plugin-transform-template-literals').default, { loose: true }] - ], - cwd: __dirname, - babelrc: false, diff --git a/schema.neo4j.json b/schema.neo4j.json index 0ed6245..6584e49 100644 --- a/schema.neo4j.json +++ b/schema.neo4j.json @@ -17,6 +17,43 @@ "analyzer_version": "string" } }, + { + "label": "Artifact", + "mergeLabel": "Artifact", + "key": "id", + "properties": { + "id": "string", + "kind": "string", + "path": "string", + "format": "string", + "roles": "string[]", + "size_bytes": "integer", + "sha256": "string", + "extraction": "string" + } + }, + { + "label": "Package", + "mergeLabel": "Package", + "key": "id", + "properties": { + "id": "string", + "ecosystem": "string", + "name": "string" + } + }, + { + "label": "ConfigKey", + "mergeLabel": "ConfigKey", + "key": "id", + "properties": { + "id": "string", + "key": "string", + "namespace": "string", + "value": "string", + "references": "string[]" + } + }, { "label": "TSModule", "mergeLabel": "CanNode", @@ -224,6 +261,88 @@ ], "properties": {} }, + { + "type": "HAS_ARTIFACT", + "from": [ + "TSApplication" + ], + "to": [ + "Artifact" + ], + "properties": {} + }, + { + "type": "DECLARES_DEPENDENCY", + "from": [ + "Artifact" + ], + "to": [ + "Package" + ], + "properties": { + "spec": "string", + "kind": "string", + "direct": "boolean", + "extras": "string[]", + "prov": "string[]" + } + }, + { + "type": "LOCKS", + "from": [ + "Artifact" + ], + "to": [ + "Package" + ], + "properties": { + "version": "string" + } + }, + { + "type": "TS_PROVIDES", + "from": [ + "Package" + ], + "to": [ + "TSExternal" + ], + "properties": {} + }, + { + "type": "TS_UNRESOLVED_IMPORT", + "from": [ + "TSApplication" + ], + "to": [ + "TSExternal" + ], + "properties": { + "prov": "string[]" + } + }, + { + "type": "DEFINES_CONFIG", + "from": [ + "Artifact" + ], + "to": [ + "ConfigKey" + ], + "properties": {} + }, + { + "type": "TS_USES_CONFIG", + "from": [ + "TSBodyNode" + ], + "to": [ + "ConfigKey" + ], + "properties": { + "prov": "string[]" + } + }, { "type": "TS_DECLARES", "from": [ @@ -404,6 +523,9 @@ ], "constraints": [ "CREATE CONSTRAINT application_id IF NOT EXISTS FOR (x:Application) REQUIRE x.id IS UNIQUE", + "CREATE CONSTRAINT artifact_id IF NOT EXISTS FOR (x:Artifact) REQUIRE x.id IS UNIQUE", + "CREATE CONSTRAINT package_id IF NOT EXISTS FOR (x:Package) REQUIRE x.id IS UNIQUE", + "CREATE CONSTRAINT configkey_id IF NOT EXISTS FOR (x:ConfigKey) REQUIRE x.id IS UNIQUE", "CREATE CONSTRAINT cannode_id IF NOT EXISTS FOR (x:CanNode) REQUIRE x.id IS UNIQUE" ], "indexes": [ diff --git a/scripts/joern/compare_joern.py b/scripts/joern/compare_joern.py new file mode 100755 index 0000000..e46205c --- /dev/null +++ b/scripts/joern/compare_joern.py @@ -0,0 +1,181 @@ +#!/usr/bin/env python3 +"""Joern jssrc2cpg superset comparator (#98 / defuse-linker-call-graph.md). + +Usage: + 1. joern-parse --language jssrc -o app.cpg + 2. joern --script scripts/joern/dump-calls.sc --param cpgFile=app.cpg --param outFile=app.tsv + 3. python3 scripts/joern/compare_joern.py app.tsv [-v] + +Maps Joern's real call pairs onto analyzer signatures and verifies our call_graph covers them +(RESIDUAL must be 0). Exception classes are counted and printed, never silently waved through — +see docs/design/specs/defuse-linker-joern-ledger.md for the audited class definitions.""" +import json, re, subprocess, sys, collections + +JUNK_NAMES = {"__decorate", "__param", "__metadata", "__runInitializers", "__esDecorate", "require", "import"} + +def load_joern(tsv): + calls, methods = [], {} + params = {} + malformed = 0 + for line in open(tsv, errors="replace"): + parts = line.rstrip("\n").split("\t") + try: + if parts[0] == "C" and len(parts) == 6: + _, caller, name, direct, linked, line_no = parts + calls.append((caller, name, direct, linked, int(line_no))) + elif parts[0] == "M" and len(parts) == 4: + _, fn, ln, col = parts + methods[fn] = (int(ln), int(col)) + elif parts[0] == "P" and len(parts) == 3: + params.setdefault(parts[1], set()).add(parts[2]) + else: + malformed += 1 # identifiers containing tabs/newlines (template literals etc.) + except ValueError: + malformed += 1 + if malformed: + print(f" [note] {malformed} malformed dump rows skipped (control chars in identifiers)") + return calls, methods, params + +STRIP_EXT = re.compile(r"\.(d\.ts|tsx|ts|jsx|js|mts|cts|mjs|cjs)$") + +def build_indexes(our_sigs, edges): + """vscode-scale: pre-index anon sigs by (base, line) and edges by target.""" + import collections as _c + anon_ix = _c.defaultdict(list) + for sig in our_sigs: + if " our signature, or (None, reason).""" + if "::" not in fn: + return None, "external" + path, chain = fn.split("::", 1) + segs = chain.split(":") + if segs[0] != "program": + return None, "odd-chain" + segs = segs[1:] + prefix = STRIP_EXT.sub("", path) + if not segs: + return prefix, None # module-scope caller: the module prefix IS the source (python #131) + out = [] + consumed = ["program"] + for s in segs: + consumed.append(s) + if s == "" or s == "super": + out.append("constructor") + elif s.startswith(""): + # a lambda anywhere in the chain: line-match the progressive Joern fullName against + # our positional under the mapped base so far (pre-indexed) + jfn = path + "::" + ":".join(consumed) + ln = methods.get(jfn, (-1, -1))[0] + base = prefix + ("." + ".".join(out) if out else "") + cands = sorted(anon_ix.get((base, str(ln)), [])) + if len(cands) != 1: + return None, "lambda-unmapped" + out.append(cands[0].rsplit(".", 1)[1]) + else: + out.append(s) + return prefix + "." + ".".join(out), None + +def our_edges(fixture, dump=None): + if dump: + d = json.load(open(dump)) + else: + here = __import__("os").path.dirname(__import__("os").path.abspath(__file__)) + out = subprocess.run(["bun", "run", here + "/edges.ts", fixture], capture_output=True, text=True) + if out.returncode != 0: + print(out.stderr[-2000:]); sys.exit(1) + d = json.loads(out.stdout) + return set(map(tuple, d["edges"])), set(d["sigs"]) + +def main(fixture, tsv, dump=None): + calls, methods, jparams = load_joern(tsv) + edges, sigs = our_edges(fixture, dump) + anon_ix, edges_by_target, edges_by_src = build_indexes(sigs, edges) + covered, residual = [], [] + classes = collections.Counter() + seen = set() + for caller, name, direct, linked, line in calls: + if name in JUNK_NAMES or name == "": + classes["joern-synthetic-helper"] += 1; continue + cands = [c for c in linked.split("|") if c] if linked else [] + if len(cands) > 1: + # Joern's name-based candidate enumeration (their untyped-receiver fan) — python + # ledger's "speculative typed-attribute fan-out" class: not a resolution, not gated. + # Informational: does our graph cover at least one enumerated candidate? + classes["joern-name-fanout"] += 1 + src_f, _ = map_fullname(caller, methods, sigs, anon_ix) + hit = False + for cf in cands[:64]: + d_f, _ = map_fullname(cf, methods, sigs, anon_ix) + if d_f and src_f and (src_f, d_f) in edges: + hit = True; break + if hit: classes["joern-name-fanout-covered>=1"] += 1 + continue + callee_fn = cands[0] if cands else (direct if direct != "" else "") + if not callee_fn or callee_fn == "": + classes["joern-unresolved"] += 1; continue + src, why_s = map_fullname(caller, methods, sigs, anon_ix) + dst, why_d = map_fullname(callee_fn, methods, sigs, anon_ix) + if dst is None or dst not in sigs: + k = "external-or-unmapped-target:" + (why_d or "notin") + classes[k] += 1 + if "-v" in sys.argv and (why_d or "notin") != "external": print(" [", k, "]", caller, "->", callee_fn) + continue + if src is None or src not in sigs: + classes["caller-unmapped:" + (why_s or "notin")] += 1 + if "-v" in sys.argv: print(" [caller-unmapped]", caller, "->", callee_fn) + continue + pair = (src, dst) + if pair in seen: continue + seen.add(pair) + if pair in edges: covered.append(pair) + elif "." not in src and any(e0.startswith(src + ".") for e0 in edges_by_target.get(dst, ())): + # Joern desugars decorator factories to module-scope __decorate calls; we attribute the + # SAME invocation to the decorated callable (more precise). Same edge, finer caller. + classes["decorator-attribution-variant"] += 1 + covered.append(pair) + else: + # Joern this-misresolution variant: their single "resolution" names a free function + # ., while we hold, from the SAME caller, a typed edge to a METHOD + # .. of the same file+name (or vice versa). The receiver in source + # decides which is real; the checker types receivers, their name-link does not. + dfile, _, dname = dst.rpartition(".") + variant = False + for our_dst in edges_by_src.get(src, ()): + if our_dst == dst: continue + if our_dst.rsplit(".", 1)[-1] == dname and (our_dst.startswith(dfile + ".") or dst.startswith(our_dst.rsplit(".", 2)[0] + ".")): + variant = True; break + if variant: + classes["joern-this-misresolution (typed edge held)"] += 1 + covered.append(pair) + elif dname in jparams.get(caller, ()): + # The target's leaf name is a PARAMETER of the Joern caller: `new Promise(resolve + # => … resolve())` name-linked to a real free `resolve` — their parameters-as- + # callees family wearing a real name. Proven by their own parameter table. + classes["joern-param-shadow (fabricated target)"] += 1 + elif any(t.rsplit(".", 1)[-1] == dname for t in edges_by_src.get(src, ())): + # Weaker tier: from the same caller we hold a typed edge to a target of the SAME + # LEAF NAME in another file (e.g. the imported free `dispose` from lifecycle.ts, + # where Joern name-linked ActionBar.dispose). The checker resolved the receiver; + # their single-candidate name-link did not. + classes["joern-name-misresolution (typed same-name edge held)"] += 1 + covered.append(pair) + else: + residual.append(pair) + print(f"== {fixture}: joern real pairs {len(seen)}, covered {len(covered)}, RESIDUAL {len(residual)}") + for p in residual: print(" MISSING:", p[0], "->", p[1]) + for k, v in sorted(classes.items()): print(f" [class] {k}: {v}") + +if __name__ == "__main__": + dumps = [a for a in sys.argv[3:] if a.endswith(".json")] + main(sys.argv[1], sys.argv[2], dumps[0] if dumps else None) diff --git a/scripts/joern/dump-calls.sc b/scripts/joern/dump-calls.sc new file mode 100644 index 0000000..e736dae --- /dev/null +++ b/scripts/joern/dump-calls.sc @@ -0,0 +1,19 @@ +// Streams rows to disk (no in-memory StringBuilder — a vscode-scale dump with parameter rows +// exceeds the JVM's 2GB array cap otherwise). +@main def main(cpgFile: String, outFile: String) = { + importCpg(cpgFile) + val pw = new java.io.PrintWriter(new java.io.BufferedWriter(new java.io.FileWriter(outFile), 1 << 20)) + cpg.call.foreach { c => + if (!c.name.startsWith(" + pw.println(s"M\t${m.fullName}\t${m.lineNumber.getOrElse(-1)}\t${m.columnNumber.getOrElse(-1)}") + m.parameter.foreach { p => pw.println(s"P\t${m.fullName}\t${p.name}") } + } + pw.close() +} diff --git a/scripts/joern/edges.ts b/scripts/joern/edges.ts new file mode 100644 index 0000000..11bac19 --- /dev/null +++ b/scripts/joern/edges.ts @@ -0,0 +1,23 @@ +/** Ledger helper (#98): print the analyzer's L2 signature-level edge set + signature universe + * for a target app as JSON — consumed by compare_joern.py. */ +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { analyze } from "../../src/core"; +import { forEachCallable } from "../../src/schema"; +import type { AnalysisOptions } from "../../src/options"; + +const input = path.resolve(process.argv[2] as string); +const opts = { + input, output: null, emit: "json", appName: null, neo4jUri: null, neo4jUser: "neo4j", + neo4jPassword: "", neo4jDatabase: null, analysisLevel: 2, graphs: [], graphFieldDepth: 3, + jobs: 1, targetFiles: null, skipTests: true, eager: true, noBuild: true, phantoms: true, + cacheDir: fs.mkdtempSync(path.join(os.tmpdir(), "ledger-")), verbosity: 0, +} as AnalysisOptions; +const r = await analyze(opts); +const sigs: string[] = []; +for (const [fileKey, mod] of Object.entries(r.internal.symbol_table)) { + sigs.push(fileKey.replace(/\.d\.ts$/, "").replace(/\.(tsx|ts|jsx|js|mts|cts|mjs|cjs)$/, "")); + forEachCallable(mod, (c) => sigs.push(c.signature)); +} +process.stdout.write(JSON.stringify({ edges: r.internal.call_graph.map((e) => [e.source, e.target]), sigs })); diff --git a/src/artifacts/binding.ts b/src/artifacts/binding.ts new file mode 100644 index 0000000..5238a4d --- /dev/null +++ b/src/artifacts/binding.ts @@ -0,0 +1,79 @@ +/** + * Import→dependency binding (#101, python PR #160's `unresolved_imports`): every non-relative, + * non-builtin import specifier root the symbol table saw, checked against the declared records. + * A VALUE import of `x` needs the runtime package `x`; an `import type` is satisfiable by + * `@types/x` alone (bound_to it) — the spec'd TS rule. `--resolve-installed` additionally probes + * node_modules metadata (prov "installed-metadata"); default runs read only repo files. + */ +import * as fs from "node:fs"; +import * as path from "node:path"; +import type { TSDependency, TSImportBinding, TSModule } from "../schema"; + +/** The package root of an import specifier ("express/lib/router" → "express"; scoped keeps 2). */ +export function specifierRoot(spec: string): string | null { + if (spec.startsWith(".") || spec.startsWith("/") || spec.startsWith("#")) return null; // relative/self + if (spec.startsWith("node:")) return null; // builtin + const parts = spec.split("/"); + if (spec.startsWith("@")) return parts.length >= 2 ? `${parts[0]}/${parts[1]}` : null; + const root = parts[0] as string; + return NODE_BUILTINS.has(root) ? null : root; +} + +const NODE_BUILTINS = new Set([ + "assert", "async_hooks", "buffer", "child_process", "cluster", "console", "constants", "crypto", + "dgram", "diagnostics_channel", "dns", "domain", "events", "fs", "http", "http2", "https", + "inspector", "module", "net", "os", "path", "perf_hooks", "process", "punycode", "querystring", + "readline", "repl", "stream", "string_decoder", "timers", "tls", "trace_events", "tty", "url", + "util", "v8", "vm", "wasi", "worker_threads", "zlib", +]); + +export function bindImports( + symbol_table: Record, + deps: TSDependency[], + projectRoot: string, + resolveInstalled: boolean, +): TSImportBinding[] { + // specifier root → was it ever imported as a VALUE (vs exclusively type-only)? + const valueImport = new Map(); + for (const mod of Object.values(symbol_table)) { + for (const im of mod.imports) { + const root = specifierRoot(im.module); + if (!root) continue; + valueImport.set(root, (valueImport.get(root) ?? false) || !im.is_type_only); + } + } + + const provided = new Map(); + for (const dep of deps) for (const p of dep.provides_imports) if (!provided.has(p)) provided.set(p, dep); + + const out: TSImportBinding[] = []; + for (const [root, isValue] of [...valueImport.entries()].sort()) { + const direct = provided.get(root); + if (direct && (direct.name === root || !isValue)) continue; // runtime-declared, or types satisfy a type-only import + if (direct && direct.name.startsWith("@types/") && isValue) { + // Only @types declared, but the import is a VALUE use — partially bound, still unresolved. + out.push({ module: root, bound_to: direct.name, prov: ["heuristic"] }); + continue; + } + if (resolveInstalled) { + const version = installedVersion(projectRoot, root); + if (version !== null) { + out.push({ module: root, bound_to: root, prov: ["installed-metadata"] }); + continue; + } + } + out.push({ module: root, prov: [] }); + } + return out; +} + +/** Opt-in probe: node_modules//package.json version (never runs on default analyses). */ +export function installedVersion(projectRoot: string, name: string): string | null { + try { + const p = path.join(projectRoot, "node_modules", ...name.split("/"), "package.json"); + const doc = JSON.parse(fs.readFileSync(p, "utf-8")) as { version?: unknown }; + return typeof doc.version === "string" ? doc.version : null; + } catch { + return null; + } +} diff --git a/src/artifacts/configKeys.ts b/src/artifacts/configKeys.ts new file mode 100644 index 0000000..e41afe3 --- /dev/null +++ b/src/artifacts/configKeys.ts @@ -0,0 +1,142 @@ +/** + * Config-key extraction (#101 unit B): a config-bearing artifact's text → flattened dotted keys. + * Pure overlay — every parser returns [] on failure so the artifact node survives (the caller + * marks `extraction: "partial"`). Parses the FULL on-disk text, never the stored `source`. + */ +import type { TSConfigKey, TSSpan } from "../schema"; +import { parseYamlKeys } from "./yamlKeys"; + +const PLACEHOLDER = /\$\{([A-Za-z_][A-Za-z0-9_]*)\}|\$([A-Za-z_][A-Za-z0-9_]*)/g; + +export function referencesOf(value: unknown): string[] { + if (typeof value !== "string") return []; + const out: string[] = []; + for (const m of value.matchAll(PLACEHOLDER)) { + const name = m[1] ?? m[2]; + if (name && !out.includes(`env:${name}`)) out.push(`env:${name}`); + } + return out; +} + +const scalar = (v: unknown): v is string | number | boolean => + typeof v === "string" || typeof v === "number" || typeof v === "boolean"; + +export function keyNode(key: string, namespace: string, value: unknown, span?: TSSpan): TSConfigKey { + return { + id: "", + key, + namespace, + ...(scalar(value) ? { value } : {}), + ...(span ? { span } : {}), + references: referencesOf(value), + }; +} + +/** + * Strip `//`/block comments, then trailing commas — tsconfig/rc files are JSONC. TWO + * passes, each independently string-aware (the string alternative is tried first in both, + * so it always wins for anything that opens with `"` and is returned verbatim): + * 1. remove comments — so a trailing comma separated from its `}`/`]` only by a comment + * (`1, // note\n}`) is reachable by pass 2; + * 2. collapse `,\s*[}\]]`, but only outside strings — so `"hi, }"` and `"dist/{cjs,}"` + * still survive byte-for-byte. + * One merged pass can't do both: by the time it would strip a trailing comma, a comment + * sitting between the comma and the bracket hasn't been removed yet. + */ +export function parseJsonc(text: string): unknown { + const noComments = text.replace( + /"(?:[^"\\]|\\.)*"|\/\*[\s\S]*?\*\/|\/\/[^\n]*/g, + (m) => (m.startsWith('"') ? m : ""), + ); + const stripped = noComments.replace( + /"(?:[^"\\]|\\.)*"|,\s*([}\]])/g, + (m, bracket?: string) => (m.startsWith('"') ? m : (bracket ?? "")), + ); + return JSON.parse(stripped); +} + +function flatten(doc: unknown, prefix: string, out: TSConfigKey[], ns: string, depth: number): void { + if (depth > 24 || doc === null || typeof doc !== "object") return; + const entries: Array<[string, unknown]> = Array.isArray(doc) + ? doc.map((v, i) => [String(i), v] as [string, unknown]) + : Object.entries(doc as Record); + for (const [k, v] of entries) { + const dotted = prefix ? `${prefix}.${k}` : k; + if (scalar(v)) out.push(keyNode(dotted, ns, v)); + else flatten(v, dotted, out, ns, depth + 1); + } +} + +const ENV_LINE = /^\s*(?:export\s+)?([A-Za-z_][A-Za-z0-9_.]*)\s*=\s*(.*?)\s*$/; + +// A key set twice in one file keeps the LAST occurrence (shell/dotenv override semantics, +// parseDockerfileEnv precedent) — keyed by namespace:key so at most one TSConfigKey per +// (namespace, key) comes out of one file. Ids are (artifact, namespace, key)-derived, so two +// records sharing an id is exactly the collision Neo4j's id-dedup would otherwise mask. +export function parseEnvKeys(text: string): TSConfigKey[] { + const byKey = new Map(); + const lines = text.split("\n"); + lines.forEach((line, i) => { + if (!line.trim() || line.trim().startsWith("#")) return; + const m = ENV_LINE.exec(line); + if (!m) return; + let value = m[2] as string; + if ((value.startsWith('"') && value.endsWith('"')) || (value.startsWith("'") && value.endsWith("'"))) { + value = value.slice(1, -1); + } + const span: TSSpan = { start: [i + 1, 1], end: [i + 1, line.length + 1], bytes: [0, 0] }; + byKey.set(`env:${m[1]}`, keyNode(m[1] as string, "env", value, span)); // later line wins + }); + return [...byKey.values()]; +} + +/** INI / .properties: `[section]` prefixes a dotted key space. Last occurrence wins — see parseEnvKeys. */ +export function parseIniKeys(text: string, namespace: string): TSConfigKey[] { + const byKey = new Map(); + let section = ""; + text.split("\n").forEach((line, i) => { + const t = line.trim(); + if (!t || t.startsWith("#") || t.startsWith(";")) return; + const sec = /^\[([^\]]+)\]$/.exec(t); + if (sec) { + section = sec[1] as string; + return; + } + const eq = t.indexOf("="); + if (eq <= 0) return; + const key = t.slice(0, eq).trim(); + const value = t.slice(eq + 1).trim(); + const span: TSSpan = { start: [i + 1, 1], end: [i + 1, line.length + 1], bytes: [0, 0] }; + const dotted = section ? `${section}.${key}` : key; + byKey.set(`${namespace}:${dotted}`, keyNode(dotted, namespace, value, span)); // later line wins + }); + return [...byKey.values()]; +} + +/** + * Dispatch by artifact format. A dependency manifest or lockfile is never also a config file + * (codeanalyzer-python v1.3.0 parity) — gated on `roles`, not `format`, since both package.json + * (json) and lockfiles (json/jsonc) would otherwise pass the format check below. YAML is handled + * by yamlKeys.ts (Task 5). + */ +export function extractConfigKeys(format: string, roles: string[], text: string): TSConfigKey[] { + if (roles.includes("dependency-manifest")) return []; + switch (format) { + case "env": + return parseEnvKeys(text); + case "json": + case "jsonc": { + const out: TSConfigKey[] = []; + flatten(parseJsonc(text), "", out, "json", 0); + return out; + } + case "ini": + return parseIniKeys(text, "ini"); + case "properties": + return parseIniKeys(text, "properties"); + case "yaml": + return parseYamlKeys(text); + default: + return []; + } +} diff --git a/src/artifacts/deployEnv.ts b/src/artifacts/deployEnv.ts new file mode 100644 index 0000000..7cc9320 --- /dev/null +++ b/src/artifacts/deployEnv.ts @@ -0,0 +1,130 @@ +/** + * Deployment-env sources (#101 unit D): Dockerfile ENV, compose `environment`, and k8s container + * `env` mint BINDABLE `env`-namespace keys — the ones a later `process.env.X` read joins — in + * addition to whatever structural key the file already produced. Dockerfile ARG mints a + * `dockerfile`-namespace key that is deliberately NON-bindable: build-time only, never joins a + * runtime read. + * + * `deploymentEnvKeys` is role-gated exactly like `extractConfigKeys` (a dependency-manifest — + * e.g. pnpm-lock.yaml, format "yaml" — must not gain env keys either, same as it gains no + * structural ones). For "yaml" it takes the ALREADY-flattened keys `extractConfigKeys` / + * `parseYamlKeys` produced for this same artifact rather than re-parsing the text: one YAML parse + * per artifact. A throw from that upstream parse (malformed compose/k8s YAML) is not this + * module's problem — the caller's try/catch around the structural pass already turns it into + * `extraction: "partial"` before this function is ever reached, so a broken document mints zero + * deploy keys, not a second throw. Dockerfile has no structural pass to reuse, so it parses + * `text` directly — a line-based scan that can never throw, unlike JSON.parse/YAML.parse. + */ +import { keyNode } from "./configKeys"; +import type { TSConfigKey, TSSpan } from "../schema"; + +const DOCKER_LINE = /^\s*(ENV|ARG)\s+(.*)$/i; + +// POSIX-shell variable-name grammar (python v1.3.0's _ENV_KEY_NAME, adopted verbatim): gates +// which compose list-form leaves are real env-var declarations vs. junk. +const ENV_KEY_NAME = /^[A-Za-z_][A-Za-z0-9_]*$/; + +/** + * Line-based, never throws — a line that doesn't match is just skipped, never a reason to mark + * the whole Dockerfile "partial". `ENV K=V` and the legacy `ENV K V` both occur; ARG may be bare + * (`ARG X`, no default). A key redefined on a later line (a real Dockerfile pattern — e.g. `ARG + * VERSION` then `ENV VERSION=$VERSION`: same bare name, but ARG/ENV land in different namespaces + * so that particular pair never collides) keeps the LAST occurrence's value, matching Docker's + * own build-time override semantics — keyed by `namespace:name` in a Map so at most one + * TSConfigKey per (namespace, key) ever comes out of one Dockerfile. + */ +export function parseDockerfileEnv(text: string): TSConfigKey[] { + const byKey = new Map(); + text.split("\n").forEach((line, i) => { + const m = DOCKER_LINE.exec(line); + if (!m) return; + const directive = (m[1] as string).toUpperCase(); + const rest = (m[2] as string).trim(); + const span: TSSpan = { start: [i + 1, 1], end: [i + 1, line.length + 1], bytes: [0, 0] }; + const eq = rest.indexOf("="); + const [name, raw] = eq > 0 ? [rest.slice(0, eq), rest.slice(eq + 1)] : (() => { + const sp = rest.indexOf(" "); + return sp > 0 ? [rest.slice(0, sp), rest.slice(sp + 1)] : [rest, ""]; + })(); + const key = (name as string).trim(); + if (!key) return; // a bare "ENV " line with nothing after it — no name to mint + let value = (raw as string).trim(); + if ((value.startsWith('"') && value.endsWith('"')) || (value.startsWith("'") && value.endsWith("'"))) { + value = value.slice(1, -1); + } + const namespace = directive === "ENV" ? "env" : "dockerfile"; + byKey.set(`${namespace}:${key}`, keyNode(key, namespace, value, span)); // later line wins + }); + return [...byKey.values()]; +} + +/** + * compose `services..environment` (map or list form) and k8s + * `(spec|template.spec).containers[].env[].name`/`.value` → bindable env keys, matched against + * the ALREADY-flattened yaml keys (see module docstring — one parse per artifact). A + * multi-document stream prefixes every flattened key with its zero-based document index + * (`0.services…`, `1.spec…` — yamlKeys.ts), so both compose patterns below — anchored at + * `^services` — spell out an optional leading `.` segment; the k8s pattern's leading + * `.*` already absorbs that prefix without needing the same treatment. + */ +export function yamlEnvKeys(flat: TSConfigKey[]): TSConfigKey[] { + const out: TSConfigKey[] = []; + const seen = new Set(); + const push = (name: string, value: unknown, span?: TSSpan): void => { + if (!name || seen.has(name)) return; // first occurrence wins — deterministic, source order + seen.add(name); + out.push(keyNode(name, "env", value, span)); + }; + for (const k of flat) { + // compose list form: [doc.]services..environment. = "NAME=value". Checked BEFORE the + // map-form pattern below, even though map-form's `[^.]+` would also match a bare digit: a + // real env var name can never be pure digits (POSIX naming), so a numeric last segment is + // unambiguously a list index, never a name — and map-form must not steal it and mint a + // variable literally called "0". + const composeList = /^(?:\d+\.)?services\.[^.]+\.environment\.\d+$/.exec(k.key); + if (composeList && typeof k.value === "string") { + // "KEY=value" AND bare "KEY" both mint — compose's own syntax for a bare list entry is + // "inherit this variable from the host environment", a real bindable declaration, not + // dropped convenience. "KEY=" (an "=" present, empty right side) mints an EMPTY-STRING + // value; bare "KEY" (no "=" at all) mints NO value — same valueless shape as the k8s + // valueFrom case below. Either way, a name that isn't a valid env var name stays dropped. + const eq = k.value.indexOf("="); + const name = eq >= 0 ? k.value.slice(0, eq) : k.value; + if (ENV_KEY_NAME.test(name)) push(name, eq >= 0 ? k.value.slice(eq + 1) : undefined, k.span); + continue; + } + // compose map form: [doc.]services..environment. + const compose = /^(?:\d+\.)?services\.[^.]+\.environment\.([^.]+)$/.exec(k.key); + if (compose) { + push(compose[1] as string, k.value, k.span); + continue; + } + // k8s: [doc.](...).containers..env..name = NAME (value on the sibling ".value" key; + // absent when the entry uses valueFrom (secretRef/configMapRef) instead of a literal — mint + // the key with no value rather than dropping it, the name is still real). + const k8s = /^(.*\.containers\.\d+\.env\.\d+)\.name$/.exec(k.key); + if (k8s && typeof k.value === "string") { + const sibling = flat.find((x) => x.key === `${k8s[1]}.value`); + push(k.value, sibling?.value, k.span); + } + } + return out; +} + +/** + * Every bindable deployment-env key an artifact contributes, by format. Gated on `roles` exactly + * like `extractConfigKeys` — a dependency-manifest never gains deploy keys either, regardless of + * format. `flatYamlKeys` is the SAME array `extractConfigKeys` already produced for this artifact + * (unused for "dockerfile"; required for "yaml" — see yamlEnvKeys above). + */ +export function deploymentEnvKeys( + format: string, + roles: string[], + text: string, + flatYamlKeys: TSConfigKey[] = [], +): TSConfigKey[] { + if (roles.includes("dependency-manifest")) return []; + if (format === "dockerfile") return parseDockerfileEnv(text); + if (format === "yaml") return yamlEnvKeys(flatYamlKeys); + return []; +} diff --git a/src/artifacts/deps.ts b/src/artifacts/deps.ts new file mode 100644 index 0000000..302141c --- /dev/null +++ b/src/artifacts/deps.ts @@ -0,0 +1,152 @@ +/** + * Dependency extraction (#101, python PR #160 parity): `package.json` manifests → FLAT + * evidence-tagged `TSDependency` records on the application (`direct: true`); the JSON lockfile + * family backfills `locked_version` on those OWNING manifest's records (`prov` gains "lockfile"), + * and also mints its own `direct: false` records for packages it pins that no manifest declares + * (the transitive supply chain — see `transitiveRecords`). Defensive throughout — a malformed + * file yields no records, never an exception. + */ +import type { TSDependency } from "../schema"; + +const SECTION_KIND: ReadonlyArray<[string, TSDependency["kind"]]> = [ + ["dependencies", "runtime"], + ["devDependencies", "dev"], + ["optionalDependencies", "optional"], + ["peerDependencies", "peer"], // the spec'd additive npm token +]; + +/** Import specifiers this distribution provides: itself; `@types/x` also provides types-for-x. */ +function providesOf(name: string): string[] { + if (name.startsWith("@types/")) { + const base = name.slice("@types/".length); + // DefinitelyTyped mangles scoped names: @types/scope__pkg types @scope/pkg. + const real = base.includes("__") ? `@${base.replace("__", "/")}` : base; + return [name, real]; + } + return [name]; +} + +export function parsePackageJson(text: string, declaredIn: string): TSDependency[] { + let doc: unknown; + try { + doc = JSON.parse(text); + } catch { + return []; + } + if (typeof doc !== "object" || doc === null) return []; + const out: TSDependency[] = []; + const seen = new Set(); + for (const [section, kind] of SECTION_KIND) { + const block = (doc as Record)[section]; + if (typeof block !== "object" || block === null) continue; + for (const [name, spec] of Object.entries(block as Record)) { + if (seen.has(name)) continue; // first section wins (npm merge order above) + seen.add(name); + out.push({ + name, + spec: typeof spec === "string" ? spec : "", + kind, + extras: [], + declared_in: declaredIn, + direct: true, + provides_imports: providesOf(name), + prov: ["declared"], + }); + } + } + return out; +} + +/** + * `name → locked version` from a JSON-family lockfile. package-lock/npm-shrinkwrap: v2/v3 + * top-level `packages["node_modules/"].version` (nested entries are transitive shadows), + * v1 `dependencies{}` fallback. bun.lock (JSONC): `packages{ "": ["@", ...] }`. + */ +export function readLock(fileName: string, text: string): Record { + if (fileName === "bun.lock") return readBunLock(text); + let doc: unknown; + try { + doc = JSON.parse(text); + } catch { + return {}; + } + if (typeof doc !== "object" || doc === null) return {}; + const out: Record = {}; + const packages = (doc as Record)["packages"]; + if (typeof packages === "object" && packages !== null) { + for (const [key, entry] of Object.entries(packages as Record)) { + const m = /^node_modules\/((?:@[^/]+\/)?[^/]+)$/.exec(key); + if (!m) continue; + const version = (entry as Record | null)?.["version"]; + if (typeof version === "string") out[m[1] as string] = version; + } + if (Object.keys(out).length) return out; + } + const v1 = (doc as Record)["dependencies"]; + if (typeof v1 === "object" && v1 !== null) { + for (const [name, entry] of Object.entries(v1 as Record)) { + const version = (entry as Record | null)?.["version"]; + if (typeof version === "string") out[name] = version; + } + } + return out; +} + +function readBunLock(text: string): Record { + let doc: unknown; + try { + doc = JSON.parse(text.replace(/,\s*([}\]])/g, "$1")); // tolerate bun's trailing commas + } catch { + return {}; + } + const out: Record = {}; + const packages = (doc as Record | null)?.["packages"]; + if (typeof packages !== "object" || packages === null) return {}; + for (const [name, entry] of Object.entries(packages as Record)) { + const first = Array.isArray(entry) ? entry[0] : undefined; + if (typeof first !== "string") continue; + const at = first.lastIndexOf("@"); + if (at > 0) out[name] = first.slice(at + 1); + } + return out; +} + +/** Backfill locked_version on DECLARED records; their `prov` gains "lockfile". */ +export function applyLockVersions(deps: TSDependency[], lock: Record): void { + for (const dep of deps) { + const v = lock[dep.name]; + if (v === undefined) continue; + dep.locked_version = v; + if (!dep.prov.includes("lockfile")) dep.prov.push("lockfile"); + } +} + +/** + * Lock-only packages are TRANSITIVE: pinned with no manifest declaration. They earn records + * (`direct: false`) because the dependency SURFACE and the dependency SUPPLY CHAIN are different + * questions — a vulnerable package four levels down ships whether or not anyone named it. + * `kind` is "runtime": a lock does not record why a package is present, and inferring it would + * take a whole-graph walk this unit deliberately does not do. + */ +export function transitiveRecords( + pins: Record, + declaredNames: Set, + lockArtifactId: string, +): TSDependency[] { + const out: TSDependency[] = []; + for (const name of Object.keys(pins).sort()) { + if (declaredNames.has(name)) continue; + out.push({ + name, + spec: "", + kind: "runtime", + extras: [], + declared_in: lockArtifactId, + direct: false, + locked_version: pins[name] as string, + provides_imports: [name], + prov: ["lockfile"], + }); + } + return out; +} diff --git a/src/artifacts/index.ts b/src/artifacts/index.ts new file mode 100644 index 0000000..61be983 --- /dev/null +++ b/src/artifacts/index.ts @@ -0,0 +1,202 @@ +/** + * Repository-artifact layer (#101), parity with codeanalyzer-python PR #160 / the ratified + * 2026-08-27 spec: `inventoryArtifacts` walks the project once and returns the three + * application sections — `artifacts` (every RULES-matched non-code file, verbatim `source`, + * unbounded by decision), `dependencies` (flat, evidence-tagged: `direct:true` declared records + * from manifests, plus `direct:false` transitive records for lock-pinned packages no manifest + * names; locks backfill `locked_version` on declared records), and `unresolved_imports` (the + * hygiene signal). Level-free: attached + * identically at every `-a`. Not cached. Ids are stamped by assignIds (they embed `--app-name`). + * + * Discovery skips the source-walk's directory set; TS/JS source stays in the symbol table. + * All non-source files are captured: rules-matched files carry their designated roles, extensionless + * shebang files are `script` artifacts, and everything else is `unknown` (text or binary). + */ +import * as fs from "node:fs"; +import * as path from "node:path"; +import { DEFAULT_ARTIFACT_TEXT_MAX_BYTES } from "../options"; +import type { AnalysisOptions } from "../options"; +import type { TSArtifact, TSDependency, TSImportBinding, TSModule } from "../schema"; +import { sha256 } from "../utils"; +import { SKIP_DIRS } from "../syntactic_analysis/discovery"; +import { matchRules } from "./rules"; +import { applyLockVersions, parsePackageJson, readLock, transitiveRecords } from "./deps"; +import { bindImports } from "./binding"; +import { extractConfigKeys } from "./configKeys"; +import { deploymentEnvKeys } from "./deployEnv"; + +const SOURCE_EXTS = new Set([".ts", ".tsx", ".mts", ".cts", ".js", ".jsx", ".mjs", ".cjs"]); +const JSON_LOCKFILES = new Set(["package-lock.json", "npm-shrinkwrap.json", "bun.lock"]); + +export interface ArtifactLayer { + artifacts: Record; + dependencies: TSDependency[]; + unresolved_imports: TSImportBinding[]; +} + +export function inventoryArtifacts( + root: string, + opts: AnalysisOptions, + symbol_table: Record, +): ArtifactLayer { + const artifacts: Record = {}; + // Owning manifest's rel path → {name: locked version} (a lock pins its SIBLING package.json). + const locks: Record> = {}; + // Same key → the lock file's OWN rel path (the transitive records' `declared_in` attribution). + const lockPathOf: Record = {}; + const manifests: Array<{ rel: string; text: string }> = []; + + for (const rel of walk(root).sort()) { + const base = path.basename(rel); + const matched = matchRules(rel); + let format = matched?.format; + let roles = matched?.roles; + const abs = path.join(root, rel); + let raw: Buffer; + try { + raw = fs.readFileSync(abs); + } catch { + continue; // unreadable — skip, don't crash + } + if (!matched) { + // Never drop: a file without a rule is still inventoried. Extensionless shebang files are + // scripts; everything else decodable is `unknown`; undecodable bytes are hash-only. + const probe = decodeLossy(raw); + if (path.extname(base) === "" && raw.subarray(0, 2).toString("utf-8") === "#!") { + format = "text"; + roles = ["script"]; + } else if (probe === undefined) { + format = "binary"; + roles = ["unknown"]; + } else { + format = "text"; + roles = ["unknown"]; + } + } + const text = decodeLossy(raw); + const capture = opts.artifactText ?? true; + const cap = opts.artifactTextMaxBytes ?? DEFAULT_ARTIFACT_TEXT_MAX_BYTES; + const textByteLength = text === undefined ? 0 : Buffer.byteLength(text, "utf8"); + const stored = + !capture || text === undefined + ? "" + : textByteLength > cap + ? Buffer.from(text, "utf8").subarray(0, cap).toString("utf8") + : text; + const node: TSArtifact = { + id: "", + kind: "artifact", + path: rel, + format: format as string, + roles: roles as string[], + size_bytes: raw.length, + sha256: sha256(raw), + source: stored, + text_truncated: capture && text !== undefined && textByteLength > cap, + extraction: "none", + config_keys: [], + }; + artifacts[rel] = node; + if (text === undefined) continue; + if (base === "package.json") manifests.push({ rel, text }); + else if (JSON_LOCKFILES.has(base)) { + const ownerRel = rel.split("/").slice(0, -1).concat("package.json").join("/"); + locks[ownerRel] = readLock(base, text); + lockPathOf[ownerRel] = rel; + artifacts[rel].extraction = "full"; + } + // Config keys: attempted for every namespace-eligible format; a throw means the file is + // config-shaped but unparseable → keep the node, mark partial (overlay posture). + if (["env", "json", "jsonc", "ini", "properties", "yaml"].includes(node.format)) { + try { + const keys = extractConfigKeys(node.format, node.roles, text); + if (keys.length) { + node.config_keys = keys; + node.extraction = node.extraction === "none" ? "full" : node.extraction; + } + // Deployment-env keys (#101 unit D): additive on top of the structural keys above. + // Compose/k8s mint theirs from `keys` (the same yaml parse — no second one); a throw from + // extractConfigKeys above already skipped straight to the catch below, so a malformed + // document never reaches here either. + const deployKeys = deploymentEnvKeys(node.format, node.roles, text, keys); + if (deployKeys.length) { + node.config_keys = [...node.config_keys, ...deployKeys]; + node.extraction = "full"; + } + } catch { + node.extraction = "partial"; + } + } else if (node.format === "dockerfile") { + // Not namespace-eligible above (no structural extractConfigKeys case for it), but ENV/ARG + // still mint deploy keys; the line-based parse never throws, so no try/catch needed. + const deployKeys = deploymentEnvKeys(node.format, node.roles, text); + if (deployKeys.length) { + node.config_keys = deployKeys; + node.extraction = "full"; + } + } + } + if (opts.artifactText === false) { + for (const art of Object.values(artifacts)) { + for (const ck of art.config_keys) delete ck.value; + } + } + + // Declared records from every dependency-manifest package.json; sibling locks backfill. + const dependencies: TSDependency[] = []; + for (const { rel, text } of manifests) { + const recs = parsePackageJson(text, rel); // declared_in = REL PATH; assignIds re-stamps the id + const node = artifacts[rel]; + if (node) node.extraction = recs.length ? "full" : node.extraction; + const lock = locks[rel]; + if (lock) applyLockVersions(recs, lock); + dependencies.push(...recs); + } + + // Lock-only packages (pinned, never declared in any manifest): direct:false transitive records, + // attributed to the lock that pinned them (python PR #160 parity — supply chain, not just surface). + const declaredNames = new Set(dependencies.map((d) => d.name)); + for (const [ownerRel, pins] of Object.entries(locks).sort(([a], [b]) => a.localeCompare(b))) { + const lockRel = lockPathOf[ownerRel] as string; + const lockArtifact = artifacts[lockRel]; + if (!lockArtifact) continue; + dependencies.push(...transitiveRecords(pins, declaredNames, lockRel)); // rel path; assignIds re-stamps + } + + const unresolved_imports = bindImports(symbol_table, dependencies, root, opts.resolveInstalled ?? false); + return { artifacts, dependencies, unresolved_imports }; +} + +function walk(root: string): string[] { + const out: string[] = []; + const visit = (dir: string): void => { + let entries: fs.Dirent[]; + try { + entries = fs.readdirSync(dir, { withFileTypes: true }); + } catch { + return; + } + for (const e of entries) { + const abs = path.join(dir, e.name); + if (e.isDirectory()) { + if (SKIP_DIRS.has(e.name)) continue; // the guard is on containing DIRS — a `.env` file survives + visit(abs); + } else if (e.isFile()) { + if (SOURCE_EXTS.has(path.extname(e.name))) continue; // source lives in the symbol table + out.push(path.relative(root, abs).split(path.sep).join("/")); + } + } + }; + visit(root); + return out; +} + +/** utf-8 decode with a strict binary probe on the head; binary → undefined. */ +function decodeLossy(raw: Buffer): string | undefined { + try { + new TextDecoder("utf-8", { fatal: true }).decode(raw.subarray(0, Math.min(raw.length, 4096))); + } catch { + return undefined; + } + return new TextDecoder("utf-8", { fatal: false }).decode(raw); +} diff --git a/src/artifacts/rules.ts b/src/artifacts/rules.ts new file mode 100644 index 0000000..1e692d8 --- /dev/null +++ b/src/artifacts/rules.ts @@ -0,0 +1,94 @@ +/** + * The shipped discovery rules table (#101, python PR #160's mechanism): glob pattern against the + * repo-relative POSIX path → (format, roles). Rules decide `format` and `roles` for matched + * files; unmatched files are still inventoried by the walk in index.ts with `roles: ["unknown"]` + * or `format: "binary"` when undecodable; extensionless shebangs are captured as `script` artifacts. + */ + +export interface ArtifactRule { + pattern: RegExp; + format: string; + roles: string[]; +} + +/** + * Tiny glob→RegExp: `**` crosses directories, `*` does not. A pattern CONTAINING `/` anchors at + * the repo root; a bare basename pattern matches at any depth (workspace-member package.json, + * nested Dockerfiles). + */ +function glob(g: string): RegExp { + let re = ""; + for (let i = 0; i < g.length; i++) { + const c = g[i] as string; + if (c === "*") { + if (g[i + 1] === "*") { + re += ".*"; + i++; + if (g[i + 1] === "/") i++; // `**/` also matches zero directories + } else re += "[^/]*"; + } else if (".+^${}()|[]\\".includes(c)) re += `\\${c}`; + else re += c; + } + return g.includes("/") ? new RegExp(`^${re}$`) : new RegExp(`^(?:.*/)?${re}$`); +} + +const R = (g: string, format: string, roles: string[]): ArtifactRule => ({ pattern: glob(g), format, roles }); + +export const RULES: ArtifactRule[] = [ + // dependency manifests + locks (npm ecosystem) + R("package.json", "json", ["dependency-manifest", "tool-config"]), + R("package-lock.json", "json", ["dependency-manifest"]), + R("npm-shrinkwrap.json", "json", ["dependency-manifest"]), + R("bun.lock", "jsonc", ["dependency-manifest"]), + R("yarn.lock", "yarnlock", ["dependency-manifest"]), + R("pnpm-lock.yaml", "yaml", ["dependency-manifest"]), + // tool configs + R("tsconfig*.json", "json", ["tool-config"]), + R("jsconfig*.json", "json", ["tool-config"]), + R(".eslintrc*", "json", ["tool-config"]), + R(".prettierrc*", "json", ["tool-config"]), + R("babel.config.*", "text", ["tool-config"]), + R("vite.config.*", "text", ["tool-config"]), + R("webpack.config.*", "text", ["tool-config"]), + R("Makefile", "text", ["tool-config"]), + // containers / topology + R("Dockerfile", "dockerfile", ["container-image"]), + R("*.dockerfile", "dockerfile", ["container-image"]), + R("Dockerfile.*", "dockerfile", ["container-image"]), + R("docker-compose*.yml", "yaml", ["service-topology"]), + R("docker-compose*.yaml", "yaml", ["service-topology"]), + R("compose.yml", "yaml", ["service-topology"]), + R("compose.yaml", "yaml", ["service-topology"]), + R("k8s/**/*.yml", "yaml", ["service-topology"]), + R("k8s/**/*.yaml", "yaml", ["service-topology"]), + // ci + R(".github/workflows/*.yml", "yaml", ["ci"]), + R(".github/workflows/*.yaml", "yaml", ["ci"]), + R(".gitlab-ci.yml", "yaml", ["ci"]), + R("azure-pipelines.yml", "yaml", ["ci"]), + // env + R(".env", "env", ["env"]), + R(".env.*", "env", ["env"]), + // docs / legal + R("*.md", "text", ["docs"]), + R("*.rst", "text", ["docs"]), + R("LICENSE*", "text", ["legal"]), + R("COPYRIGHT*", "text", ["legal"]), + R("NOTICE*", "text", ["legal"]), + // config-shaped catch rows (python's `unknown` rows) + R("*.toml", "toml", ["unknown"]), + R("*.ini", "ini", ["unknown"]), + R("*.cfg", "ini", ["unknown"]), +]; + +/** First matching rule wins; roles union across ALL matching rules (a compose file is both). */ +export function matchRules(relPath: string): { format: string; roles: string[] } | null { + let format: string | null = null; + const roles: string[] = []; + for (const r of RULES) { + if (!r.pattern.test(relPath)) continue; + if (format === null) format = r.format; + for (const role of r.roles) if (!roles.includes(role)) roles.push(role); + } + return format === null ? null : { format, roles }; +} diff --git a/src/artifacts/yamlKeys.ts b/src/artifacts/yamlKeys.ts new file mode 100644 index 0000000..7d30b1c --- /dev/null +++ b/src/artifacts/yamlKeys.ts @@ -0,0 +1,104 @@ +/** + * YAML config-key flattening (#101 unit B). Uses the `yaml` package's document AST so spans are + * real offsets and anchors/flow style/multiline scalars parse correctly — a hand-rolled subset + * would silently mis-parse them. `parseAllDocuments` (not `parseDocument`) so a `---`-separated + * multi-document stream — the standard shape of a Kubernetes manifest — is covered instead of + * silently truncated to its first document: a single-document file keeps today's unprefixed key + * shape exactly, a multi-document file prefixes each document's keys with its zero-based index + * (`0.services.web.image`). One `LineCounter` spans the WHOLE parse (not one per document) since + * `Node.range` offsets are absolute into the full source text, not reset per document. Throws + * when any document has errors — the caller's existing catch records `extraction: "partial"` and + * keeps the artifact node (overlay posture): a parse failure is a different fact from "nothing to + * extract", which a silent [] can't distinguish. + * + * Aliases (`*name`) are resolved against their OWNING document before the isMap/isSeq/isScalar + * dispatch, so an aliased map/seq/scalar flattens exactly like an inline one instead of vanishing + * silently — the whole reason for taking this dependency instead of hand-rolling was that + * anchors/aliases parse correctly (fix round 2). A merge key (`<<: *defaults` or `<<: [*a, *b]`) + * splices its source map's entries in at the CURRENT prefix (no literal ".<<." segment) — with + * real YAML precedence (fix round 3): an explicit key always wins over a merged one regardless of + * document order, and among multiple merge sources (`<<: [*a, *b]`) an earlier source wins over a + * later one. `mapEntries` computes one map's own (key → winning value node) pairs — explicit keys + * first, in one pass, so they claim their name before any merge source is even inspected — then + * each merge source in turn, each one only filling names nobody has claimed yet, which is what + * makes the result order-independent (a `<<` written before or after the explicit key it loses to + * resolves the same either way). It recurses into itself for a merge source that has its own + * nested `<<`, capped by the same depth budget `walk` uses, so a cyclic merge (a map merging + * itself) terminates the same way a cyclic alias does. + */ +import { + LineCounter, + parseAllDocuments, + isMap, + isSeq, + isScalar, + isAlias, + type Node as YamlNode, + type YAMLMap, +} from "yaml"; +import { keyNode } from "./configKeys"; +import type { TSConfigKey, TSSpan } from "../schema"; + +export function parseYamlKeys(text: string): TSConfigKey[] { + const lc = new LineCounter(); + const docs = parseAllDocuments(text, { lineCounter: lc, keepSourceTokens: false }); + const bad = docs.find((d) => d.errors.length); + if (bad) throw bad.errors[0]; + const out: TSConfigKey[] = []; + const spanOf = (n: YamlNode): TSSpan | undefined => { + const r = n.range; + if (!r) return undefined; + const s = lc.linePos(r[0]); + const e = lc.linePos(r[1]); + return { start: [s.line, s.col], end: [e.line, e.col], bytes: [r[0], r[1]] }; + }; + const multi = docs.length > 1; + for (const [i, doc] of docs.entries()) { + // walk and mapEntries are defined per-document (not hoisted above the loop) so `doc` — needed + // to resolve this document's own aliases — is closed over correctly; parseAllDocuments may + // yield several. + const mapEntries = (map: YAMLMap, depth: number): Array<[string, unknown]> => { + const entries = new Map(); // insertion order IS document order (spec-guaranteed) + const mergeSources: unknown[] = []; + for (const item of map.items) { + if (isScalar(item.key) && item.key.value === "<<") { + const resolved = isAlias(item.value) ? item.value.resolve(doc) : item.value; + if (isSeq(resolved)) mergeSources.push(...resolved.items); + else mergeSources.push(item.value); + continue; + } + // explicit keys are collected in one pass, before any merge source is even resolved, so + // an explicit key claims its name regardless of where "<<" sits in this same items list. + entries.set(isScalar(item.key) ? String(item.key.value) : String(item.key), item.value); + } + if (depth <= 24) { + for (const src of mergeSources) { + const resolvedSrc = isAlias(src) ? src.resolve(doc) : src; + if (!isMap(resolvedSrc)) continue; + for (const [k, v] of mapEntries(resolvedSrc, depth + 1)) { + if (!entries.has(k)) entries.set(k, v); // explicit, or an earlier source, already won + } + } + } + return [...entries]; + }; + const walk = (node: unknown, prefix: string, depth: number): void => { + if (depth > 24) return; + const n = isAlias(node) ? node.resolve(doc) : node; + if (isMap(n)) { + for (const [k, v] of mapEntries(n, depth)) { + walk(v, prefix ? `${prefix}.${k}` : k, depth + 1); + } + } else if (isSeq(n)) { + n.items.forEach((item, idx) => walk(item, `${prefix}.${idx}`, depth + 1)); + } else if (isScalar(n)) { + const v = n.value; + if (typeof v === "string" || typeof v === "number" || typeof v === "boolean") { + out.push(keyNode(prefix, "yaml", v, spanOf(n as YamlNode))); + } + } + }; + walk(doc.contents, multi ? String(i) : "", 0); + } + return out; +} diff --git a/src/build/neo4j/project.ts b/src/build/neo4j/project.ts index 6c217f1..6e16c78 100644 --- a/src/build/neo4j/project.ts +++ b/src/build/neo4j/project.ts @@ -11,6 +11,7 @@ */ import type { TSAnalysis, TSApplication, TSBodyNode, TSCallable, TSField, TSModule, TSType } from "../../schema"; +import { purlNpm } from "../../schema/ids"; import { SCHEMA_VERSION } from "./schema"; import { type GraphRows, type NodeRef, type Props, RowBuilder, prune } from "./rows"; @@ -66,6 +67,70 @@ export function project(app: TSAnalysis, _appName?: string): GraphRows { projectScope(b, mod, modRef, fileKey); } + // Repository-artifact layer (#101, python PR #160 parity): language-NEUTRAL :Artifact and + // :Package (purl id) nodes — the deliberate exception to TS-prefixing, so sibling analyzers + // MERGE onto the same nodes — plus this analyzer's own claims (TS_PROVIDES / + // TS_UNRESOLVED_IMPORT) joining packages into the existing :TSExternal ghost id space. + // `source` text stays off the graph (hash + size dereference to it). + const importGhost = (name: string): NodeRef => + b.node([CAN, "TSExternal"], "id", `${root.id}/@external/${name}`, prune({ + id: `${root.id}/@external/${name}`, kind: "external", module: name, + })); + for (const art of Object.values(root.artifacts ?? {})) { + const aRef = b.node(["Artifact"], "id", art.id, prune({ + id: art.id, kind: "artifact", path: art.path, format: art.format, + roles: art.roles.length ? art.roles : null, size_bytes: art.size_bytes, + sha256: art.sha256, extraction: art.extraction, + })); + b.edge("HAS_ARTIFACT", appRef, aRef); + for (const ck of art.config_keys) { + const kRef = b.node(["ConfigKey"], "id", ck.id, prune({ + id: ck.id, key: ck.key, namespace: ck.namespace, + value: ck.value !== undefined ? String(ck.value) : null, + references: ck.references.length ? ck.references : null, + })); + b.edge("DEFINES_CONFIG", aRef, kRef); + } + } + { + const lockIds = Object.values(root.artifacts ?? {}) + .filter((a) => /(^|\/)(package-lock\.json|npm-shrinkwrap\.json|bun\.lock|yarn\.lock|pnpm-lock\.yaml)$/.test(a.path)) + .map((a) => a.id) + .sort(); + const seen = new Set(); + for (const d of root.dependencies ?? []) { + const pkgId = purlNpm(d.name); + const pkgRef = b.node(["Package"], "id", pkgId, prune({ id: pkgId, ecosystem: "npm", name: d.name })); + b.edge("DECLARES_DEPENDENCY", { label: "Artifact", keyProp: "id", value: d.declared_in }, pkgRef, prune({ + spec: d.spec || null, kind: d.kind, direct: d.direct, extras: d.extras.length ? d.extras : null, + prov: d.prov.length ? d.prov : null, + }), d.kind); + if (d.locked_version) { + for (const lockId of lockIds) { + const k = `LOCKS\0${lockId}\0${pkgId}`; + if (seen.has(k)) continue; + seen.add(k); + b.edge("LOCKS", { label: "Artifact", keyProp: "id", value: lockId }, pkgRef, prune({ version: d.locked_version })); + } + } + for (const top of d.provides_imports) { + const k = `PROV\0${pkgId}\0${top}`; + if (seen.has(k)) continue; + seen.add(k); + b.edge("TS_PROVIDES", pkgRef, importGhost(top)); + } + } + for (const u of root.unresolved_imports ?? []) { + b.edge("TS_UNRESOLVED_IMPORT", appRef, importGhost(u.module), prune({ prov: u.prov.length ? u.prov : null })); + } + } + // config_use literal/dataflow tier (#101 unit C2/C3): src is a body-node ordinal id already + // projected as a :CanNode above; dst is a :ConfigKey id, already projected in the artifact + // loop. config_reads stay JSON-only — they record absence, not an edge. + for (const u of root.config_uses ?? []) { + b.edge("TS_USES_CONFIG", ref(u.src), { label: "ConfigKey", keyProp: "id", value: u.dst }, prune({ prov: u.prov })); + } + // External library targets (shared nodes — no _module). for (const ext of Object.values(root.external_symbols ?? {})) { b.node([CAN, "TSExternal"], "id", ext.id, prune({ id: ext.id, kind: "external", name: ext.name, module: ext.module })); diff --git a/src/build/neo4j/schema.ts b/src/build/neo4j/schema.ts index acb7f1d..9f32c2a 100644 --- a/src/build/neo4j/schema.ts +++ b/src/build/neo4j/schema.ts @@ -64,6 +64,32 @@ export const NODE_LABELS: NodeLabel[] = [ analyzer_name: "string", analyzer_version: "string", }, }, + // Repository-artifact layer (#101, python PR #160 parity): language-NEUTRAL labels — the + // deliberate exception to TS-prefixing, so sibling analyzers MERGE onto the same + // :Artifact/:Package/:ConfigKey nodes. Edges that stay this analyzer's own claim keep the TS_ + // prefix. Additive within 2.1.0 — SCHEMA_VERSION moves only when every analyzer re-baselines + // together. + { + label: "Artifact", + mergeLabel: "Artifact", + key: "id", + properties: { + id: "string", kind: "string", path: "string", format: "string", roles: "string[]", + size_bytes: "integer", sha256: "string", extraction: "string", + }, + }, + { + label: "Package", + mergeLabel: "Package", + key: "id", + properties: { id: "string", ecosystem: "string", name: "string" }, + }, + { + label: "ConfigKey", + mergeLabel: "ConfigKey", + key: "id", + properties: { id: "string", key: "string", namespace: "string", value: "string", references: "string[]" }, + }, { label: "TSModule", mergeLabel: CAN, @@ -144,6 +170,19 @@ export const NODE_LABELS: NodeLabel[] = [ export const REL_TYPES: RelType[] = [ { type: "TS_HAS_MODULE", from: ["TSApplication"], to: ["TSModule"], properties: {} }, + // Repository-artifact layer (#101, python PR #160 vocabulary) + { type: "HAS_ARTIFACT", from: ["TSApplication"], to: ["Artifact"], properties: {} }, + { + type: "DECLARES_DEPENDENCY", + from: ["Artifact"], + to: ["Package"], + properties: { spec: "string", kind: "string", direct: "boolean", extras: "string[]", prov: "string[]" }, + }, + { type: "LOCKS", from: ["Artifact"], to: ["Package"], properties: { version: "string" } }, + { type: "TS_PROVIDES", from: ["Package"], to: ["TSExternal"], properties: {} }, + { type: "TS_UNRESOLVED_IMPORT", from: ["TSApplication"], to: ["TSExternal"], properties: { prov: "string[]" } }, + { type: "DEFINES_CONFIG", from: ["Artifact"], to: ["ConfigKey"], properties: {} }, + { type: "TS_USES_CONFIG", from: ["TSBodyNode"], to: ["ConfigKey"], properties: { prov: "string[]" } }, { type: "TS_DECLARES", from: ["TSModule", "TSNamespace", "TSCallable"], diff --git a/src/cli.ts b/src/cli.ts index c2bdb46..f0de69c 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -1,6 +1,7 @@ import * as path from "node:path"; import { Command, Option } from "commander"; -import type { AnalysisOptions, CallGraphProviderName, EmitTarget } from "./options"; +import type { AnalysisOptions, EmitTarget } from "./options"; +import { DEFAULT_ARTIFACT_TEXT_MAX_BYTES } from "./options"; import { ALL_GRAPHS, type GraphSelector } from "./schema"; /** @@ -57,12 +58,13 @@ export function buildProgram(): Command { .option("--lazy", "reuse the cache (default)") .option("--no-build", "skip dependency materialization (use a prepared node_modules)") .option("--no-phantoms", "disable phantom (external) nodes for imported/required library calls") + .option("--resolve-installed", "probe node_modules metadata for import→package binding (default: repo files only)") + .option("--no-artifact-text", "keep the artifact inventory but drop captured raw text") .option( - "--call-graph-provider ", - "call-graph backend: union (default, tsc ∪ jelly) | tsc | jelly | both (deprecated alias of union)", - "union", + "--artifact-text-max-bytes ", + "per-file byte cap for captured artifact text; larger files are truncated and flagged", + String(DEFAULT_ARTIFACT_TEXT_MAX_BYTES), ) - .option("--tsc-only", "use the tsc resolver only — opt out of Jelly edges (overrides --call-graph-provider)") .option("-c, --cache-dir ", "cache/intermediate directory") .option("-v, --verbose", "increase verbosity (repeatable)", (_v: string, prev: number) => prev + 1, 0) .allowExcessArguments(true); @@ -137,23 +139,6 @@ export function parseArgs(argv: string[]): AnalysisOptions { if (emit !== "schema" && !o.input) program.error("required option '-i, --input ' not specified"); const targets: string[] | null = Array.isArray(o.targetFiles) && o.targetFiles.length ? o.targetFiles.map(String) : null; - // --tsc-only is the forced opt-out: it wins over --call-graph-provider. Otherwise `both` is a - // deprecated alias of `union` (warn, but honor it); unknown values fall back to the union default. - let cgProvider: CallGraphProviderName; - if (o.tscOnly) { - cgProvider = "tsc"; - } else if (o.callGraphProvider === "tsc") { - cgProvider = "tsc"; - } else if (o.callGraphProvider === "jelly") { - cgProvider = "jelly"; - } else { - if (o.callGraphProvider === "both") { - // stderr only — stdout may carry compact JSON when -o is omitted. - console.error("warning: --call-graph-provider both is deprecated; it now behaves as 'union' (tsc ∪ jelly)."); - } - cgProvider = "union"; - } - return { input: o.input ? path.resolve(String(o.input)) : "", output: o.output ? path.resolve(String(o.output)) : null, @@ -173,7 +158,17 @@ export function parseArgs(argv: string[]): AnalysisOptions { // commander maps --no-build / --no-phantoms to opts.build/phantoms === false noBuild: o.build === false, phantoms: o.phantoms !== false, - callGraphProvider: cgProvider, + resolveInstalled: Boolean(o.resolveInstalled), + artifactText: o.artifactText !== false, + // Malformed input (e.g. "abc") must fall back, not silently disable truncation via NaN -- + // every `> cap` comparison against NaN is false. + artifactTextMaxBytes: (() => { + // An empty value is malformed too -- Number("") is 0, which would cap every + // artifact's text at zero bytes. An explicit 0 still means exactly that. + const raw = String(o.artifactTextMaxBytes ?? "").trim(); + const n = raw === "" ? NaN : Number(raw); + return Number.isFinite(n) && n >= 0 ? n : DEFAULT_ARTIFACT_TEXT_MAX_BYTES; + })(), cacheDir: o.cacheDir ? path.resolve(String(o.cacheDir)) : null, verbosity: typeof o.verbose === "number" ? o.verbose : 0, }; diff --git a/src/core.ts b/src/core.ts index 5355e86..48aec15 100644 --- a/src/core.ts +++ b/src/core.ts @@ -1,13 +1,15 @@ import * as path from "node:path"; import { buildProgramGraphs, startExtraction } from "./dataflow"; -import { mergeCallGraphs, selectProvider } from "./semantic_analysis"; +import { type LinkerResolutions, mergeCallGraphs, runDefuseLinker, tscProvider } from "./semantic_analysis"; import { loadCache, saveCache } from "./utils"; import { materialize } from "./build"; +import { inventoryArtifacts } from "./artifacts"; import type { AnalysisOptions } from "./options"; import type { AnalysisInternal } from "./schema"; import { type AnalysisResult, finalizeAnalysis } from "./schema/emit"; import { buildSymbolTable } from "./syntactic_analysis"; import { Logger } from "./utils"; +import { checkerFailures, resetCheckerFailures } from "./schema/checker"; export type { AnalysisResult } from "./schema/emit"; @@ -20,6 +22,7 @@ export type { AnalysisResult } from "./schema/emit"; export async function analyze(opts: AnalysisOptions): Promise { const log = new Logger(opts.verbosity); log.info(`analyzing ${opts.input} (level ${opts.analysisLevel})`); + resetCheckerFailures(); const cacheDir = opts.cacheDir ?? path.join(opts.input, ".codeanalyzer"); const mat = materialize(opts, log); @@ -42,40 +45,55 @@ export async function analyze(opts: AnalysisOptions): Promise { } const extraction = opts.analysisLevel >= 3 ? startExtraction(project, symbol_table, mat.tsConfigFilePath, opts, log) : null; - // Call graph via the selected provider (union of tsc+jelly by default; --tsc-only / jelly opt-in). - // Only worth running at level >= 2: finalizeAnalysis discards call_graph/external_symbols/ - // synthesized_callables at -a 1 (homeExternals/homeSynthesized in src/schema/emit.ts are - // gated to `level >= 2`), so running the solve — including the heavier Jelly leg — at -a 1 - // would compute a result that's thrown away. Levels 3/4 need the provider for callee - // resolution and are always >= 2, so this gate is safe. - // - // Run the provider PER PROGRAM (each with its own Project + its slice of callables via `only`), - // then merge the results the same way the union provider merges tsc∪jelly. Signature gating uses - // the full merged symbol_table (passed to every program), so a cross-program in-project call - // resolves. Single-program projects run the loop once — behavior is unchanged. - const provider = selectProvider(opts.callGraphProvider); - log.info(`call graph provider: ${provider.name}`); - let cg: ReturnType = { edges: [], external_symbols: {}, synthesized_callables: {} }; + // Call graph: the tsc resolver, per program (each with its own Project + its slice of callables + // via `only`), merged across programs. Only worth running at level >= 2: finalizeAnalysis + // discards call_graph/external_symbols/synthesized_callables at -a 1 (homeExternals/ + // homeSynthesized in src/schema/emit.ts are gated to `level >= 2`), so running the solve at + // -a 1 would compute a result that's thrown away. Levels 3/4 need it for callee resolution and + // are always >= 2, so this gate is safe. Signature gating uses the full merged symbol_table + // (passed to every program), so a cross-program in-project call resolves. + let cg: ReturnType = { edges: [], external_symbols: {}, synthesized_callables: {} }; + const resolutions: LinkerResolutions = new Map(); if (opts.analysisLevel >= 2) { for (const prog of programs) { - const pcg = provider.build({ + const ctx = { project: prog.project, symbol_table, root: opts.input, log, phantoms: opts.phantoms, only: prog.fileKeys, - }); - cg = mergeCallGraphs(cg, pcg); + }; + cg = mergeCallGraphs(cg, tscProvider.build(ctx)); + // The defuse linker overlays the tsc base: it reads the callee_signature backfill the tsc + // leg just wrote, resolves what remains (tiers T1–T5, defuseLinker.ts), and returns its + // body-node resolutions out-of-band (never persisted — cache provenance rule). + const linked = runDefuseLinker(ctx); + cg = mergeCallGraphs(cg, linked.result); + for (const [caller, m] of linked.resolutions) { + const ex = resolutions.get(caller); + if (!ex) resolutions.set(caller, m); + else for (const [k, v] of m) if (!ex.has(k)) ex.set(k, v); + } } } const call_graph = cg.edges; + // Repository-artifact layer (#101, python PR #160 parity): level-free, identical at every -a. + const layer = inventoryArtifacts(opts.input, opts, symbol_table); + log.info( + `artifacts: ${Object.keys(layer.artifacts).length} files, ${layer.dependencies.length} dependency records, ` + + `${layer.unresolved_imports.length} unresolved imports`, + ); + const app: AnalysisInternal = { symbol_table, call_graph, external_symbols: cg.external_symbols, synthesized_callables: cg.synthesized_callables, + artifacts: layer.artifacts, + dependencies: layer.dependencies, + unresolved_imports: layer.unresolved_imports, }; // Level 3 join: stages 5–7 (summary wavefront + SDG) consume the extraction AND the @@ -85,5 +103,9 @@ export async function analyze(opts: AnalysisOptions): Promise { // Cache the id-free base (ids/body/heritage are per-run layers stamped by finalizeAnalysis; // the cached tree must stay --app-name-free). saveCache(cacheDir, { symbol_table }); - return finalizeAnalysis(app, pg, opts); + // Never let "some edges are missing" look like "there were no edges": a node the checker could + // not resolve is skipped (see schema/checker.ts), and the count is said out loud. + const skipped = checkerFailures(); + if (skipped) log.warn(`${skipped} symbol resolution(s) skipped — the TypeScript checker could not resolve them; affected call edges are absent`); + return finalizeAnalysis(app, pg, opts, resolutions, project); } diff --git a/src/dataflow/attach.ts b/src/dataflow/attach.ts index ceea6a9..11f59b0 100644 --- a/src/dataflow/attach.ts +++ b/src/dataflow/attach.ts @@ -114,7 +114,7 @@ function emitL3(li: LocalIds, nodes: GraphNode[], cfgEdges: CfgEdge[] | undefine // prov = the def-use METHOD: `solveDefUse` computes forward may-reaching-definitions over // k-limited access paths with a flow-insensitive copy/field-alias substrate (defuse.ts). It // is NOT SSA and NOT points-to-oracle-backed — so we tag it "reaching-defs", not "ssa". - // A real points-to layer (Jelly, PR F) would emit additional edges tagged "points-to". + // A real points-to layer (PR F) would emit additional edges tagged "points-to". // "reaching-defs" is a SANCTIONED ADDITIVE prov token — a deliberate, documented deviation // from the shared cross-analyzer vocabulary's canonical "ssa" tag for the L3 syntactic DDG. // Recorded in `.claude/SCHEMA_DECISIONS.md` (issue #32); JSON, Neo4j (`ddg.prov: string[]`, diff --git a/src/dataflow/configUse.ts b/src/dataflow/configUse.ts new file mode 100644 index 0000000..8308590 --- /dev/null +++ b/src/dataflow/configUse.ts @@ -0,0 +1,202 @@ +/** + * config_use dataflow tiers (#101 unit C3). Widens the literal tier (src/semantic_analysis/ + * configUse.ts) with AST symbol resolution PLUS a reassignment check — deliberately NOT the + * def-use substrate (defuse.ts): its reaching-definitions are keyed by k-limited access paths and + * carry no string-literal VALUES, so closing a literal through them is machinery beyond this + * unit. Resolving only an identifier with exactly one string-literal initializer that is never + * reassigned is strictly MORE conservative than reaching-definitions — it under-approximates, so + * it can never emit a wrong edge. + * + * - INTRA (-a 3): the read's key expression is a local identifier bound to exactly one string + * literal initializer, never reassigned. + * - INTERPROC (-a 4): the key is a parameter, and every resolved internal call site passes the + * same string literal at that position (one call boundary, no fixpoint). + * + * Superset-monotonic: only ADDS `config_uses`, only REMOVES the corresponding `config_reads` — + * reads that stay unresolved are left byte-identical, not re-tagged. + * + * Imports the schema/configUseRules LEAF files directly (never the `../schema` barrel, which + * re-exports `emit.ts` — the module that imports this one): going through the barrel here would + * close an import cycle back on emit.ts (mirrors semantic_analysis/configUse.ts's own rule). + */ +import { Node, SyntaxKind, type ParameterDeclaration, type Project, type VariableDeclaration } from "ts-morph"; +import type { AnalysisInternal, TSCallable, TSConfigRead } from "../schema/schema"; +import { forEachCallable } from "../schema/schema"; +import { symbolAt } from "../schema/checker"; +import { type ConfigUseSets, keyIndex } from "../semantic_analysis/configUse"; +import { ACCESS_RULES } from "../semantic_analysis/configUseRules"; + +/** + * `project` gives the AST the tiers read. A no-op below `-a 3` (the literal tier alone stands). + * Deterministic: `uses` stays sorted. + */ +export function widenConfigUses(app: AnalysisInternal, project: Project, literal: ConfigUseSets, level: number): ConfigUseSets { + if (level < 3) return literal; + const idx = keyIndex(app); + const rootNamespaces = new Map(ACCESS_RULES.map((r) => [r.root, r.namespaces])); + const uses = [...literal.uses]; + const resolvedSites = new Set(); + + for (const read of literal.reads) { + if (read.reason !== "non-literal") continue; + const key = resolveKeyThroughDataflow(read, project, app, level); + if (key === null) continue; + const namespaces = rootNamespaces.get(read.callee) ?? ["env"]; + const dsts = [...new Set(namespaces.flatMap((ns) => idx.get(`${ns} ${key}`) ?? []))].sort(); + if (!dsts.length) continue; // resolved a literal, but no declared key names it — leave the read standing + for (const dst of dsts) uses.push({ src: read.site, dst, prov: ["dataflow"] }); + resolvedSites.add(read.site); + } + + const reads = literal.reads.filter((r) => !resolvedSites.has(r.site)); + uses.sort((a, b) => a.src.localeCompare(b.src) || a.dst.localeCompare(b.dst)); + return { uses, reads }; +} + +/** + * The literal a read's key expression closes on, or null. Only ElementAccessExpression reads + * (`process.env[expr]`) are in scope: a PropertyAccessExpression key is always static already + * (the literal tier resolved or rejected it), and a CALL-rule read's node is a call expression, + * which fails the type check below and is left unwidened — out of scope for this unit. + */ +function resolveKeyThroughDataflow(read: TSConfigRead, project: Project, app: AnalysisInternal, level: number): string | null { + const node = accessNodeFor(read.site, project, app); + if (!node || !Node.isElementAccessExpression(node)) return null; + const arg = node.getArgumentExpression(); + if (!arg || !Node.isIdentifier(arg)) return null; + const decl = symbolAt(arg)?.getDeclarations()?.[0]; + if (!decl) return null; + // INTRA: `const/let/var key = "LITERAL"` (or a non-interpolated template literal), never reassigned. + if (Node.isVariableDeclaration(decl)) { + const init = decl.getInitializer(); + const value = literalTextOf(init); + return value !== undefined && !isReassigned(decl) ? value : null; + } + // INTERPROC (-a 4): a parameter whose every resolved caller passes one identical literal. + if (level >= 4 && Node.isParameterDeclaration(decl)) return uniqueLiteralArgument(decl, app, project); + return null; +} + +/** + * The value of a string literal OR a non-interpolated template literal, or undefined — mirrors + * the literal tier's own acceptance (`literalArgumentAt` in semantic_analysis/configUse.ts) so + * the same expression shape resolves the same way at every tier. An INTERPOLATED template + * (`` `PAYMENT_${x}` ``) parses as a distinct `TemplateExpression` node kind, never this one, so + * it is excluded for free — no separate `${`-substring guard needed here. + */ +function literalTextOf(node: Node | undefined): string | undefined { + if (node && (Node.isStringLiteral(node) || Node.isNoSubstitutionTemplateLiteral(node))) return node.getLiteralValue(); + return undefined; +} + +/** Every callable in `app`, one flat list — config reads are rare, so re-walking per read costs nothing. */ +function allCallables(app: AnalysisInternal): TSCallable[] { + const out: TSCallable[] = []; + for (const mod of Object.values(app.symbol_table)) forEachCallable(mod, (c) => out.push(c)); + return out; +} + +function findCallable(app: AnalysisInternal, pred: (c: TSCallable) => boolean): TSCallable | undefined { + return allCallables(app).find(pred); +} + +/** The AST node whose exact byte span is [start, end) in `absPath`, or undefined. */ +function nodeAtSpan(project: Project, absPath: string, start: number, end: number): Node | undefined { + let n = project.getSourceFile(absPath)?.getDescendantAtPos(start); + while (n && (n.getStart() !== start || n.getEnd() !== end)) n = n.getParent(); + return n; +} + +/** + * A read's `site` (`@`, possibly `/2`-suffixed) back to the AST node + * at that body node's recorded span. Mirrors callGraph.ts's `indexCallExpressions` precedent + * (span-keyed AST lookup), keyed here by the byte offsets already recorded on the body node + * instead of a rebuilt line/col index — config_access spans are captured off ONE AST node + * (buildConfigAccess), so the exact-span walk-up below never has to disambiguate a collision. + */ +function accessNodeFor(site: string, project: Project, app: AnalysisInternal): Node | undefined { + const at = site.lastIndexOf("@"); + if (at < 0) return undefined; + const c = findCallable(app, (x) => x.id === site.slice(0, at)); + const bytes = c?.body[site.slice(at + 1)]?.span?.bytes; + return c && bytes ? nodeAtSpan(project, c.abs_path, bytes[0], bytes[1]) : undefined; +} + +function isAssignmentOperator(k: SyntaxKind): boolean { + return k >= SyntaxKind.FirstAssignment && k <= SyntaxKind.LastAssignment; +} + +/** + * Any assignment whose left side CONTAINS an identifier resolving to the same declaration — not + * merely IS one. Fix round 1: the original identity check (`left === id`) missed destructuring + * reassignment (`({ key } = obj)`, `[key] = arr`), where the tracked identifier is a binding + * target nested inside the left side, not the whole of it. A local binding can only be + * referenced within its own lexical scope, so scanning the whole file's assignments covers every + * possible reference without a separate closure-boundary walk. + * + * Deliberately over-inclusive: `obj[key] = value` also counts (the left side's subtree contains + * `key`, even though `key` is only a computed index there, not itself rebound). That costs a + * missed edge, never a wrong one — the same posture as every other check in this tier. + */ +function isReassigned(decl: VariableDeclaration): boolean { + const nameNode = decl.getNameNode(); + if (!Node.isIdentifier(nameNode)) return true; // destructuring binding — conservative, never widen + const symbol = symbolAt(nameNode); + // The checker could not resolve the binding: stay conservative rather than let an undefined + // symbol compare equal to another unresolved one below and fake a match. + if (!symbol) return true; + // A shorthand `{ key }` binds `key` via a SEPARATE symbol (one per ShorthandPropertyAssignment + // declaration slot) — plain `.getSymbol()` on its identifier never equals `symbol`, even though + // it IS the same reassigned variable. `getShorthandAssignmentValueSymbol` is the checker's own + // resolver for exactly this indirection (confirmed empirically: `.getSymbol()` alone misses it). + const checker = decl.getProject().getTypeChecker(); + const targetsSymbol = (left: Node): boolean => { + if (Node.isIdentifier(left)) return symbolAt(left) === symbol; + return left.getDescendantsOfKind(SyntaxKind.Identifier).some((id) => { + if (symbolAt(id) === symbol) return true; + const p = id.getParent(); + return Node.isShorthandPropertyAssignment(p) && checker.getShorthandAssignmentValueSymbol(p) === symbol; + }); + }; + return decl + .getSourceFile() + .getDescendantsOfKind(SyntaxKind.BinaryExpression) + .some((bin) => isAssignmentOperator(bin.getOperatorToken().getKind()) && targetsSymbol(bin.getLeft())); +} + +/** + * A parameter's enclosing callable, matched on file + start offset the same way builders.ts + * picks a signature node: the function-like node itself, unless it's an arrow/function-expression + * bound directly to a `const`/`let` — then the VariableDeclaration IS the signature node, and + * TSCallable.span follows it, not the function keyword (buildCallable's sigNode/fnNode split). + */ +function uniqueLiteralArgument(decl: ParameterDeclaration, app: AnalysisInternal, project: Project): string | null { + const fn = decl.getParent(); + if (!fn) return null; + const params = (fn as unknown as { getParameters?: () => Node[] }).getParameters?.() ?? []; + const paramIndex = params.findIndex((p) => p === decl); + if (paramIndex < 0) return null; + const fnParent = fn.getParent(); + const sigStart = fnParent && Node.isVariableDeclaration(fnParent) && fnParent.getInitializer() === fn ? fnParent.getStart() : fn.getStart(); + const absPath = decl.getSourceFile().getFilePath(); + const callable = findCallable(app, (c) => c.abs_path === absPath && c.span.bytes[0] === sigStart); + if (!callable) return null; + + // Matched on the BODY node's resolved `callee` id (backfillCallees), not call_sites' + // callee_signature: the id also carries defuse-linker resolutions, which callee_signature + // deliberately never persists (cache provenance rule, l2Callees.ts) — a stronger signal for + // the same cost, so fewer resolvable calls are missed. + let literal: string | undefined; + for (const caller of allCallables(app)) { + for (const node of Object.values(caller.body)) { + if (node.kind !== "call" || node.callee !== callable.id || !node.span) continue; + const callNode = nodeAtSpan(project, caller.abs_path, node.span.bytes[0], node.span.bytes[1]); + const argExpr = callNode && Node.isCallExpression(callNode) ? callNode.getArguments()[paramIndex] : undefined; + const value = literalTextOf(argExpr); + if (value === undefined) return null; // missing/dynamic arg breaks agreement + if (literal === undefined) literal = value; + else if (literal !== value) return null; // callers disagree + } + } + return literal ?? null; // no resolved caller — nothing to widen +} diff --git a/src/dataflow/defuse.ts b/src/dataflow/defuse.ts index 2fc4a84..5dd18f2 100644 --- a/src/dataflow/defuse.ts +++ b/src/dataflow/defuse.ts @@ -18,7 +18,7 @@ * * Aliasing (MVP substrate, per issue #2 / SCHEMA_DECISIONS.md): flow-insensitive union-find over * bases connected by direct copies (`const q = p`); a write through one name weakly updates the - * other. Points-to-backed aliasing via Jelly's solved state is the staged upgrade (PR F). + * other. Points-to-backed aliasing is the staged upgrade (PR F). * * Def-use: classic forward may reaching-definitions. Strong (killing) defs are whole-base writes * to locals/params; every field write is weak. Captured/module/this bases get a synthetic def at diff --git a/src/dataflow/worker.ts b/src/dataflow/worker.ts index 90d377a..20a80c8 100644 --- a/src/dataflow/worker.ts +++ b/src/dataflow/worker.ts @@ -13,7 +13,7 @@ */ import { Project } from "ts-morph"; import type { PdgEdge } from "../schema"; -import { defaultCompilerOptions, discoverSourceFiles } from "../syntactic_analysis"; +import { createProject, discoverSourceFiles } from "../syntactic_analysis"; import { extractCallableData, indexCallableDecls } from "./extract"; import type { CallableGraphData } from "./model"; import { sccFixpoint, type CallSiteRef, type FunctionSummary } from "./summaries"; @@ -49,9 +49,7 @@ function projectFor(root: string, tsConfigFilePath: string | null, skipTests: bo const key = `${root}|${tsConfigFilePath ?? ""}|${skipTests}`; let project = projects.get(key); if (project) return project; - project = tsConfigFilePath - ? new Project({ tsConfigFilePath, skipAddingFilesFromTsConfig: true }) - : new Project({ compilerOptions: defaultCompilerOptions() }); + project = createProject(tsConfigFilePath); for (const f of discoverSourceFiles(root, skipTests)) { try { project.addSourceFileAtPath(f.absPath); diff --git a/src/main.ts b/src/main.ts index 40dd5c2..3a82795 100644 --- a/src/main.ts +++ b/src/main.ts @@ -1,26 +1,3 @@ #!/usr/bin/env node -/** - * Multi-call binary entry. The compiled `cants` executable bundles BOTH the analyzer and the - * `@cs-au-dk/jelly` CLI; this dispatcher picks which one runs based on argv: - * - * cants __jelly -> run the embedded Jelly CLI (used internally by jellyProvider) - * cants -> run the normal analyzer - * - * Both programs self-execute on import (analyzer's main() / Jelly's program.parse()), so dispatch is - * "reshape argv, then dynamically import the right module". Bun's --compile bundles both branches. - * The CANTS_SELF_JELLY marker tells jellyProvider it can re-exec THIS binary for Jelly instead of - * shelling out to `node`; it is intentionally unset in source/dev runs (where the dispatcher is - * bypassed and the provider falls back to `node @cs-au-dk/jelly/lib/main.js`). - */ -export {}; // mark as a module so top-level await is permitted - -const argv = process.argv; -if (argv[2] === "__jelly") { - // Jelly's commander reads process.argv as [node, script, ...args]; drop our "__jelly" sentinel. - process.argv = [argv[0], "jelly", ...argv.slice(3)]; - // @ts-ignore — @cs-au-dk/jelly ships no type declarations for the lib subpath - await import("@cs-au-dk/jelly/lib/main.js"); -} else { - process.env.CANTS_SELF_JELLY = process.execPath; - await import("./index"); -} +/** Binary entry — the analyzer CLI (self-executes on import). */ +import "./index"; diff --git a/src/options/options.ts b/src/options/options.ts index 80d10d8..7829245 100644 --- a/src/options/options.ts +++ b/src/options/options.ts @@ -1,7 +1,8 @@ import type { GraphSelector } from "../schema"; export type EmitTarget = "json" | "neo4j" | "schema"; -export type CallGraphProviderName = "union" | "tsc" | "jelly"; +/** Default per-file byte cap for captured artifact text (256 KiB, python v1.3.0 parity). */ +export const DEFAULT_ARTIFACT_TEXT_MAX_BYTES = 256 * 1024; /** Normalized analysis options (produced by the CLI layer, consumed by core). */ export interface AnalysisOptions { @@ -46,9 +47,13 @@ export interface AnalysisOptions { noBuild: boolean; /** Emit phantom (external) nodes/edges for imported/required library call targets. Default on. */ phantoms: boolean; - /** Call-graph backend: union of tsc+jelly (default), tsc resolver only (--tsc-only), or jelly. */ - callGraphProvider: CallGraphProviderName; /** Where caches/intermediate state live; null ⇒ /.codeanalyzer. */ + /** Opt-in: probe node_modules metadata for import→package binding (prov "installed-metadata"). */ + resolveInstalled?: boolean; + /** Capture verbatim artifact text into `source` (default true). */ + artifactText?: boolean; + /** Per-file byte cap for captured text; larger files store a flagged prefix. */ + artifactTextMaxBytes?: number; cacheDir: string | null; /** Verbosity (repeatable -v). */ verbosity: number; diff --git a/src/schema/assignIds.ts b/src/schema/assignIds.ts index e493c11..1abe2b8 100644 --- a/src/schema/assignIds.ts +++ b/src/schema/assignIds.ts @@ -8,7 +8,7 @@ * id-uniqueness gate's collision list. */ -import { applicationIdOf, idFromSig, memberKey, moduleIdOf, modulePrefixOf } from "./ids"; +import { applicationIdOf, artifactIdOf, configKeyIdOf, idFromSig, memberKey, moduleIdOf, modulePrefixOf } from "./ids"; import type { AnalysisInternal, TSCallable, TSField, TSType } from "./schema"; export interface AssignedIds { @@ -54,10 +54,39 @@ export function assignIds(app: AnalysisInternal, appName: string): AssignedIds { const moduleId = moduleIdOf(appId, fileKey); const modulePrefix = modulePrefixOf(fileKey); mod.id = moduleId; + // Module-scope execution is a call-graph SOURCE (python #131 parity: a call in module scope + // is attributed to the MODULE). The prefix is the module's "signature", so those edges + // re-identify onto the module node's id instead of dangling. + register(modulePrefix, moduleId); doFields(moduleId, mod.fields); for (const fn of Object.values(mod.functions ?? {})) doCallable(moduleId, modulePrefix, fn); for (const t of Object.values(mod.types ?? {})) doType(moduleId, modulePrefix, t); } + // Repository-artifact layer: same per-run rule (ids embed --app-name). Artifact ids are + // language-NEUTRAL (`can://artifact/...`); dependency/import records are flat evidence rows + // with no node id of their own (the graph's :Package node is purl-keyed). + for (const [relPath, art] of Object.entries(app.artifacts ?? {})) { + art.id = artifactIdOf(appName, relPath); + // Deployment-env id disambiguation (#101 unit D fix round 1, python v1.3.0 parity verbatim): + // the `key` FIELD always stays the bare variable name — env-namespace resolution still joins + // on a plain `key ===` match — but a bare name can collide across mints on the SAME artifact + // (Dockerfile: `ARG VERSION` + `ENV VERSION=$VERSION`; yaml: a top-level `PAYMENT_HOST:` leaf + // vs. the `env` dual-mint from `services.web.environment.PAYMENT_HOST`), so only the ID gets + // an internal prefix: `arg.` for a dockerfile-namespace (ARG) mint, `env.` for a yaml + // artifact's env dual-mint. Dockerfile's own ENV mint and every ordinary structural key stay + // unprefixed. + for (const ck of art.config_keys) { + let idKey = ck.key; + if (ck.namespace === "dockerfile") idKey = `arg.${ck.key}`; + else if (art.format === "yaml" && ck.namespace === "env") idKey = `env.${ck.key}`; + ck.id = configKeyIdOf(art.id, idKey); + } + } + for (const dep of app.dependencies ?? []) { + const artPath = dep.declared_in; // scanners record the REL PATH; re-stamp onto the id + dep.declared_in = artifactIdOf(appName, artPath.startsWith("can://") ? artPath.split("/").slice(4).join("/") : artPath); + } + return { appId, idBySig, callableBySig, collisions }; } diff --git a/src/schema/checker.ts b/src/schema/checker.ts new file mode 100644 index 0000000..9689750 --- /dev/null +++ b/src/schema/checker.ts @@ -0,0 +1,51 @@ +/** + * Guarded access to tsc's checker. + * + * ts-morph's symbol queries run the TypeScript checker, and the checker THROWS on some nodes it + * cannot resolve instead of returning undefined. The reproducible case: a `.js` file that no + * tsconfig `include` covers is still discovered as source, so it lands in a program with no + * default lib — and resolving an ordinary global there (`throw new Error(...)`) dies inside + * `getSymbolOfDeclaration`. Observed on vscode, where one such mock + * (`extensions/microsoft-authentication/packageMocks/dpapi/dpapi.js`) aborted the entire + * 9,351-module analysis at every level above 1. + * + * Every caller already has an unresolved path — a call site whose callee will not resolve simply + * contributes no edge — so degrading a throw to `undefined` costs the edges at that one node and + * nothing else. Unguarded, it costs the whole run. + * + * The failures are counted, not swallowed silently: `analyze()` reports the total, because + * "some edges are missing" must never be indistinguishable from "there were no edges". + */ +import type { Node, Symbol as TsSymbol } from "ts-morph"; + +let failures = 0; + +/** `node.getSymbol()`, returning undefined where the checker throws. */ +export function symbolAt(node: Node): TsSymbol | undefined { + try { + return node.getSymbol(); + } catch { + failures++; + return undefined; + } +} + +/** `symbol.getAliasedSymbol()`, returning undefined where the checker throws. */ +export function aliasedSymbolOf(symbol: TsSymbol): TsSymbol | undefined { + try { + return symbol.getAliasedSymbol(); + } catch { + failures++; + return undefined; + } +} + +/** How many checker calls have thrown since the last reset. */ +export function checkerFailures(): number { + return failures; +} + +/** Per-run reset — the count is reported per analysis, and one process may run several. */ +export function resetCheckerFailures(): void { + failures = 0; +} diff --git a/src/schema/emit.ts b/src/schema/emit.ts index 4ba933e..6a38883 100644 --- a/src/schema/emit.ts +++ b/src/schema/emit.ts @@ -5,17 +5,19 @@ * construct the wire shapes directly (src/syntactic_analysis/builders.ts): * * assignIds — can:// ids (per-run: ids embed --app-name; the cache stays id-free) - * populateL1Body — call_sites → body{} `call` nodes, callee: null + * populateL1Body — call_sites/config_accesses → body{} `call`/`config_access` nodes, callee: null * resolveHeritage — extends_ids / implements_ids (resolved-only) * [L2] homeExternals / homeSynthesized / backfillCallees / reidentifyCallGraph * [L3/4] applyDataflow — program_graphs → body{} + cfg/cdg/ddg/summary + param_in/param_out * * The returned application is a DEEP, INTERNAL-FIELD-STRIPPED copy: the live tree keeps - * `call_sites`, `abs_path`, and the cache metadata for the resolver/dataflow/cache, while every - * consumer of the emission (JSON writer, Neo4j projection, tests) sees exactly the wire. + * `call_sites`, `config_accesses`, `abs_path`, and the cache metadata for the resolver/dataflow/ + * cache, while every consumer of the emission (JSON writer, Neo4j projection, tests) sees exactly + * the wire. */ import * as path from "node:path"; +import type { Project } from "ts-morph"; import type { AnalysisOptions } from "../options"; import { ANALYZER_VERSION } from "../utils/version"; import type { AnalysisInternal, TSAnalysis, TSApplication } from "./schema"; @@ -26,6 +28,8 @@ import { resolveHeritageIds } from "./heritage"; import { homeExternals, homeSynthesized } from "./homing"; import { backfillCallees, reidentifyCallGraph } from "./l2Callees"; import { applyDataflow } from "../dataflow/attach"; +import { resolveLiteralConfigUses } from "../semantic_analysis/configUse"; +import { widenConfigUses } from "../dataflow/configUse"; const LANGUAGE = "typescript"; const SCHEMA_VERSION = "2.1.0"; @@ -33,8 +37,34 @@ const ANALYZER_NAME = "codeanalyzer-typescript"; /** Highest analysis level this emitter populates today (L1 tree, L2 call graph, L3/L4 dataflow). */ const MAX_IMPLEMENTED = 4; -/** INTERNAL model fields — never on the wire (see schema.ts header). */ -const INTERNAL_KEYS = new Set(["call_sites", "abs_path", "content_hash", "last_modified", "file_size"]); +/** + * Structural internal-field strip on the WIRE CLONE: module cache trio + callable join fields. + * Structural (walks the tree shape) rather than key-name-based, for two load-bearing reasons: + * the artifact layer's `content_hash` is WIRE payload (a name-keyed replacer would eat it), and + * a `JSON.stringify` deep-copy roundtrip builds one multi-GB string at vscode-L4 scale and OOMs + * (measured). `structuredClone` + targeted deletes never materializes a string. + */ +function stripInternal(root: TSApplication): void { + const stripCallable = (c: Record): void => { + delete c["call_sites"]; + delete c["config_accesses"]; + delete c["abs_path"]; + for (const nested of Object.values((c["callables"] as Record>) ?? {})) stripCallable(nested); + for (const t of Object.values((c["types"] as Record>) ?? {})) stripType(t); + }; + const stripType = (t: Record): void => { + for (const m of Object.values((t["callables"] as Record>) ?? {})) stripCallable(m); + for (const f of Object.values((t["functions"] as Record>) ?? {})) stripCallable(f); + for (const nt of Object.values((t["types"] as Record>) ?? {})) stripType(nt); + }; + for (const mod of Object.values(root.symbol_table) as unknown as Record[]) { + delete mod["content_hash"]; + delete mod["last_modified"]; + delete mod["file_size"]; + for (const fn of Object.values((mod["functions"] as Record>) ?? {})) stripCallable(fn); + for (const t of Object.values((mod["types"] as Record>) ?? {})) stripType(t); + } +} // ---------------------------------------------------------------------------------------------- // entry point @@ -49,7 +79,13 @@ export interface AnalysisResult { dangling: string[]; // call-graph endpoints with no id home (L2 no-dangling gate; should be empty) } -export function finalizeAnalysis(app: AnalysisInternal, pg: ProgramGraphs | null, opts: AnalysisOptions): AnalysisResult { +export function finalizeAnalysis( + app: AnalysisInternal, + pg: ProgramGraphs | null, + opts: AnalysisOptions, + resolutions?: Map>, + project?: Project, +): AnalysisResult { const level = opts.analysisLevel; const appName = (opts.appName ?? (opts.input ? path.basename(opts.input) : "") ?? "").trim() || "app"; @@ -58,17 +94,44 @@ export function finalizeAnalysis(app: AnalysisInternal, pg: ProgramGraphs | null populateL1Body(app); resolveHeritageIds(app, idBySig); - const root: TSApplication = { id: appId, kind: "application", symbol_table: app.symbol_table, call_graph: [], param_in: [], param_out: [] }; + const root: TSApplication = { + id: appId, + kind: "application", + symbol_table: app.symbol_table, + call_graph: [], + param_in: [], + param_out: [], + artifacts: app.artifacts ?? {}, + dependencies: app.dependencies ?? [], + unresolved_imports: app.unresolved_imports ?? [], + config_uses: app.config_uses ?? [], + config_reads: app.config_reads ?? [], + }; // L2 — home the off-tree edge endpoints, backfill `callee`, re-identify the call graph. const dangling: string[] = []; if (level >= 2) { root.external_symbols = homeExternals(app, appId, idBySig); root.synthesized_callables = homeSynthesized(app, appId, idBySig); - backfillCallees(app, idBySig); + backfillCallees(app, idBySig, resolutions); + // config_use literal tier (#101): needs the artifact layer's keys and, for CALL rules, the + // resolved call graph (`callee` ids only exist after backfillCallees) — so it runs here, not + // in core.ts. `src`/`dst` reference can:// ids assignIds already stamped above. + const literal = resolveLiteralConfigUses(app, root.external_symbols ?? {}); + root.config_uses = literal.uses; + root.config_reads = literal.reads; root.call_graph = reidentifyCallGraph(app.call_graph ?? [], idBySig, dangling); } + // config_use dataflow tiers (#101 unit C3): widens the literal tier over AST symbol resolution + // (intra) and resolved internal call sites (interproc, -a 4). Needs the real AST, so it's + // gated on `project` too, not just the level — core.ts always has one to pass at level >= 3. + if (level >= 3 && project) { + const widened = widenConfigUses(app, project, { uses: root.config_uses, reads: root.config_reads }, level); + root.config_uses = widened.uses; + root.config_reads = widened.reads; + } + // L3/L4 — grow body{} + cfg/cdg/ddg/summary on callables and param_in/param_out on the app. let k_limit: number | undefined; if (level >= 3 && pg) { @@ -84,9 +147,9 @@ export function finalizeAnalysis(app: AnalysisInternal, pg: ProgramGraphs | null analyzer: { name: ANALYZER_NAME, version: ANALYZER_VERSION }, application: root, }; - // The wire copy: deep, detached from the live tree, internal fields stripped by key. - const application = JSON.parse( - JSON.stringify(envelope, (key, value) => (INTERNAL_KEYS.has(key) ? undefined : value)), - ) as TSAnalysis; + // The wire copy: deep, detached from the live tree, internals stripped STRUCTURALLY — + // structuredClone instead of a stringify roundtrip (the string form OOMs at vscode-L4 scale). + const application = structuredClone(envelope) as TSAnalysis; + stripInternal(application.application); return { application, internal: app, ...(pg ? { program_graphs: pg } : {}), idBySig, collisions, dangling }; } diff --git a/src/schema/ids.ts b/src/schema/ids.ts index 860f0b5..7876a2e 100644 --- a/src/schema/ids.ts +++ b/src/schema/ids.ts @@ -29,6 +29,34 @@ export function idFromSig(moduleId: string, modulePrefix: string, sig: string): return `${moduleId}/${tail.split(".").join("/")}`; } +/** + * Repository-artifact ids. Leading "./" and "/" are dropped as SEPARATORS only — dotfiles + * (`.env`, `.github/...`) keep their leading dot (python's rule). + */ +export function artifactIdOf(appName: string, relPath: string): string { + let rel = relPath.replace(/\\/g, "/"); + while (rel.startsWith("./")) rel = rel.slice(2); + rel = rel.replace(/^\/+/, ""); + // Language-NEUTRAL namespace (python PR #160): the first segment is `artifact`, not a + // language — sibling analyzers over the same repo emit the SAME id for the same file. + return `can://artifact/${appName}/${rel}`; +} + +/** Config-key id: the owning artifact's id, `@key/`, then the dotted path. */ +export function configKeyIdOf(artifactId: string, dotted: string): string { + return `${artifactId}@key/${dotted}`; +} + +/** Package URL for an npm package name — the cross-language package id (`pkg:npm/...`). */ +export function purlNpm(name: string): string { + if (name.startsWith("@")) { + const slash = name.indexOf("/"); + const scope = encodeURIComponent(name.slice(0, slash)); // "@scope" → "%40scope" (purl spec) + return `pkg:npm/${scope}/${name.slice(slash + 1)}`; + } + return `pkg:npm/${name}`; +} + /** The map key for a callable/type within its parent: the last signature segment (+ accessor tag). */ export function memberKey(sig: string, accessorKind?: string | null): string { const seg = sig.split(".").pop() ?? sig; diff --git a/src/schema/index.ts b/src/schema/index.ts index 828c38b..e4ad97c 100644 --- a/src/schema/index.ts +++ b/src/schema/index.ts @@ -2,5 +2,6 @@ // syntactic and semantic phases). export * from "./schema"; export * from "./signatures"; +export * from "./checker"; export * from "./graphs"; export * from "./emit"; diff --git a/src/schema/l1Body.ts b/src/schema/l1Body.ts index 903c562..2dddc70 100644 --- a/src/schema/l1Body.ts +++ b/src/schema/l1Body.ts @@ -1,12 +1,15 @@ /** * L1 body population — python's `l1_body.py`: materialize each callable's `body{}` `call` nodes * from the INTERNAL `call_sites`, `callee: null` (the sanctioned null→id refinement happens at - * L2, l2Callees.ts). + * L2, l2Callees.ts). Also materializes `config_access` nodes from `config_accesses` (#101 unit + * C1) — a TS-native addition: the dominant env-read idiom (`process.env.FOO`) is a property + * access, not a call, so python's call-based detector table has nothing to mirror here. * * Rebuilds `body{}` WHOLESALE every run and deletes the derived edge lists — body, cfg/cdg/ddg/ * summary, and callee resolution are per-run projections (they embed per-run ids and per-level - * depth), while `call_sites` are the cached source of truth. That wholesale rebuild is what makes - * the whole pass chain idempotent across repeated emissions at different levels. + * depth), while `call_sites`/`config_accesses` are the cached source of truth. That wholesale + * rebuild is what makes the whole pass chain idempotent across repeated emissions at different + * levels. */ import type { AnalysisInternal, TSBodyNode, TSCallable, TSCallsite, TSModule } from "./schema"; @@ -50,7 +53,25 @@ function callNodeOf(cs: TSCallsite): TSBodyNode { function resetCallable(c: TSCallable): void { const body: Record = {}; - for (const [key, cs] of callBodyKeys(c.call_sites)) body[key] = callNodeOf(cs); + // Defensive reads: a warm .codeanalyzer cache written by an earlier build of the SAME + // analyzer_version (loadCache invalidates on version change, not shape) can hand this a + // TSCallable that predates a field added mid-version — `config_accesses` (#101 unit C1) is + // exactly that case. `?? []` is the whole fix; it does not change the cache format or the + // invalidation rule, only tolerates data narrower than today's contract. + for (const [key, cs] of callBodyKeys(c.call_sites ?? [])) body[key] = callNodeOf(cs); + // config_access nodes share the body key space with calls: allocate AFTER them, disambiguating + // against keys already present so a read and a call on one line never collide. + for (const ca of c.config_accesses ?? []) { + const base = `${ca.start_line}:${ca.start_column}`; + let key = base; + for (let k = 2; key in body; k++) key = `${base}/${k}`; + body[key] = { + kind: "config_access", + span: { start: [ca.start_line, ca.start_column], end: [ca.end_line, ca.end_column], bytes: ca.bytes }, + root: ca.root, + ...(ca.key !== undefined ? { key: ca.key } : {}), + }; + } c.body = body; delete c.cfg; delete c.cdg; diff --git a/src/schema/l2Callees.ts b/src/schema/l2Callees.ts index 7ab3fa0..57b48ba 100644 --- a/src/schema/l2Callees.ts +++ b/src/schema/l2Callees.ts @@ -14,14 +14,23 @@ import type { AnalysisInternal, TSCallEdge, TSCallGraphEdge, TSModule } from "./ import { forEachCallable } from "./schema"; import { callBodyKeys } from "./l1Body"; -export function backfillCallees(app: AnalysisInternal, idBySig: Map): void { +export function backfillCallees( + app: AnalysisInternal, + idBySig: Map, + resolutions?: Map>, +): void { for (const mod of Object.values(app.symbol_table) as TSModule[]) { forEachCallable(mod, (c) => { + const linked = resolutions?.get(c.signature); for (const [key, cs] of callBodyKeys(c.call_sites)) { - if (!cs.callee_signature) continue; + // The resolver's in-place backfill wins; the linker's returned map fills the gaps. Linker + // resolutions are deliberately NOT persisted into callee_signature (cache provenance rule + // — see defuseLinker.ts header). + const sig = cs.callee_signature ?? linked?.get(key); + if (!sig) continue; const node = c.body[key]; if (!node || node.kind !== "call") continue; - node.callee = idBySig.get(cs.callee_signature) ?? null; + node.callee = idBySig.get(sig) ?? null; } }); } diff --git a/src/schema/schema.ts b/src/schema/schema.ts index 7587bb0..d280181 100644 --- a/src/schema/schema.ts +++ b/src/schema/schema.ts @@ -16,8 +16,8 @@ * - `id` fields are stamped per-run by `assignIds` (ids embed `--app-name`; the cached tree must * stay app-name-free). Builders initialize them to "". * - INTERNAL fields (never on the wire; serialize.ts strips them by key): `call_sites`, - * `abs_path`, `content_hash`, `last_modified`, `file_size`. They exist for the call-graph - * resolver, the dataflow join, and the analysis cache. + * `config_accesses`, `abs_path`, `content_hash`, `last_modified`, `file_size`. They exist for + * the call-graph resolver, the dataflow join, and the analysis cache. * * All field names are snake_case so `JSON.stringify` emits keys the SDK Pydantic models parse. */ @@ -124,6 +124,7 @@ export interface TSCallsite { receiver_expr?: string; receiver_type?: string; argument_types: string[]; + arguments: string[]; // raw source text per argument — INTERNAL, feeds the config-use key match type_arguments: string[]; // explicit call type args, foo() return_type?: string; callee_signature?: string; // absent when recorded; backfilled by the resolver call graph @@ -136,13 +137,26 @@ export interface TSCallsite { bytes: [number, number]; // char offsets [start, end] into module.source } +/** INTERNAL — a recognized configuration read (env root access). Never on the wire; the wire's + * view is the `config_access` node in the owning callable's `body{}` (built by the l1Body pass). */ +export interface TSConfigAccess { + root: string; // "process.env" | "import.meta.env" | "Bun.env" + key?: string; // present when statically known + start_line: number; + start_column: number; + end_line: number; + end_column: number; + bytes: [number, number]; +} + // ---------------------------------------------------------------------------------------------- // Body nodes — a callable's `body{}` map, keyed by local id (`line:col`, or `@tag` synthetic). -// L1: `call` nodes; L3 adds statements + @entry/@exit; L4 adds formal/actual param vertices. +// L1: `call` and `config_access` nodes; L3 adds statements + @entry/@exit; L4 adds formal/actual +// param vertices. // ---------------------------------------------------------------------------------------------- export interface TSBodyNode { - kind: string; // "call" | "statement" | "entry" | "exit" | "formal_in" | "actual_in" | … + kind: string; // "call" | "config_access" | "statement" | "entry" | "exit" | "formal_in" | "actual_in" | … span?: TSSpan; callee?: string | null; // `call` nodes: null at L1, refined to an id at L2 (the one sanctioned null) of?: string; // synthetic param vertices: the flowed name ("arg0", "$ret", a global path) @@ -156,6 +170,9 @@ export interface TSBodyNode { return_type?: string; is_constructor_call?: boolean; is_optional_chain?: boolean; + // config_access attributes (copied from the recorded access by the l1Body pass) + root?: string; + key?: string; } // ---------------------------------------------------------------------------------------------- @@ -258,6 +275,7 @@ export interface TSCallable { // INTERNAL (stripped from the wire) abs_path: string; // ABSOLUTE file path of the declaration — the resolver's AST-index key call_sites: TSCallsite[]; + config_accesses: TSConfigAccess[]; // INTERNAL (stripped from the wire) } // ---------------------------------------------------------------------------------------------- @@ -324,6 +342,90 @@ export interface TSModule { file_size?: number; } +// ---------------------------------------------------------------------------------------------- +// Repository-artifact layer (#101; parity with codeanalyzer-python PR #160 / spec +// 2026-08-27-artifacts-and-dependencies-design.md): recognized non-code files as nodes with +// LANGUAGE-NEUTRAL ids, plus evidence-tagged dependency records and the unresolved-import +// hygiene signal. Application-anchored, level-free — identical at every -a level. Capture is +// broad (every rules-matched file becomes a node, verbatim source, unbounded by decision); +// extraction is narrow (only dependency-manifest roles feed `dependencies` this unit). +// ---------------------------------------------------------------------------------------------- + +/** A configuration key flattened out of a config-bearing artifact (#101 unit B). */ +export interface TSConfigKey { + id: string; // `${artifactId}@key/${dotted}` — stamped per-run by assignIds + key: string; // dotted path; numeric segments for arrays ("services.web.ports.0") + namespace: string; // env|json|yaml|toml|ini|properties|dockerfile + value?: string | number | boolean; // present by default; absent under --no-artifact-text + span?: TSSpan; // best-effort: exact for yaml (the parser retains node positions); + // line-based for env/ini/dockerfile; ABSENT for json/jsonc — JSON.parse discards + // source positions, and re-deriving one by searching the text for the key token would + // point at the wrong occurrence whenever a key name repeats under different parents + // (routine in tsconfig/compose). Absent is honest; a wrong span is a lie a consumer would act on. + references: string[]; // recognized ${VAR}/$VAR tokens, deduped, in order +} + +/** A recognized non-code file (config, manifest, CI, container spec). */ +export interface TSArtifact { + id: string; // can://artifact// — language-NEUTRAL namespace, stamped per-run + kind: "artifact"; + path: string; // repo-relative POSIX path (also the map key) + format: string; // json | jsonc | yaml | toml | ini | requirements? | dockerfile | yarnlock | text | env | binary + roles: string[]; // dependency-manifest | tool-config | container-image | service-topology | ci | env | packaging | legal | docs | script | unknown + size_bytes: number; + sha256: string; + source: string; // verbatim, unbounded by decision (spec §3) + text_truncated: boolean; // true when `source` is a prefix, not the full file + extraction: "none" | "partial" | "full"; + config_keys: TSConfigKey[]; // contained children; containment mirrors DEFINES_CONFIG +} + +/** One third-party dependency (declared or lockfile-only transitive), evidence-tagged via `prov`. */ +export interface TSDependency { + name: string; // npm-native, @scope kept + spec: string; // as declared ("^4.17.21"); "" when the section value is not a string + kind: "runtime" | "dev" | "optional" | "peer" | "build"; // `peer` is the spec'd additive npm token + extras: string[]; // npm has none — always [] (shared shape parity) + declared_in: string; // TSArtifact id (a manifest for direct:true, the lock for direct:false) + direct: boolean; // false = lockfile-only transitive (no manifest declares it) + locked_version?: string; + provides_imports: string[]; // import specifiers this distribution provides (npm: the name; @types/x: x) + prov: string[]; // declared | lockfile | installed-metadata | heuristic +} + +/** A non-relative import no declared dependency accounts for (the dependency-hygiene signal). */ +export interface TSImportBinding { + module: string; // the specifier root ("express", "@scope/pkg") + bound_to?: string; // best-effort distribution name when partially bound + prov: string[]; +} + +// ---------------------------------------------------------------------------------------------- +// config_use literal tier (#101 unit C2/C3): joins a recognized config READ (a `config_access` or +// detector-table CALL body node) to the declared `TSConfigKey`(s) it names. Runs with the L2 stage +// (src/semantic_analysis/configUse.ts) because CALL rules need the resolved call graph. `src`/ +// `site` are GLOBAL ordinal ids (`@`); `dst` is a TSConfigKey id. +// ---------------------------------------------------------------------------------------------- + +/** One resolved config read: a recognized read whose key closed on exactly one literal that + * matches a declared ConfigKey. `src` is the read's GLOBAL ordinal id; `dst` the key's id. */ +export interface TSConfigUse { + src: string; + dst: string; + prov: Array<"literal" | "dataflow">; +} + +/** A recognized read that resolved to no declared key — first class, so an untraceable read is + * as visible as a traced one. `config_reads` SHRINKS as levels rise (higher tiers resolve some); + * that is deliberate and is the layer's one non-monotonic section. */ +export interface TSConfigRead { + site: string; // GLOBAL ordinal id + callee: string; // the read root ("process.env") or the resolved callee id for call rules + key?: string; // set only for reason "undefined-key" + reason: "non-literal" | "undefined-key"; + prov: Array<"literal" | "dataflow">; +} + // ---------------------------------------------------------------------------------------------- // Call-graph edge (identity-only, provider output; endpoints are signature strings until the // call-graph-ids pass rewrites them onto can:// ids at L2) @@ -354,10 +456,10 @@ export interface TSExternalSymbol { module: string; // the import/require specifier, e.g. "node:fs", "express", "@scope/pkg" } -// A first-party anonymous callback that Jelly resolves as a call-graph endpoint but the symbol -// table never names (the canonicalizer returns null for anonymous functions). The map key IS the -// synthesized signature `:`, so an edge `source`/ -// `target` byte-matches it just like a real `Callable.signature` or `TSExternalSymbol.signature`. +// A first-party anonymous callback a call-graph builder resolved as an edge endpoint but could +// not name against the symbol table (a residual-fallback safety net; since 2.1.0 the tree names +// anonymous callables positionally, so this map is normally empty). The map key IS the +// synthesized signature, so an edge `source`/`target` byte-matches it like a real signature. export interface TSSynthesizedCallable { name: string; // display name — always ""; the signature carries the precise identity path: string; // owning module key (project-relative POSIX path WITH extension) @@ -375,6 +477,13 @@ export interface AnalysisInternal { call_graph: TSCallEdge[]; external_symbols: Record; synthesized_callables: Record; + /** Repository-artifact layer (level-free). */ + artifacts?: Record; + dependencies?: TSDependency[]; + unresolved_imports?: TSImportBinding[]; + /** config_use literal tier (#101 unit C2/C3) — stamped by finalizeAnalysis, not by core.ts. */ + config_uses?: TSConfigUse[]; + config_reads?: TSConfigRead[]; } // ---------------------------------------------------------------------------------------------- @@ -405,6 +514,13 @@ export interface TSApplication { call_graph: TSCallGraphEdge[]; // L2 — callable → callable (empty at L1) param_in: TSParamEdge[]; // L4 (empty until L4) param_out: TSParamEdge[]; // L4 + /** Repository-artifact layer — identical at every level (#101, python PR #160 parity). */ + artifacts: Record; + dependencies: TSDependency[]; + unresolved_imports: TSImportBinding[]; + /** config_use literal tier (#101 unit C2/C3) — empty until L2; CALL rules need the call graph. */ + config_uses: TSConfigUse[]; + config_reads: TSConfigRead[]; // TS-additive (parity): edge endpoints outside the containment tree need an id home. external_symbols?: Record; // L2 — library call targets, keyed by id // L2 — 2.1.0 compatibility index: pre-2.1.0 anonymous-callable id → the tree id that replaced @@ -417,7 +533,7 @@ export interface TSApplication { export interface TSCallGraphEdge { src: string; dst: string; - prov: string[]; // provenance, e.g. ["tsc"], ["jelly"] + prov: string[]; // provenance, e.g. ["tsc"], ["defuse"], ["import"] weight: number; } diff --git a/src/schema/signatures.ts b/src/schema/signatures.ts index e228fdf..4aa772b 100644 --- a/src/schema/signatures.ts +++ b/src/schema/signatures.ts @@ -6,6 +6,7 @@ */ import { Node } from "ts-morph"; import { fileKeyOf, signatureOf, constructorSignatureOf } from "./schema"; +import { aliasedSymbolOf, symbolAt } from "./checker"; /** The name a node contributes to a signature's dotted member chain, or null if it contributes none. */ export function contributorName(node: Node): string | null { @@ -29,7 +30,7 @@ export function contributorName(node: Node): string | null { /** * The segment an unnamed function-like node contributes. Position is the only discriminant a - * nameless callable has, and it is the one both the resolver and Jelly can compute independently + * nameless callable has, and it is the one every call-graph builder can compute independently * — which is what keeps caller-side and callee-side ids byte-identical. Angle brackets mark the * segment synthetic (the ``/`` convention). It joins the dotted chain, so an * anonymous callable lives in the durable id tier and never collides with the `@line:col` @@ -82,16 +83,16 @@ export function computeSignatureForDecl(node: Node, root: string): string | null return signatureOf(modulePrefix, ...parts); } -/** Resolve the declaration a call/new expression targets, following import aliases. */ +/** Resolve the declaration a call/new/tagged-template expression targets, following import aliases. */ export function resolveCalleeDecl(call: Node): Node | undefined { - if (!Node.isCallExpression(call) && !Node.isNewExpression(call)) return undefined; - const expr = call.getExpression(); + if (!Node.isCallExpression(call) && !Node.isNewExpression(call) && !Node.isTaggedTemplateExpression(call)) return undefined; + const expr = Node.isTaggedTemplateExpression(call) ? call.getTag() : call.getExpression(); let symNode: Node = expr; if (Node.isPropertyAccessExpression(expr)) symNode = expr.getNameNode(); else if (Node.isElementAccessExpression(expr)) return undefined; // dynamic dispatch — best-effort skip - let sym = symNode.getSymbol(); + let sym = symbolAt(symNode); if (!sym) return undefined; - const aliased = sym.getAliasedSymbol(); + const aliased = aliasedSymbolOf(sym); if (aliased) sym = aliased; const decls = sym.getDeclarations(); return decls && decls.length ? decls[0] : undefined; diff --git a/src/semantic_analysis/callGraph.ts b/src/semantic_analysis/callGraph.ts index e20ea08..a60835a 100644 --- a/src/semantic_analysis/callGraph.ts +++ b/src/semantic_analysis/callGraph.ts @@ -20,7 +20,33 @@ import { type TSType, forEachCallable, } from "../schema"; -import { resolveCalleeSignature } from "../schema"; +import { fileKeyOf, resolveCalleeSignature } from "../schema"; +import { isCallableDecl } from "../schema"; + +/** The nearest ancestor that is itself a callable declaration (incl. `const f = () => …`), or undefined. */ +function enclosingCallable(node: Node): Node | undefined { + for (const a of node.getAncestors()) { + if (isCallableDecl(a)) return a; + if (Node.isVariableDeclaration(a)) { + const init = a.getInitializer?.(); + if (init && (Node.isArrowFunction(init) || Node.isFunctionExpression(init))) return a; + } + } + return undefined; +} + +/** Inside a NON-static class property initializer — owned by the constructor, not module scope. */ +export function inInstancePropInit(node: Node): boolean { + for (const a of node.getAncestors()) { + if (isCallableDecl(a)) return false; + if (Node.isPropertyDeclaration(a)) return !(a as unknown as { isStatic?: () => boolean }).isStatic?.(); + } + return false; +} + +function fileKeyOfNode(node: Node, root: string): { fileKey: string; modulePrefix: string } { + return fileKeyOf(node.getSourceFile().getFilePath(), root); +} import type { Logger } from "../utils"; import { type ExternalIndex, buildExternalIndex, resolvePhantom } from "./phantoms"; @@ -33,7 +59,7 @@ export interface CallGraphResult { edges: TSCallEdge[]; external_symbols: Record; // Anonymous callbacks resolved as edge endpoints that the symbol table doesn't name. Empty for - // the tsc resolver (its edges are gated to real symbol-table signatures); populated by Jelly. + // the tsc resolver (its edges are gated to real symbol-table signatures); a residual-fallback net. synthesized_callables: Record; } @@ -130,6 +156,42 @@ export function buildCallGraph( let rtaCount = 0; let phantomCount = 0; let unresolved = 0; + + // Module-scope sweep (python #131 parity): a call with NO enclosing callable — top-level + // statements, class property initializers, namespace bodies — is attributed to the MODULE + // (source = the module prefix, re-identified onto the module node at L2). These sites are + // never recorded in call_sites (modules have no body{}), so resolve them straight off the AST. + for (const node of callExprIndex.values()) { + if (enclosingCallable(node) || inInstancePropInit(node)) continue; + const fileKey = fileKeyOfNode(node, root); + if (only && !only.has(fileKey.fileKey)) continue; + const source = fileKey.modulePrefix; + const r = resolveCalleeSignature(node, root, allSignatures); + if (r?.external) { + if (phantoms) { + if (!external_symbols[r.signature]) external_symbols[r.signature] = { name: r.external.member, module: r.external.module }; + addPhantomEdge(source, r.signature, r.external.module); + phantomCount++; + } else unresolved++; + continue; + } + if (!r) { + if (phantoms) { + const ph = resolvePhantom(node, extIndexFor(node)); + if (ph) { + if (!external_symbols[ph.signature]) external_symbols[ph.signature] = { name: ph.member, module: ph.module }; + addPhantomEdge(source, ph.signature, ph.module); + phantomCount++; + continue; + } + } + unresolved++; + continue; + } + addEdge(source, r.signature, false); + resolved++; + } + for (const caller of callables) { for (const site of caller.call_sites) { const node = callExprIndex.get( @@ -252,13 +314,13 @@ function indexClasses( } } -function indexCallExpressions(project: Project): Map { +export function indexCallExpressions(project: Project): Map { const idx = new Map(); for (const sf of project.getSourceFiles()) { const fp = sf.getFilePath(); if (sf.isDeclarationFile() || fp.includes("/node_modules/")) continue; sf.forEachDescendant((n) => { - if (Node.isCallExpression(n) || Node.isNewExpression(n)) { + if (Node.isCallExpression(n) || Node.isNewExpression(n) || Node.isTaggedTemplateExpression(n)) { const s = sf.getLineAndColumnAtPos(n.getStart()); const e = sf.getLineAndColumnAtPos(n.getEnd()); // Full span (start AND end) keys the node uniquely; chained calls like `f(x).g(y)` diff --git a/src/semantic_analysis/configUse.ts b/src/semantic_analysis/configUse.ts new file mode 100644 index 0000000..4b62a5d --- /dev/null +++ b/src/semantic_analysis/configUse.ts @@ -0,0 +1,129 @@ +/** + * config_use literal tier (#101 unit C3). Runs with the call graph — the L2 stage — because call + * rules need resolved callees. Joins a read's statically-known key to declared ConfigKeys on + * (namespace, key); a read that resolves to nothing becomes a first-class `config_reads` record. + * Deterministic: every output list is sorted. + * + * Imports the schema/l1Body LEAF files directly (never the `../schema` barrel, which re-exports + * `emit.ts` — the module that imports this one): going through the barrel here would close a + * import cycle back on emit.ts. + */ +import type { AnalysisInternal, TSCallable, TSConfigRead, TSConfigUse } from "../schema/schema"; +import { forEachCallable } from "../schema/schema"; +import { callBodyKeys } from "../schema/l1Body"; +import { ACCESS_RULES, CALL_RULES, type CallRule } from "./configUseRules"; + +/** (namespace, key) → declared ConfigKey ids, sorted. */ +export function keyIndex(app: AnalysisInternal): Map { + const idx = new Map(); + for (const art of Object.values(app.artifacts ?? {})) { + for (const ck of art.config_keys) { + const k = `${ck.namespace} ${ck.key}`; + const arr = idx.get(k) ?? []; + arr.push(ck.id); + idx.set(k, arr); + } + } + for (const arr of idx.values()) arr.sort(); + return idx; +} + +export interface ConfigUseSets { + uses: TSConfigUse[]; + reads: TSConfigRead[]; +} + +/** + * `externalsById` is the ASSEMBLED root's `external_symbols` (id-keyed, homed by `homeExternals`), + * not `AnalysisInternal.external_symbols` (signature-keyed) — a `call` node's resolved `callee` is + * already the can:// external id, so the lookup here is a direct id hit, no signature round-trip. + */ +export function resolveLiteralConfigUses( + app: AnalysisInternal, + externalsById: Record, +): ConfigUseSets { + const idx = keyIndex(app); + const uses: TSConfigUse[] = []; + const reads: TSConfigRead[] = []; + const rootNamespaces = new Map(ACCESS_RULES.map((r) => [r.root, r.namespaces])); + + for (const mod of Object.values(app.symbol_table)) { + forEachCallable(mod, (c) => { + for (const [local, node] of Object.entries(c.body)) { + // CALL rules: a `call` node whose resolved callee matches module+callable, with the key + // at `key_arg`. The key literal comes from the recorded call site's arguments; a call + // whose key argument is not a literal is a non-literal read, same as a dynamic access. + if (node.kind === "call") { + const rule = matchCallRule(node.callee, externalsById); + if (!rule) continue; + const site = `${c.id}@${local}`; + const callee = node.callee as string; // matchCallRule only returns non-null for a string callee + const key = literalArgumentAt(c, local, rule.key_arg); + if (key === undefined) { + reads.push({ site, callee, reason: "non-literal", prov: ["literal"] }); + continue; + } + const dsts = rule.namespaces.flatMap((ns) => idx.get(`${ns} ${key}`) ?? []); + if (!dsts.length) { + reads.push({ site, callee, key, reason: "undefined-key", prov: ["literal"] }); + continue; + } + for (const dst of [...new Set(dsts)].sort()) uses.push({ src: site, dst, prov: ["literal"] }); + continue; + } + if (node.kind !== "config_access") continue; + const site = `${c.id}@${local}`; + const root = node.root ?? ""; + const namespaces = rootNamespaces.get(root) ?? ["env"]; + if (node.key === undefined) { + reads.push({ site, callee: root, reason: "non-literal", prov: ["literal"] }); + continue; + } + const key = node.key; + const dsts = namespaces.flatMap((ns) => idx.get(`${ns} ${key}`) ?? []); + if (!dsts.length) { + reads.push({ site, callee: root, key, reason: "undefined-key", prov: ["literal"] }); + continue; + } + for (const dst of [...new Set(dsts)].sort()) uses.push({ src: site, dst, prov: ["literal"] }); + } + }); + } + uses.sort((a, b) => a.src.localeCompare(b.src) || a.dst.localeCompare(b.dst)); + reads.sort((a, b) => a.site.localeCompare(b.site) || (a.key ?? "").localeCompare(b.key ?? "")); + return { uses, reads }; +} + +/** + * A call node's resolved `callee` id names an external as + * `can://…/@external//`. Match prefix-aware on module (a rule for `config` + * matches `config` and `config/lib/x`) and exactly on the member. + */ +export function matchCallRule(callee: unknown, externals: Record): CallRule | null { + if (typeof callee !== "string") return null; + const ext = externals[callee]; + if (!ext) return null; + for (const rule of CALL_RULES) { + if (rule.callable !== ext.name) continue; + if (ext.module === rule.module || ext.module.startsWith(`${rule.module}/`)) return rule; + } + return null; +} + +/** The string literal at `argIndex` of the call site backing this body key, or undefined. A + * backtick argument with an interpolation (`` `PAYMENT_${x}` ``) is NOT a literal — the source + * text is captured verbatim, so treating it as one would record a `${x}`-shaped fact as a real + * key. A backtick with no `${` is a genuine static string and still resolves. */ +export function literalArgumentAt(c: TSCallable, bodyKey: string, argIndex: number): string | undefined { + for (const [key, cs] of callBodyKeys(c.call_sites)) { + if (key !== bodyKey) continue; + const raw = cs.arguments?.[argIndex]; + if (raw === undefined) return undefined; + const trimmed = raw.trim(); + const m = /^["'`](.*)["'`]$/.exec(trimmed); + if (!m) return undefined; + if (trimmed.startsWith("`") && m[1]?.includes("${")) return undefined; // interpolated — non-literal + return m[1] as string; + } + return undefined; +} diff --git a/src/semantic_analysis/configUseRules.ts b/src/semantic_analysis/configUseRules.ts new file mode 100644 index 0000000..92b44c6 --- /dev/null +++ b/src/semantic_analysis/configUseRules.ts @@ -0,0 +1,31 @@ +/** + * Shipped config-use detector table (#101 unit C2). Two rule kinds: ACCESS rules name env roots + * whose member/element reads are configuration reads (recognized in builders.ts, which mints the + * `config_access` body node); CALL rules name a module+callable whose argument at `key_arg` + * carries the key. No user-extension flag — same posture as the artifact rules table. + */ +export interface AccessRule { + root: string; + namespaces: string[]; +} +export interface CallRule { + id: string; + module: string; // matched prefix-aware against the resolved callee's external module + callable: string; + key_arg: number; + namespaces: string[]; +} + +export const ACCESS_RULES: AccessRule[] = [ + { root: "process.env", namespaces: ["env"] }, + { root: "import.meta.env", namespaces: ["env"] }, + { root: "Bun.env", namespaces: ["env"] }, +]; + +export const CALL_RULES: CallRule[] = [ + { id: "deno.env.get", module: "Deno.env", callable: "get", key_arg: 0, namespaces: ["env"] }, + { id: "config.get", module: "config", callable: "get", key_arg: 0, namespaces: ["json", "yaml"] }, + { id: "config.has", module: "config", callable: "has", key_arg: 0, namespaces: ["json", "yaml"] }, + { id: "nconf.get", module: "nconf", callable: "get", key_arg: 0, namespaces: ["json", "yaml", "env"] }, + { id: "dotenv.parse", module: "dotenv", callable: "parse", key_arg: 0, namespaces: ["env"] }, +]; diff --git a/src/semantic_analysis/defuseLinker.ts b/src/semantic_analysis/defuseLinker.ts new file mode 100644 index 0000000..67fdc61 --- /dev/null +++ b/src/semantic_analysis/defuseLinker.ts @@ -0,0 +1,565 @@ +/** + * The defuse linker — the local pass that backfills call edges the tsc resolver missed + * (docs/design/specs/defuse-linker-call-graph.md, #98; python parity: the Jedi + defuse-linker + * architecture that replaced PyCG). Per-callable and bounded-round only — NO whole-program + * fixpoint; sorted iteration throughout, so the output is deterministic by construction. + * + * Tiers, applied in order (a precise resolution is never widened by a later tier): + * T1 local value chase — alias chains `const f = handler; f()` through bounded + * symbol→declaration hops (the checker's alias-following covers imports). + * T2 decorator invocations — `@Get(':id')` on a method/accessor becomes an edge + * decorated-callable → decorator target, EDGE-ONLY (decorator calls live outside walkBody's + * reach, so there is no body call node to refine — matching the historical Jelly shape). + * T3 external-callback rule — a function value passed to an external/unresolved callee emits + * enclosing-callable → function-value, EDGE-ONLY (`.map(u => …)`, `app.get('/x', handler)`). + * Deliberate divergence from python (JS is callback-central); recorded in the spec. + * T4 bounded interprocedural votes — (a) a parameter-invoking site (`cb()`) resolves to the + * function values passed at that position by resolved-internal callers (two rounds: round + * one's resolutions vote before round two); (b) `const f = factory(); f()` resolves through + * the factory's unique returned function. + * T5 CHA-by-name — receiver sites that survive every typed tier resolve to every internal + * callable of that method name (bounded per site) — the over-approximation Joern emits for + * untyped receivers. Edge-only (ambiguous by definition). + * + * Linker resolutions are returned in a map and applied to the L1 `call` body nodes by + * `backfillCallees` — NEVER written into `callee_signature` (the symbol table round-trips the + * analysis cache; a persisted resolution would resurface on a warm run with tsc provenance). + */ +import { Node, SyntaxKind } from "ts-morph"; +import { CALL_DEP, type TSCallEdge, type TSCallable, type TSCallsite, type TSExternalSymbol, forEachCallable } from "../schema"; +import { aliasedSymbolOf, computeSignatureForDecl, externalHomeOf, fileKeyOf, isCallableDecl, resolveCalleeSignature, symbolAt } from "../schema"; +import { callBodyKeys } from "../schema/l1Body"; +import type { CallGraphContext } from "./provider"; +import type { CallGraphResult } from "./callGraph"; +import { inInstancePropInit, indexCallExpressions } from "./callGraph"; + +/** Per-call-site resolutions for the sanctioned `callee: null→id` refinement: callerSig → bodyKey → calleeSig. */ +export type LinkerResolutions = Map>; + +export interface LinkerOutput { + result: CallGraphResult; + resolutions: LinkerResolutions; +} + +// ponytail: fixed small bounds; tune from the Joern ledger, not from flags (spec: no backend flag). +const ALIAS_CHASE_LIMIT = 8; // hops through `const f = g` chains +const CHA_FAN_LIMIT = 16; // max name-matched targets per T5 site + +export function runDefuseLinker(ctx: CallGraphContext): LinkerOutput { + const { project, symbol_table, root, log } = ctx; + + // The signature universe (full table — cross-program targets resolve) + the name→sigs CHA index. + const allSignatures = new Set(); + const byName = new Map(); + for (const mod of Object.values(symbol_table)) { + forEachCallable(mod, (c) => { + allSignatures.add(c.signature); + const arr = byName.get(c.name) ?? []; + arr.push(c.signature); + byName.set(c.name, arr); + }); + } + for (const sigs of byName.values()) sigs.sort(); + + // Callables to iterate: this program's modules only, sorted for determinism. + const callables: TSCallable[] = []; + for (const [key, mod] of Object.entries(symbol_table)) { + if (ctx.only && !ctx.only.has(key)) continue; + forEachCallable(mod, (c) => callables.push(c)); + } + callables.sort((a, b) => a.signature.localeCompare(b.signature)); + + const callExprIndex = indexCallExpressions(project); + /** The callee expression of a call/new/tagged-template node. */ + const calleeExprOf = (node: Node): Node => + Node.isTaggedTemplateExpression(node) ? node.getTag() : (node as unknown as { getExpression: () => Node }).getExpression(); + const nodeOf = (c: TSCallable, cs: TSCallsite): Node | undefined => + callExprIndex.get(`${c.abs_path}#${cs.start_line}:${cs.start_column}-${cs.end_line}:${cs.end_column}`); + + // A resolved-but-external callee is one whose signature is not in the symbol table. + const isExternalSig = (sig: string | undefined): boolean => !!sig && !allSignatures.has(sig); + + // --------------------------------------------------------------------------------------------- + // edge/resolution accumulation + // --------------------------------------------------------------------------------------------- + const edges = new Map(); + const addEdge = (source: string, target: string): void => { + const k = `${source} ${target}`; + const ex = edges.get(k); + if (ex) ex.weight++; + else edges.set(k, { source, target, type: CALL_DEP, weight: 1, provenance: ["defuse"], tags: {} }); + }; + const external_symbols: Record = {}; + const resolutions: LinkerResolutions = new Map(); + const resolve = (callerSig: string, bodyKey: string, targetSig: string): void => { + addEdge(callerSig, targetSig); + let m = resolutions.get(callerSig); + if (!m) resolutions.set(callerSig, (m = new Map())); + m.set(bodyKey, targetSig); + }; + + /** + * The signature of the first-party callable a VALUE expression denotes, chasing bounded alias + * chains: a bare arrow/function expression, an identifier for a function declaration, a + * `const f = () => …` binding, or `const f = g` (g eventually a function) — else null. + */ + const functionValueSig = (expr: Node): string | null => { + // IIFE / parenthesized function values: `(() => …)()`, `(function f() {})()`. + while (Node.isParenthesizedExpression(expr)) expr = expr.getExpression(); + if (Node.isArrowFunction(expr) || Node.isFunctionExpression(expr)) { + const s = computeSignatureForDecl(expr, root); + return s && allSignatures.has(s) ? s : null; + } + if (!Node.isIdentifier(expr)) return null; + let node: Node = expr; + for (let hop = 0; hop < ALIAS_CHASE_LIMIT; hop++) { + let sym = symbolAt(node); + if (!sym) return null; + const aliased = aliasedSymbolOf(sym); + if (aliased) sym = aliased; + const decl = sym.getDeclarations()?.[0]; + if (!decl) return null; + if (Node.isFunctionDeclaration(decl) || Node.isArrowFunction(decl) || Node.isFunctionExpression(decl) || Node.isMethodDeclaration(decl)) { + const s = computeSignatureForDecl(decl, root); + return s && allSignatures.has(s) ? s : null; + } + if (Node.isVariableDeclaration(decl)) { + const init = decl.getInitializer(); + if (!init) return null; + if (Node.isArrowFunction(init) || Node.isFunctionExpression(init)) { + const s = computeSignatureForDecl(decl, root); + return s && allSignatures.has(s) ? s : null; + } + if (Node.isIdentifier(init)) { + node = init; // alias chain: keep chasing + continue; + } + return null; + } + return null; + } + return null; + }; + + /** The parameter index of `expr` within `enclosing`, when it names one of its parameters. */ + const paramIndexOf = (expr: Node, enclosing: TSCallable): number | null => { + if (!Node.isIdentifier(expr)) return null; + const decl = symbolAt(expr)?.getDeclarations()?.[0]; + if (!decl || !Node.isParameterDeclaration(decl)) return null; + const name = expr.getText(); + const idx = enclosing.parameters.findIndex((p) => p.name === name); + return idx >= 0 ? idx : null; + }; + + // --------------------------------------------------------------------------------------------- + // main site sweep: T1 chase, T3 callback rule, and the T4/T5 worklists + // --------------------------------------------------------------------------------------------- + interface ParamSite { + enclosing: TSCallable; + bodyKey: string; + paramIndex: number; + propertyName?: string; // `template.onChange(...)` where `template` is the parameter + } + interface FactorySite { + enclosing: TSCallable; + bodyKey: string; + factorySig: string; // resolved-internal callee of the binding's initializer call + } + interface ReceiverSite { + enclosing: TSCallable; + cs: TSCallsite; + } + interface ThisFieldSite { + enclosing: TSCallable; + bodyKey: string; + node: Node; + fieldName: string; + } + const paramSites: ParamSite[] = []; + const factorySites: FactorySite[] = []; + const receiverSites: ReceiverSite[] = []; + const thisFieldSites: ThisFieldSite[] = []; + // Reverse index for T4 voting: internal target sig → the AST argument lists of its call sites. + const argsByTarget = new Map(); + const recordCallArgs = (targetSig: string, node: Node | undefined): void => { + if (!node || !allSignatures.has(targetSig)) return; + const args = (node as unknown as { getArguments?: () => Node[] }).getArguments?.() ?? []; + if (!args.length) return; + const arr = argsByTarget.get(targetSig) ?? []; + arr.push(args); + argsByTarget.set(targetSig, arr); + }; + + let t1 = 0; + let t3 = 0; + for (const c of callables) { + for (const [bodyKey, cs] of callBodyKeys(c.call_sites)) { + const node = nodeOf(c, cs); + if (cs.callee_signature) { + recordCallArgs(cs.callee_signature, node); + } else if (node) { + const expr = calleeExprOf(node); + // T1 — local value chase on the callee expression itself. + const chased = functionValueSig(expr); + if (chased) { + resolve(c.signature, bodyKey, chased); + recordCallArgs(chased, node); + t1++; + } else { + const pIdx = paramIndexOf(expr, c); + if (pIdx !== null) { + paramSites.push({ enclosing: c, bodyKey, paramIndex: pIdx }); + } else if (Node.isIdentifier(expr)) { + // T4b — `const f = factory(); f()`: binding initialized by a resolved-internal call. + const decl = symbolAt(expr)?.getDeclarations()?.[0]; + const init = decl && Node.isVariableDeclaration(decl) ? decl.getInitializer() : undefined; + if (init && Node.isCallExpression(init)) { + const r = resolveCalleeSignature(init, root, allSignatures); + if (r && !r.external && allSignatures.has(r.signature)) { + factorySites.push({ enclosing: c, bodyKey, factorySig: r.signature }); + } + } + } else if (Node.isPropertyAccessExpression(expr) && cs.receiver_expr != null && !cs.is_constructor_call) { + const recvIdx = paramIndexOf(expr.getExpression(), c); + if (recvIdx !== null) { + // T4a property form: the receiver IS a parameter — candidates are the matching + // object-literal property values passed at that position by resolved callers. + paramSites.push({ enclosing: c, bodyKey, paramIndex: recvIdx, propertyName: expr.getName() }); + } else if (cs.receiver_expr === "this") { + // T4c — `this.field(...)`: the field's value flows from the constructor (a + // parameter property or a `this.field = …` assignment); resolved below, feeding + // the T4 vote rounds. Falls back to T5 if the chain yields nothing. + thisFieldSites.push({ enclosing: c, bodyKey, node, fieldName: cs.method_name }); + } else { + receiverSites.push({ enclosing: c, cs }); + } + } + } + } + // T3 — external-callback rule: function values handed to an external/unresolved callee. + if (node && (!cs.callee_signature || isExternalSig(cs.callee_signature))) { + const args = (node as unknown as { getArguments?: () => Node[] }).getArguments?.() ?? []; + for (const arg of args) { + const fn = functionValueSig(arg); + if (fn && fn !== c.signature) { + addEdge(c.signature, fn); + t3++; + } + } + } + } + } + + // --------------------------------------------------------------------------------------------- + // Module-scope sweep (python #131 parity: module-scope execution is attributed to the MODULE). + // These sites have no call_sites record and no body node — T1 chase and the T3 callback rule + // apply edge-only, with the module prefix as the source. + // --------------------------------------------------------------------------------------------- + const enclosingCallable = (node: Node): Node | undefined => { + for (const a of node.getAncestors()) if (isCallableDecl(a)) return a; + return undefined; + }; + for (const [, node] of [...callExprIndex.entries()].sort(([a], [b]) => a.localeCompare(b))) { + if (enclosingCallable(node) || inInstancePropInit(node)) continue; + const fk = fileKeyOf(node.getSourceFile().getFilePath(), root); + if (ctx.only && !ctx.only.has(fk.fileKey)) continue; + const source = fk.modulePrefix; + const r = resolveCalleeSignature(node, root, allSignatures); + if (r && !r.external) recordCallArgs(r.signature, node); // top-level `register("a", cb)` feeds the vote rounds + if (!r) { + // T1 at module scope: `const f = handler; f()` in top-level code. + const expr = calleeExprOf(node); + const chased = functionValueSig(expr); + if (chased) { + addEdge(source, chased); + t1++; + } + } + // T3 at module scope — the dominant express idiom: `app.get('/x', handler)` top-level. + if (!r || r.external) { + const args = (node as unknown as { getArguments?: () => Node[] }).getArguments?.() ?? []; + for (const arg of args) { + const fn = functionValueSig(arg); + if (fn && fn !== source) { + addEdge(source, fn); + t3++; + } + } + } + } + + // --------------------------------------------------------------------------------------------- + // T2 — decorator invocations (edge-only). The SOURCE is where the decorator executes: the + // decorated callable for method/accessor/parameter decorators; the MODULE for class and + // property decorators (a decorator on a top-level definition runs in module scope — python + // #131's rule, and Joern's own attribution). + // --------------------------------------------------------------------------------------------- + let t2 = 0; + const files = [...project.getSourceFiles()] + .filter((sf) => !sf.isDeclarationFile() && !sf.getFilePath().includes("/node_modules/")) + .sort((a, b) => a.getFilePath().localeCompare(b.getFilePath())); + for (const sf of files) { + sf.forEachDescendant((n) => { + if (!Node.isDecorator(n)) return; + // The edge SOURCE is where the decorator executes: the decorated callable for method/ + // accessor/parameter decorators; the MODULE prefix for class and property decorators. + let owner = n.getParent(); + if (owner && Node.isParameterDeclaration(owner)) owner = owner.getParent(); + let ownerSig: string | null = null; + if (owner && (Node.isMethodDeclaration(owner) || Node.isGetAccessorDeclaration(owner) || Node.isSetAccessorDeclaration(owner))) { + ownerSig = computeSignatureForDecl(owner, root); + if (!ownerSig || !allSignatures.has(ownerSig)) return; + } else if (owner && (Node.isClassDeclaration(owner) || Node.isPropertyDeclaration(owner))) { + ownerSig = fileKeyOf(sf.getFilePath(), root).modulePrefix; + } else { + return; + } + const expr = n.getExpression(); + let targetSig: string | null = null; + let external: { module: string; member: string } | null = null; + if (Node.isCallExpression(expr)) { + const r = resolveCalleeSignature(expr, root, allSignatures); + if (r) { + targetSig = r.signature; + external = r.external ?? null; + } + } else if (Node.isIdentifier(expr)) { + const direct = functionValueSig(expr); + if (direct) targetSig = direct; + else { + const sym = symbolAt(expr); + const decl = + (sym ? aliasedSymbolOf(sym)?.getDeclarations()?.[0] : undefined) ?? sym?.getDeclarations()?.[0]; + const home = decl ? externalHomeOf(decl) : null; + if (home) { + const member = expr.getText(); + targetSig = `${home.module}.${member}`; + external = { module: home.module, member }; + } + } + } + if (!targetSig) return; + if (external) { + if (!ctx.phantoms) return; + if (!external_symbols[targetSig]) external_symbols[targetSig] = { name: external.member, module: external.module }; + } + addEdge(ownerSig, targetSig); + t2++; + }); + } + + // --------------------------------------------------------------------------------------------- + // T4c — `this.field(...)` through the constructor: a field assigned from a ctor parameter + // (parameter property or `this.f = param`) calls whatever function values the class's `new` + // sites passed at that position; a field assigned a function value in the ctor calls it + // directly. Candidates feed argsByTarget so the T4 rounds resolve the callbacks' OWN + // param-invoking sites (`write()` inside a registered migration callback). Bounded: direct + // ctor args only, no transitive flow. + // --------------------------------------------------------------------------------------------- + const classFieldSources = new Map>(); + const fieldSourcesOf = (cls: Node): Map => { + let m = classFieldSources.get(cls); + if (m) return m; + m = new Map(); + const ctor = (cls as unknown as { getConstructors?: () => Node[] }).getConstructors?.()?.[0]; + if (ctor) { + const params = (ctor as unknown as { getParameters: () => Node[] }).getParameters(); + params.forEach((p, i) => { + const pp = p as unknown as { getName: () => string; getModifiers?: () => Node[] }; + if ((pp.getModifiers?.() ?? []).length) m?.set(pp.getName(), { paramIndex: i }); + }); + const paramNames = new Map(params.map((p, i) => [(p as unknown as { getName: () => string }).getName(), i])); + ctor.forEachDescendant((d) => { + if (!Node.isBinaryExpression(d) || d.getOperatorToken().getKind() !== SyntaxKind.EqualsToken) return; + const lhs = d.getLeft(); + if (!Node.isPropertyAccessExpression(lhs) || lhs.getExpression().getKind() !== SyntaxKind.ThisKeyword) return; + const rhs = d.getRight(); + const idx = Node.isIdentifier(rhs) ? paramNames.get(rhs.getText()) : undefined; + if (idx !== undefined) m?.set(lhs.getName(), { paramIndex: idx }); + else { + const direct = functionValueSig(rhs); + if (direct) m?.set(lhs.getName(), { direct }); + } + }); + } + classFieldSources.set(cls, m); + return m; + }; + /** Function values an ARGUMENT node denotes — directly, or through one bounded parameter hop: + * when the arg is a parameter of the function containing the call, the values passed for that + * parameter at ITS resolved call sites are the candidates (`register(key, migrate)` → + * `new Migration(key, migrate)`). One hop, no fixpoint. */ + const argFlowCandidates = (arg: Node): string[] => { + const direct = functionValueSig(arg); + if (direct) return [direct]; + if (!Node.isIdentifier(arg)) return []; + const decl = symbolAt(arg)?.getDeclarations()?.[0]; + if (!decl || !Node.isParameterDeclaration(decl)) return []; + const owner = decl.getParent(); + if (!owner) return []; + const ownerSig = computeSignatureForDecl(owner, root); + if (!ownerSig) return []; + const idx = ((owner as unknown as { getParameters?: () => Node[] }).getParameters?.() ?? []).findIndex((p) => p === decl); + if (idx < 0) return []; + const out = new Set(); + for (const args of argsByTarget.get(ownerSig) ?? []) { + const a = args[idx]; + const fn = a ? functionValueSig(a) : null; + if (fn) out.add(fn); + } + return [...out].sort(); + }; + let t4c = 0; + for (const site of thisFieldSites.sort((a, b) => a.enclosing.signature.localeCompare(b.enclosing.signature) || a.bodyKey.localeCompare(b.bodyKey))) { + const cls = site.node.getAncestors().find((a) => Node.isClassDeclaration(a) || Node.isClassExpression(a)); + const src = cls ? fieldSourcesOf(cls).get(site.fieldName) : undefined; + const candidates = new Set(); + if (src?.direct) candidates.add(src.direct); + if (src?.paramIndex !== undefined && cls) { + const clsSig = computeSignatureForDecl(cls, root); + for (const args of argsByTarget.get(`${clsSig}.constructor`) ?? []) { + const arg = args[src.paramIndex]; + for (const fn of arg ? argFlowCandidates(arg) : []) candidates.add(fn); + } + } + if (!candidates.size) { + const cs = site.enclosing.call_sites.find((c2) => `${c2.start_line}:${c2.start_column}` === site.bodyKey.split("/")[0]); + if (cs) receiverSites.push({ enclosing: site.enclosing, cs }); // fall back to T5 + continue; + } + const sorted = [...candidates].sort(); + for (const target of sorted) { + addEdge(site.enclosing.signature, target); + recordCallArgs(target, site.node); // the callbacks' own param sites resolve in the T4 rounds + t4c++; + } + if (sorted.length === 1) { + let m = resolutions.get(site.enclosing.signature); + if (!m) resolutions.set(site.enclosing.signature, (m = new Map())); + m.set(site.bodyKey, sorted[0] as string); + } + } + + // --------------------------------------------------------------------------------------------- + // T4 — bounded votes, two rounds (round one's resolutions vote before round two). + // --------------------------------------------------------------------------------------------- + let t4 = 0; + for (let round = 0; round < 2 && paramSites.length; round++) { + const unresolvedNext: ParamSite[] = []; + for (const site of paramSites.sort((a, b) => a.enclosing.signature.localeCompare(b.enclosing.signature) || a.bodyKey.localeCompare(b.bodyKey))) { + const candidates = new Set(); + for (const args of argsByTarget.get(site.enclosing.signature) ?? []) { + const arg = args[site.paramIndex]; + if (!arg) continue; + if (site.propertyName !== undefined) { + // object-literal property flow: `render({ onChange: fn })` → `template.onChange()` + if (Node.isObjectLiteralExpression(arg)) { + const prop = arg.getProperty(site.propertyName); + const init = prop && Node.isPropertyAssignment(prop) ? prop.getInitializer() : undefined; + const fn = init ? functionValueSig(init) : null; + if (fn) candidates.add(fn); + else if (prop && Node.isMethodDeclaration(prop)) { + const s2 = computeSignatureForDecl(prop, root); + if (s2 && allSignatures.has(s2)) candidates.add(s2); + } + } + } else { + const fn = functionValueSig(arg); + if (fn) candidates.add(fn); + } + } + if (!candidates.size) { + unresolvedNext.push(site); + continue; + } + const sorted = [...candidates].sort(); + for (const target of sorted) { + addEdge(site.enclosing.signature, target); + t4++; + // Round-one resolutions feed round two's votes: a cb() site that now targets `target` + // makes the enclosing callable a resolved-internal caller of it. + } + if (sorted.length === 1) { + let m = resolutions.get(site.enclosing.signature); + if (!m) resolutions.set(site.enclosing.signature, (m = new Map())); + m.set(site.bodyKey, sorted[0] as string); + } + } + paramSites.length = 0; + paramSites.push(...unresolvedNext); + } + // T4b — factory returns: resolve through the factory's unique returned function value. + const returnSummary = new Map(); + const uniqueReturnedFn = (factorySig: string): string | null => { + if (returnSummary.has(factorySig)) return returnSummary.get(factorySig) as string | null; + let out: string | null = null; + // Find the factory's AST via any recorded call-site node? Cheaper: search the sorted callables + // list (same program) for the signature, then its declaration through the call-expression + // index is unavailable — walk the source file at its span instead. + const fc = callables.find((c) => c.signature === factorySig); + if (fc) { + const sf = project.getSourceFile(fc.abs_path); + const declNode = sf?.getDescendantAtPos(fc.span.bytes[0]); + const fnNode = declNode ? [declNode, ...declNode.getAncestors()].find((a) => computeSignatureForDecl(a, root) === factorySig) : undefined; + if (fnNode) { + returnSummary.set(factorySig, null); // cycle guard before descending + const returned = new Set(); + fnNode.forEachDescendant((d) => { + if (!Node.isReturnStatement(d)) return; + const e = d.getExpression(); + if (!e) return; + const fn = functionValueSig(e); + if (fn) { + returned.add(fn); + return; + } + // chained: `return makeInner()` — follow ONE resolved-internal level, memoized + if (Node.isCallExpression(e)) { + const r = resolveCalleeSignature(e, root, allSignatures); + if (r && !r.external && allSignatures.has(r.signature)) { + const inner = uniqueReturnedFn(r.signature); + if (inner) { + returned.add(inner); + return; + } + } + } + returned.add(""); + }); + if (returned.size === 1 && !returned.has("")) out = [...returned][0] as string; + } + } + returnSummary.set(factorySig, out); + return out; + }; + for (const site of factorySites.sort((a, b) => a.enclosing.signature.localeCompare(b.enclosing.signature) || a.bodyKey.localeCompare(b.bodyKey))) { + const target = uniqueReturnedFn(site.factorySig); + if (target) { + resolve(site.enclosing.signature, site.bodyKey, target); + t4++; + } + } + + // --------------------------------------------------------------------------------------------- + // T5 — CHA-by-name fallback (edge-only, bounded fan). + // --------------------------------------------------------------------------------------------- + let t5 = 0; + for (const site of receiverSites.sort((a, b) => a.enclosing.signature.localeCompare(b.enclosing.signature))) { + if (!site.cs) continue; + const candidates = (byName.get(site.cs.method_name) ?? []).filter((s) => s !== site.enclosing.signature); + // Over-cap names (get/set/toString-class fan) are skipped outright, not truncated — a partial + // arbitrary subset would be neither sound-leaning nor deterministic in meaning. + if (!candidates.length || candidates.length > CHA_FAN_LIMIT) continue; + for (const target of candidates) { + addEdge(site.enclosing.signature, target); + t5++; + } + } + + const sortedEdges = [...edges.values()].sort((a, b) => a.source.localeCompare(b.source) || a.target.localeCompare(b.target)); + log.info(`call graph (defuse): ${sortedEdges.length} edges — t1=${t1} chase, t2=${t2} decorator, t3=${t3} callback, t4=${t4} votes, t4c=${t4c} ctor-field, t5=${t5} cha`); + return { + result: { edges: sortedEdges, external_symbols, synthesized_callables: {} }, + resolutions, + }; +} diff --git a/src/semantic_analysis/index.ts b/src/semantic_analysis/index.ts index 76df475..edc5d5b 100644 --- a/src/semantic_analysis/index.ts +++ b/src/semantic_analysis/index.ts @@ -1,5 +1,4 @@ -// Call-graph construction: the tsc (ts-morph checker) resolver graph + RTA. +// Call-graph construction: the tsc (ts-morph checker) resolver graph + RTA + the defuse linker. export * from "./callGraph"; -// The provider seam (union | tsc | jelly) + the Jelly backend. export * from "./provider"; -export * from "./jellyProvider"; +export * from "./defuseLinker"; diff --git a/src/semantic_analysis/jellyProvider.ts b/src/semantic_analysis/jellyProvider.ts deleted file mode 100644 index d7f6af7..0000000 --- a/src/semantic_analysis/jellyProvider.ts +++ /dev/null @@ -1,397 +0,0 @@ -/** - * Jelly call-graph provider. Shells out to `@cs-au-dk/jelly` (CLI/JSON only — no library API), - * then maps each Jelly function node back onto a symbol-table signature by source span: - * - * jelly id "fileIdx:sl:sc:el:ec" -> files[fileIdx] + (sl,sc) -> ts-morph node -> signature - * - * Named declarations reuse the existing canonicalizer (computeSignatureForDecl); anonymous inline - * callbacks — which the canonicalizer returns null for — get a SYNTHESIZED signature of the form - * `:`, mirroring how Jelly itself identifies - * anonymous functions purely by location. Jelly's columns are 1-based (it exports column+1), which - * lines up with ts-morph's 1-based columns, so (file,startLine,startColumn) is a direct join key. - * - * This provider is read-only w.r.t. the symbol table: it emits edges over its own node universe - * (real signatures ∪ synthesized) for diffing. Materializing synthesized callables into the symbol - * table is a later step, only needed when jelly is promoted to authoritative. - * - * Whole-program, scoped to declared deps: Jelly follows imports into node_modules, but we exclude - * every installed package NOT listed in the project's package.json `dependencies`. This keeps the - * directly-used library surface (the deps that actually matter for edge resolution) while cutting - * transitive bloat — critically `@ts-morph/common`, which bundles the entire ~8.7MB TypeScript - * compiler and makes unbounded whole-program analysis OOM. - * - * Tier 2 — dependency functions are materialized as external symbols so edges crossing the - * first-party↔library boundary are KEPT in both directions, tagged `ts.external`/`ts.module` (like - * the tsc phantom mechanism). Keying differs by direction because Jelly carries no function names: - * • first-party → dep: keyed `module.member` via the first-party call site (resolvePhantom), - * matching the tsc phantom keys so the two providers' external symbols are comparable. - * • dep → first-party: no first-party call site to read, so keyed by package + location. This is - * the entrypoint signal — a framework invoking your handler. - * Only dep→dep edges (neither endpoint first-party) are dropped. - */ -import { execFileSync } from "node:child_process"; -import * as fs from "node:fs"; -import { createRequire } from "node:module"; -import * as os from "node:os"; -import * as path from "node:path"; -import { Node, type SourceFile } from "ts-morph"; -import { - CALL_DEP, - computeSignatureForDecl, - fileKeyOf, - type TSCallEdge, - type TSExternalSymbol, - type TSSynthesizedCallable, -} from "../schema"; -import type { CallGraphResult } from "./callGraph"; -import { type ExternalIndex, buildExternalIndex, resolvePhantom } from "./phantoms"; -import type { CallGraphContext, CallGraphProvider } from "./provider"; - -const requireFrom = createRequire(import.meta.url); - -interface JellyJson { - files: string[]; - functions: Record; // id -> "fileIdx:startLine:startCol:endLine:endCol" - fun2fun: [number, number][]; // [callerId, calleeId] - call2fun: [number, number][]; // [callSiteId, calleeId] - calls: Record; // callSiteId -> "fileIdx:sl:sc:el:ec" (span in the CALLER file) -} - -/** Locate Jelly's entry script: explicit override, else the installed package. */ -function resolveJellyMain(): string { - if (process.env.JELLY_BIN) return process.env.JELLY_BIN; - try { - return requireFrom.resolve("@cs-au-dk/jelly/lib/main.js"); - } catch { - throw new Error("@cs-au-dk/jelly not installed and JELLY_BIN unset"); - } -} - -/** The project's declared runtime dependencies — the packages we want Jelly to descend into. */ -function declaredDeps(root: string): Set { - try { - const pkg = JSON.parse(fs.readFileSync(path.join(root, "package.json"), "utf8")) as { - dependencies?: Record; - }; - return new Set(Object.keys(pkg.dependencies ?? {})); - } catch { - return new Set(); - } -} - -/** Every installed package name under node_modules (descending one level into @scope dirs). */ -function installedPackages(root: string): string[] { - const nm = path.join(root, "node_modules"); - let top: string[]; - try { - top = fs.readdirSync(nm); - } catch { - return []; - } - const out: string[] = []; - for (const e of top) { - if (e.startsWith(".")) continue; - if (e.startsWith("@")) { - try { - for (const sub of fs.readdirSync(path.join(nm, e))) if (!sub.startsWith(".")) out.push(`${e}/${sub}`); - } catch { - /* unreadable scope dir */ - } - } else { - out.push(e); - } - } - return out; -} - -/** Installed packages NOT declared as dependencies — excluded so whole-program stays tractable. */ -function excludedPackages(root: string): string[] { - const keep = declaredDeps(root); - return installedPackages(root).filter((p) => !keep.has(p)); -} - -function runJelly(ctx: CallGraphContext, entryFiles: string[]): JellyJson { - const out = path.join(os.tmpdir(), `cants-jelly-${process.pid}.json`); - const excluded = excludedPackages(ctx.root); - ctx.log.debug(`call graph (jelly): excluding ${excluded.length} non-declared packages from whole-program scope`); - // Whole-program over first-party + declared deps only (no --ignore-dependencies, but exclude the - // rest). `--` terminates the variadic --exclude-packages list before the positional entry files. - const jellyArgs = ["-j", out]; - if (excluded.length) jellyArgs.push("--exclude-packages", ...excluded, "--"); - jellyArgs.push(...entryFiles); - - // Two launch modes. Compiled single-binary (CANTS_SELF_JELLY set by src/main.ts): re-exec THIS - // executable with the hidden `__jelly` subcommand — the Jelly CLI is bundled in, so no external - // `node` or node_modules is needed. Dev/source or explicit JELLY_BIN override: shell out to - // `node @cs-au-dk/jelly/lib/main.js` as before. - const self = process.env.CANTS_SELF_JELLY && !process.env.JELLY_BIN ? process.env.CANTS_SELF_JELLY : null; - const cmd = self ?? "node"; - const args = self ? ["__jelly", ...jellyArgs] : [resolveJellyMain(), ...jellyArgs]; - try { - execFileSync(cmd, args, { - cwd: ctx.root, - stdio: ["ignore", "ignore", "ignore"], - maxBuffer: 256 * 1024 * 1024, - timeout: 600_000, - env: { ...process.env, NODE_OPTIONS: `${process.env.NODE_OPTIONS ?? ""} --max-old-space-size=8192`.trim() }, - }); - return JSON.parse(fs.readFileSync(out, "utf8")) as JellyJson; - } finally { - try { - fs.rmSync(out, { force: true }); - } catch { - /* best-effort cleanup */ - } - } -} - -/** getDescendantAtPos lands on the token at the span start; climb to the enclosing function node. */ -function climbToFunctionLike(node: Node | undefined): Node | undefined { - let n = node; - while ( - n && - !( - Node.isArrowFunction(n) || - Node.isFunctionExpression(n) || - Node.isFunctionDeclaration(n) || - Node.isMethodDeclaration(n) || - Node.isConstructorDeclaration(n) || - Node.isGetAccessorDeclaration(n) || - Node.isSetAccessorDeclaration(n) - ) - ) { - n = n.getParent(); - } - return n; -} - -/** Climb from the token at a call-site span to the enclosing call/new expression. */ -function climbToCallExpr(node: Node | undefined): Node | undefined { - let n = node; - while (n && !(Node.isCallExpression(n) || Node.isNewExpression(n))) n = n.getParent(); - return n; -} - -/** Synthetic signature for an anonymous callback: nearest signed enclosing scope + location suffix. */ -function synthesize(fnNode: Node, root: string): string { - let host: Node | undefined = fnNode.getParent(); - while (host && computeSignatureForDecl(host, root) === null) host = host.getParent(); - const hostSig = host ? computeSignatureForDecl(host, root) : null; - const { line, column } = fnNode.getSourceFile().getLineAndColumnAtPos(fnNode.getStart()); - return `${hostSig ?? ""}:<${line}:${column}>`; -} - -/** - * If a Jelly file path lives under node_modules, split it into the owning package name and the - * path within that package. Uses the LAST `node_modules/` so nested deps resolve to the innermost - * package, and handles `@scope/name`. - */ -function depPackage(rel: string): { pkg: string; inPkg: string } | null { - const marker = "node_modules/"; - const idx = rel.lastIndexOf(marker); - if (idx < 0) return null; - const parts = rel.slice(idx + marker.length).split("/"); - if (parts[0].startsWith("@")) { - if (parts.length < 2) return null; - return { pkg: `${parts[0]}/${parts[1]}`, inPkg: parts.slice(2).join("/") }; - } - return { pkg: parts[0], inPkg: parts.slice(1).join("/") }; -} - -/** - * Map a function node to its signature. A `const foo = () => …` arrow is named by its - * VariableDeclaration in the symbol table, so normalize to that parent before deciding - * real-vs-synthesized — otherwise every named const-arrow would wrongly synthesize. - */ -function signatureFor(fn: Node, root: string): { sig: string; synth: boolean } { - const parent = fn.getParent(); - const decl = parent && Node.isVariableDeclaration(parent) ? parent : fn; - const real = computeSignatureForDecl(decl, root); - if (real) return { sig: real, synth: false }; - return { sig: synthesize(fn, root), synth: true }; -} - -export const jellyProvider: CallGraphProvider = { - name: "jelly", - build(ctx): CallGraphResult { - const entryFiles = ctx.project - .getSourceFiles() - .map((sf) => sf.getFilePath() as string) - .filter((fp) => !fp.includes("/node_modules/") && !fp.endsWith(".d.ts")) - .map((fp) => path.relative(ctx.root, fp)) - .filter((rel) => rel.length > 0 && !rel.startsWith("..")); - - if (entryFiles.length === 0) { - ctx.log.info("call graph (jelly): no first-party source files to analyze"); - return { edges: [], external_symbols: {}, synthesized_callables: {} }; - } - - const cg = runJelly(ctx, entryFiles); - - // Phase 1: classify each Jelly function. First-party functions round-trip to a ts-morph node and - // reuse the canonicalizer (real or synthesized). Dependency functions are recorded by package + - // location; an external symbol is minted for them lazily, when an edge reveals how they connect. - const id2sig = new Map(); // first-party id -> signature - const firstPartyIds = new Set(); - const depMeta = new Map(); - let synthesized = 0; - let unresolved = 0; - - // Anonymous callbacks get a synthesized signature with no symbol-table node; remember their - // location so the projection can materialize a node and the edge won't dangle (issue #13). - const synthesizedCallables: Record = {}; - const recordIfSynth = (fn: Node, sig: string, synth: boolean): void => { - if (!synth || synthesizedCallables[sig]) return; - const { line, column } = fn.getSourceFile().getLineAndColumnAtPos(fn.getStart()); - synthesizedCallables[sig] = { - name: "", - path: fileKeyOf(fn.getSourceFile().getFilePath(), ctx.root).fileKey, - start_line: line, - start_column: column, - }; - }; - for (const [id, loc] of Object.entries(cg.functions)) { - const [fileIdx, sl, sc] = loc.split(":").map(Number); - const rel = cg.files[fileIdx]; - if (rel === undefined) { - unresolved++; - continue; - } - const dep = depPackage(rel); - if (dep) { - depMeta.set(id, { pkg: dep.pkg, inPkg: dep.inPkg, sl, sc }); - continue; - } - const sf = ctx.project.getSourceFile(path.resolve(ctx.root, rel)); - if (!sf) { - unresolved++; - continue; - } - let offset: number; - try { - offset = sf.compilerNode.getPositionOfLineAndCharacter(sl - 1, sc - 1); - } catch { - unresolved++; - continue; - } - const fn = climbToFunctionLike(sf.getDescendantAtPos(offset)); - if (!fn) { - unresolved++; // module-level node and other non-function spans - continue; - } - const { sig, synth } = signatureFor(fn, ctx.root); - id2sig.set(id, sig); - firstPartyIds.add(id); - if (synth) synthesized++; - recordIfSynth(fn, sig, synth); - } - - const external_symbols: Record = {}; - const edges = new Map(); - let boundary = 0; - let dropped = 0; - const addEdge = (source: string, target: string, tags: Record): void => { - const k = `${source} ${target}`; - const ex = edges.get(k); - if (ex) ex.weight++; - else edges.set(k, { source, target, type: CALL_DEP, weight: 1, provenance: ["jelly"], tags }); - }; - // Location-keyed fallback signature for a dep function — used when no call-site member name is - // available (the dep→first-party direction, where the call site is inside the library). - const depLocSig = (d: { pkg: string; inPkg: string; sl: number; sc: number }): string => { - const sig = `${d.pkg}:${d.inPkg}:<${d.sl}:${d.sc}>`; - if (!external_symbols[sig]) external_symbols[sig] = { name: `${d.inPkg}:${d.sl}:${d.sc}`, module: d.pkg }; - return sig; - }; - // Per-file import/require index, for naming a library member at a first-party call site. - const extIndexCache = new Map(); - const extIndexFor = (sf: SourceFile): ExternalIndex => { - const key = sf.getFilePath(); - let idx = extIndexCache.get(key); - if (!idx) { - idx = buildExternalIndex(sf as unknown as Node); - extIndexCache.set(key, idx); - } - return idx; - }; - - // Phase 2a — first-party → dependency, NAMED via the call site. call2fun maps call-site → callee; - // we resolve the site in first-party source to (caller function, library member) and key the - // external symbol as `module.member`, matching the tsc phantom path so the two are comparable. - for (const [callId, calleeId] of cg.call2fun) { - const dep = depMeta.get(String(calleeId)); - if (!dep) continue; // callee is first-party (via fun2fun) or unresolved - const cloc = cg.calls[String(callId)]; - if (!cloc) continue; - const [cFileIdx, csl, csc] = cloc.split(":").map(Number); - const crel = cg.files[cFileIdx]; - if (crel === undefined || depPackage(crel)) continue; // the call site must be first-party - const sf = ctx.project.getSourceFile(path.resolve(ctx.root, crel)); - if (!sf) continue; - let coff: number; - try { - coff = sf.compilerNode.getPositionOfLineAndCharacter(csl - 1, csc - 1); - } catch { - continue; - } - const callNode = climbToCallExpr(sf.getDescendantAtPos(coff)); - if (!callNode) continue; - const callerFn = climbToFunctionLike(callNode); - if (!callerFn) continue; // top-level call, no enclosing function - const { sig: callerSig, synth: callerSynth } = signatureFor(callerFn, ctx.root); - recordIfSynth(callerFn, callerSig, callerSynth); - const ph = resolvePhantom(callNode, extIndexFor(sf)); - let sig: string; - if (ph) { - sig = ph.signature; // module.member - if (!external_symbols[sig]) external_symbols[sig] = { name: ph.member, module: ph.module }; - } else { - sig = depLocSig(dep); // unresolved import — fall back to the location key - } - addEdge(callerSig, sig, { "ts.external": "true", "ts.module": external_symbols[sig].module }); - boundary++; - } - - // Phase 2b — fun2fun for first-party→first-party (internal) and dependency→first-party (the - // entrypoint signal: a library invoking your code). first-party→dep is named in 2a; dep→dep and - // edges with an unresolved endpoint are dropped. - for (const [callerId, calleeId] of cg.fun2fun) { - const cid = String(callerId); - const tid = String(calleeId); - const srcFP = firstPartyIds.has(cid); - const tgtFP = firstPartyIds.has(tid); - if (srcFP && tgtFP) { - addEdge(id2sig.get(cid)!, id2sig.get(tid)!, {}); - continue; - } - if (!srcFP && tgtFP) { - const dep = depMeta.get(cid); - if (!dep) { - dropped++; // unresolved caller - continue; - } - addEdge(depLocSig(dep), id2sig.get(tid)!, { "ts.external": "true", "ts.module": dep.pkg }); - boundary++; - continue; - } - if (!(srcFP && !tgtFP)) dropped++; // dep→dep / unresolved (first-party→dep is counted in 2a) - } - - // Keep only synthesized callables that an edge actually references — no orphan nodes. - const referenced = new Set(); - for (const e of edges.values()) { - referenced.add(e.source); - referenced.add(e.target); - } - const synthesized_callables: Record = {}; - for (const [sig, sc] of Object.entries(synthesizedCallables)) if (referenced.has(sig)) synthesized_callables[sig] = sc; - - ctx.log.info( - `call graph (jelly): ${Object.keys(cg.functions).length} jelly funcs, ${firstPartyIds.size} first-party ` + - `(${synthesized} synthesized, ${Object.keys(synthesized_callables).length} materialized), ` + - `${Object.keys(external_symbols).length} external symbols, ${unresolved} unresolved, ` + - `${edges.size} edges (${boundary} library-boundary), ${dropped} dropped`, - ); - return { edges: [...edges.values()], external_symbols, synthesized_callables }; - }, -}; diff --git a/src/semantic_analysis/provider.ts b/src/semantic_analysis/provider.ts index 4caf5cc..ec76134 100644 --- a/src/semantic_analysis/provider.ts +++ b/src/semantic_analysis/provider.ts @@ -1,20 +1,16 @@ /** - * Call-graph provider seam. The orchestrator builds the graph through a CallGraphProvider so the - * backend is swappable: - * • `union` (default) — run tsc + jelly and emit the MERGED edge/node set (tsc ∪ jelly), tagged - * by `provenance` so consumers can still tell the two apart. - * • `tsc` — the always-on ts-morph resolver only (the explicit `--tsc-only` opt-out). - * • `jelly` — the cs-au-dk flow-based analyzer only. - * `both` is a deprecated alias of `union`: it used to run each and log a diff while emitting tsc - * only, which silently discarded every jelly edge and external symbol (see issue #11). + * Call-graph build seam. One backend: the tsc (ts-morph checker) resolver. `tscProvider` stays an + * object (rather than a bare function) so tests can spy on the build being skipped at -a 1. + * `mergeCallGraphs` merges edge/node sets by (source, target) with provenance union — the defuse + * linker's edges overlay the tsc base through it (an edge found by both carries + * `["defuse", "tsc"]` after the wire sort). */ import type { Project } from "ts-morph"; import type { TSExternalSymbol, TSModule } from "../schema"; import type { Logger } from "../utils"; import { buildCallGraph, type CallGraphResult } from "./callGraph"; -import { jellyProvider } from "./jellyProvider"; -/** Everything a provider needs to produce a call graph over the analyzed project. */ +/** Everything the builder needs to produce a call graph over the analyzed project. */ export interface CallGraphContext { project: Project; symbol_table: Record; @@ -32,7 +28,7 @@ export interface CallGraphProvider { build(ctx: CallGraphContext): CallGraphResult; } -/** The always-available backend — wraps the existing tsc resolver with zero behavior change. */ +/** The one backend — the ts-morph checker resolver (+ RTA + phantoms). */ export const tscProvider: CallGraphProvider = { name: "tsc", build: (ctx) => buildCallGraph(ctx.project, ctx.symbol_table, ctx.root, ctx.log, ctx.phantoms, ctx.only), @@ -41,11 +37,9 @@ export const tscProvider: CallGraphProvider = { /** * Merge two call-graph results into their union. Pure (no I/O) so it can be unit-tested directly. * - * Edges are keyed by `(source, target)`. A duplicate edge sums its weight, unions its `provenance` - * (so an edge found by both providers carries `["tsc", "jelly"]`), and merges its tags (base wins - * on conflict — the tsc edge is the authoritative one for the shared key). External symbols union - * by signature, base winning on conflict. `a` is treated as the base (tsc), `b` as the overlay - * (jelly). + * Edges are keyed by `(source, target)`. A duplicate edge sums its weight, unions its `provenance`, + * and merges its tags (base wins on conflict — the base edge is authoritative for the shared key). + * External symbols union by signature, base winning on conflict. */ export function mergeCallGraphs(a: CallGraphResult, b: CallGraphResult): CallGraphResult { const byKey = new Map(); @@ -67,54 +61,3 @@ export function mergeCallGraphs(a: CallGraphResult, b: CallGraphResult): CallGra const synthesized_callables = { ...b.synthesized_callables, ...a.synthesized_callables }; return { edges: [...byKey.values()], external_symbols, synthesized_callables }; } - -/** Count how the two edge sets overlap — preserves the old `both`-mode diagnostic. */ -function diffSummary(tsc: CallGraphResult, jelly: CallGraphResult): string { - const key = (e: { source: string; target: string }): string => `${e.source} ${e.target}`; - const tscKeys = new Set(tsc.edges.map(key)); - const jellyKeys = new Set(jelly.edges.map(key)); - let shared = 0; - for (const k of jellyKeys) if (tscKeys.has(k)) shared++; - return ( - `${shared} shared, ${tscKeys.size - shared} tsc-only, ${jellyKeys.size - shared} jelly-only ` + - `(tsc=${tscKeys.size}, jelly=${jellyKeys.size})` - ); -} - -/** - * Run tsc + jelly and emit their union. This is the default: jelly's edges and external symbols are - * PERSISTED (tagged `provenance: ["jelly"]`) instead of being discarded after a diff. If jelly - * fails, degrade to tsc only rather than failing the whole analysis. - */ -export const unionProvider: CallGraphProvider = { - name: "union", - build(ctx) { - const tsc = tscProvider.build(ctx); - let jelly: CallGraphResult; - try { - jelly = jellyProvider.build(ctx); - } catch (e) { - ctx.log.info(`call graph (union): jelly failed (${(e as Error).message}); emitting tsc only`); - return tsc; - } - ctx.log.info(`call graph diff: ${diffSummary(tsc, jelly)}`); - const merged = mergeCallGraphs(tsc, jelly); - ctx.log.info( - `call graph (union): ${merged.edges.length} edges, ` + - `${Object.keys(merged.external_symbols).length} external symbols`, - ); - return merged; - }, -}; - -export function selectProvider(name: string): CallGraphProvider { - switch (name) { - case "tsc": - return tscProvider; - case "jelly": - return jellyProvider; - default: - // "union" (the default) and the deprecated "both" alias both land here. - return unionProvider; - } -} diff --git a/src/syntactic_analysis/builders.ts b/src/syntactic_analysis/builders.ts index e87bdd0..402605a 100644 --- a/src/syntactic_analysis/builders.ts +++ b/src/syntactic_analysis/builders.ts @@ -16,6 +16,7 @@ import { type TSCallableKind, type TSCallsite, type TSComment, + type TSConfigAccess, type TSDecorator, type TSExport, type TSField, @@ -294,7 +295,10 @@ function buildAttributeField(prop: Node): TSField { function buildCallsite(call: Node): TSCallsite { const isNew = Node.isNewExpression(call); - const expr = (call as unknown as { getExpression: () => Node }).getExpression(); + // A tagged template (`inline\`url(...)\``) is a call whose callee is the tag. + const expr = Node.isTaggedTemplateExpression(call) + ? call.getTag() + : (call as unknown as { getExpression: () => Node }).getExpression(); let method_name = expr.getText(); let receiver_expr: string | undefined; let receiver_type: string | undefined; @@ -305,8 +309,9 @@ function buildCallsite(call: Node): TSCallsite { receiver_type = inferredType(expr.getExpression()); is_optional_chain = boolOf(expr, "hasQuestionDotToken"); } - const args = (call as unknown as { getArguments: () => Node[] }).getArguments(); + const args = (call as unknown as { getArguments?: () => Node[] }).getArguments?.() ?? []; // tagged templates have none const argument_types = args.map((a) => inferredType(a) ?? "unknown"); + const argsText = args.map((a) => a.getText()); const typeArgs = (call as unknown as { getTypeArguments?: () => Node[] }).getTypeArguments?.() ?? []; const type_arguments = typeArgs.map((t) => t.getText()); const return_type = inferredType(call); @@ -315,6 +320,7 @@ function buildCallsite(call: Node): TSCallsite { ...(receiver_expr != null ? { receiver_expr } : {}), ...(receiver_type != null ? { receiver_type } : {}), argument_types, + arguments: argsText, type_arguments, ...(return_type != null ? { return_type } : {}), is_constructor_call: isNew, @@ -346,10 +352,42 @@ function namedBoundary(node: Node): Boundary { interface BodyHandlers { onCall: (n: Node) => void; + onConfigAccess: (n: Node, root: string, key?: string) => void; onNestedCallable: (n: Node) => void; onNestedClass: (n: Node) => void; } +// Config-read recognition (#101 unit C1): the join target for the next task is a body-node +// ordinal id, so every env-root read must mint a `config_access` node — even a dynamic-key one +// (no `key`), which the dataflow tier resolves later. Missing one is a silently unjoinable read. +const ENV_ROOTS = new Set(["process.env", "import.meta.env", "Bun.env"]); + +/** `process.env.X` / `process.env["X"]` / `import.meta.env.X` — a read, not a call. */ +function envRootAccess(node: Node): { root: string; key?: string } | null { + if (Node.isPropertyAccessExpression(node)) { + const root = node.getExpression().getText(); + if (!ENV_ROOTS.has(root)) return null; + return { root, key: node.getName() }; + } + if (Node.isElementAccessExpression(node)) { + const root = node.getExpression().getText(); + if (!ENV_ROOTS.has(root)) return null; + const arg = node.getArgumentExpression(); + return { root, ...(arg && Node.isStringLiteral(arg) ? { key: arg.getLiteralValue() } : {}) }; + } + return null; +} + +/** Shared by every `onConfigAccess` handler, mirroring `buildCallsite`'s role for `onCall`. */ +function buildConfigAccess(n: Node, root: string, key?: string): TSConfigAccess { + return { + root, + ...(key !== undefined ? { key } : {}), + ...span(n), + bytes: [n.getStart(), n.getEnd()], + }; +} + function walkBody(body: Node, h: BodyHandlers): void { const visit = (node: Node): void => { const b = namedBoundary(node); @@ -362,16 +400,27 @@ function walkBody(body: Node, h: BodyHandlers): void { return; } if (b === "skip") return; - if (Node.isCallExpression(node) || Node.isNewExpression(node)) h.onCall(node); + const access = envRootAccess(node); + if (access) h.onConfigAccess(node, access.root, access.key); + // Destructuring (`const { PORT, HOST } = process.env`) is a VariableDeclaration whose + // initializer is an env root: mint ONE access per bound element (renamed bindings read the + // PROPERTY name, not the local one — `const { PORT: p } = process.env` reads "PORT"). + if (Node.isVariableDeclaration(node)) { + const init = node.getInitializer(); + const name = node.getNameNode(); + if (init && ENV_ROOTS.has(init.getText()) && Node.isObjectBindingPattern(name)) { + for (const el of name.getElements()) { + h.onConfigAccess(el, init.getText(), el.getPropertyNameNode()?.getText() ?? el.getName()); + } + } + } + if (Node.isCallExpression(node) || Node.isNewExpression(node) || Node.isTaggedTemplateExpression(node)) h.onCall(node); node.forEachChild(visit); }; - // A concise arrow body can *be* a callable (`() => () => x`). Visiting only the body's children - // would skip it and attribute its call sites to the callable that merely returns it. - if (namedBoundary(body) !== null) { - visit(body); - return; - } - body.forEachChild(visit); + // Visit the body NODE itself, not only its children: a concise arrow body can *be* a callable + // (`() => () => x`) — the boundary handler claims it — or *be* the call (`u => u.describe()`), + // which a children-only walk would silently skip (the call-site gap Jelly used to paper over). + visit(body); } function computeCC(body: Node): number { @@ -454,22 +503,28 @@ export function buildCallable( if (!sig) return null; const call_sites: TSCallsite[] = []; + const config_accesses: TSConfigAccess[] = []; const callables: Record = {}; const types: Record = {}; + const handlers = { + onCall: (n: Node) => call_sites.push(buildCallsite(n)), + onConfigAccess: (n: Node, root: string, key?: string) => config_accesses.push(buildConfigAccess(n, root, key)), + onNestedCallable: (n: Node) => { + const r = buildNestedCallable(n, root); + if (r) callables[memberKey(r.sig, r.callable.accessor_kind)] = r.callable; + }, + onNestedClass: (n: Node) => { + const r = buildClass(n, root); + types[memberKey(r.sig)] = r.cls; + }, + }; const body = (fnNode as unknown as { getBody?: () => Node | undefined }).getBody?.(); - if (body) { - walkBody(body, { - onCall: (n) => call_sites.push(buildCallsite(n)), - onNestedCallable: (n) => { - const r = buildNestedCallable(n, root); - if (r) callables[memberKey(r.sig, r.callable.accessor_kind)] = r.callable; - }, - onNestedClass: (n) => { - const r = buildClass(n, root); - types[memberKey(r.sig)] = r.cls; - }, - }); + if (body) walkBody(body, handlers); + // Parameter DEFAULT initializers execute in the callee's own activation (`f(x = mk())`), so + // their calls (and nested arrows) belong to this callable — they live outside getBody(). + for (const p of (fnNode as unknown as { getParameters?: () => Node[] }).getParameters?.() ?? []) { + walkBody(p, handlers); } const nameNode = sigNode as unknown as { getName?: () => string | undefined }; @@ -506,6 +561,7 @@ export function buildCallable( body: {}, abs_path: sigNode.getSourceFile().getFilePath(), call_sites, + config_accesses, }; if (Object.keys(callables).length) callable.callables = callables; if (Object.keys(types).length) callable.types = types; @@ -540,6 +596,7 @@ function implicitConstructor(classSig: string, filePath: string): { sig: string; body: {}, abs_path: filePath, call_sites: [], + config_accesses: [], }, }; } @@ -645,6 +702,31 @@ export function buildClass(cls: Node, root: string): { sig: string; cls: TSType } } + // Instance property INITIALIZERS execute in the constructor (`private readonly registry = + // Registry.as(...)`): their call sites AND config reads belong to the ctor (explicit or the + // synthesized implicit one) — same activation record, so `private readonly host = process.env + // .HOST` mints its config_access on the ctor, not module scope — and an initializer ARROW is a + // class-scoped positional anon callable — `contributorName` signs it `Class.`, so it + // lives in the class's callables{}, keeping signature ↔ containment aligned. Static initializers + // run at class-definition time and stay with the module-scope sweep (callGraph.ts). + { + const ctorCallable = callables[memberKey(constructorSignatureOf(sig))]; + for (const p of c.getProperties()) { + if (boolOf(p, "isStatic")) continue; + const init = (p as unknown as { getInitializer?: () => Node | undefined }).getInitializer?.(); + if (!init || !ctorCallable) continue; + walkBody(init, { + onCall: (n) => ctorCallable.call_sites.push(buildCallsite(n)), + onConfigAccess: (n, envRoot, key) => ctorCallable.config_accesses.push(buildConfigAccess(n, envRoot, key)), + onNestedCallable: (n) => { + const r = buildNestedCallable(n, root); + if (r) callables[memberKey(r.sig, r.callable.accessor_kind)] = r.callable; + }, + onNestedClass: () => {}, // class expression inside a property initializer — out of scope + }); + } + } + const base_classes: string[] = []; const implements_types: string[] = []; const ext = c.getExtends?.(); @@ -847,6 +929,7 @@ function buildStatemented(container: Node, root: string, varScope: "module" | "n // loops above already claimed, so nothing is collected twice. walkBody(container, { onCall: () => {}, + onConfigAccess: () => {}, // module/namespace scope — no callable to attribute the read to onNestedCallable: (n) => { if (!Node.isArrowFunction(n) && !Node.isFunctionExpression(n)) return; // already bucketed const r = buildCallable(n, n, Node.isArrowFunction(n) ? "arrow" : "function_expression", root); diff --git a/src/syntactic_analysis/discovery.ts b/src/syntactic_analysis/discovery.ts index 60c31b4..c0abae5 100644 --- a/src/syntactic_analysis/discovery.ts +++ b/src/syntactic_analysis/discovery.ts @@ -3,6 +3,13 @@ import * as path from "node:path"; import { relPosix } from "../utils"; const SOURCE_EXTS = new Set([".ts", ".tsx", ".mts", ".cts"]); +// JS sources are first-class too (the analyzer is a TS/JS analyzer; vendored .js like vscode's +// marked.js was invisible, #98). Sibling rules keep one module per prefix: +// - a .js beside a same-prefix REAL .ts source is compiled output → the .js is skipped +// (the compiler's own allowJs duplicate rule); +// - a .d.ts beside a same-prefix .js is hand-written declarations FOR that source → the .d.ts +// is skipped as a module (the checker still reads it from disk for importers' types). +const JS_EXTS = new Set([".js", ".jsx", ".mjs", ".cjs"]); export const SKIP_DIRS = new Set([ "node_modules", @@ -23,7 +30,7 @@ const TEST_DIRS = new Set(["__tests__", "__test__", "test", "tests", "spec", "__ /** Test-ness is judged on the path RELATIVE TO the project root, never the absolute path. */ function isTestFile(relKey: string): boolean { const base = path.basename(relKey); - if (/\.(test|spec)\.(ts|tsx|mts|cts)$/.test(base)) return true; + if (/\.(test|spec)\.(ts|tsx|mts|cts|js|jsx|mjs|cjs)$/.test(base)) return true; return relKey.split("/").some((p) => TEST_DIRS.has(p)); } @@ -32,9 +39,11 @@ export interface DiscoveredFile { fileKey: string; // project-relative POSIX path with extension } -/** Recursively discover .ts/.tsx sources under root, skipping vendored and (optionally) test trees. */ +/** Recursively discover TS/JS sources under root, skipping vendored and (optionally) test trees. */ export function discoverSourceFiles(root: string, skipTests: boolean): DiscoveredFile[] { - const out: DiscoveredFile[] = []; + const tsFiles: DiscoveredFile[] = []; + const jsCandidates: DiscoveredFile[] = []; + const realTsPrefixes = new Set(); // non-.d.ts TS sources only const walk = (dir: string): void => { let entries: fs.Dirent[]; try { @@ -50,14 +59,33 @@ export function discoverSourceFiles(root: string, skipTests: boolean): Discovere walk(abs); } else if (e.isFile()) { const ext = path.extname(e.name); - if (!SOURCE_EXTS.has(ext)) continue; + const isTs = SOURCE_EXTS.has(ext); + const isJs = JS_EXTS.has(ext); + if (!isTs && !isJs) continue; const fileKey = relPosix(root, abs); if (skipTests && isTestFile(fileKey)) continue; - out.push({ absPath: abs, fileKey }); + if (isTs) { + if (!fileKey.endsWith(".d.ts")) realTsPrefixes.add(fileKey.replace(/\.(tsx|ts|mts|cts)$/, "")); + tsFiles.push({ absPath: abs, fileKey }); + } else { + jsCandidates.push({ absPath: abs, fileKey }); + } } } }; walk(root); + const out: DiscoveredFile[] = []; + const jsPrefixes = new Set(); + for (const j of jsCandidates) { + const prefix = j.fileKey.replace(/\.(jsx|js|mjs|cjs)$/, ""); + if (realTsPrefixes.has(prefix)) continue; // compiled sibling of a TS source → skip + jsPrefixes.add(prefix); + out.push(j); + } + for (const t of tsFiles) { + if (t.fileKey.endsWith(".d.ts") && jsPrefixes.has(t.fileKey.replace(/\.d\.ts$/, ""))) continue; // decls FOR an analyzed .js + out.push(t); + } out.sort((a, b) => a.fileKey.localeCompare(b.fileKey)); return out; } diff --git a/src/syntactic_analysis/symbolTable.ts b/src/syntactic_analysis/symbolTable.ts index b2ae5b0..a0a2dc9 100644 --- a/src/syntactic_analysis/symbolTable.ts +++ b/src/syntactic_analysis/symbolTable.ts @@ -67,9 +67,7 @@ export function buildSymbolTable( const projectOf = new Map(); const programs: BuiltProgram[] = []; for (const s of specs) { - const project = s.configPath - ? new Project({ tsConfigFilePath: s.configPath, skipAddingFilesFromTsConfig: true }) - : new Project({ compilerOptions: defaultCompilerOptions() }); + const project = createProject(s.configPath); const files = assignment.get(s)!; const fileKeys = new Set(); for (const f of files) { @@ -112,6 +110,23 @@ export function buildSymbolTable( } /** The fallback compiler options when the target has no tsconfig (shared with graph workers). */ +/** + * THE ts-morph Project constructor — every program in the analyzer comes from here. + * + * `allowJs` is forced on over whatever the tsconfig says. Discovered `.js` files are added to the + * program that owns their PATH (JS source discovery, #98), regardless of that config's `include`, + * and a JS file sitting in a program whose options exclude it has no valid checker state: resolving + * any identifier inside it throws inside tsc instead of returning undefined. A TypeScript project's + * tsconfig normally leaves `allowJs` unset — which means false — so this is the ordinary case, not + * a corner one. It cost vscode its entire call graph (see schema/checker.ts). The override merges: + * `allowJs` is the only option it changes, every other tsconfig setting survives. + */ +export function createProject(configPath: string | null): Project { + return configPath + ? new Project({ tsConfigFilePath: configPath, skipAddingFilesFromTsConfig: true, compilerOptions: { allowJs: true } }) + : new Project({ compilerOptions: defaultCompilerOptions() }); +} + export function defaultCompilerOptions(): ts.CompilerOptions { return { target: ts.ScriptTarget.ES2022, diff --git a/test/anonymous-callables.test.ts b/test/anonymous-callables.test.ts index 076416e..20b2ca4 100644 --- a/test/anonymous-callables.test.ts +++ b/test/anonymous-callables.test.ts @@ -38,7 +38,6 @@ function options(level: number): AnalysisOptions { eager: true, noBuild: true, phantoms: true, - callGraphProvider: "tsc", cacheDir: null, verbosity: 0, } as unknown as AnalysisOptions; diff --git a/test/artifacts.test.ts b/test/artifacts.test.ts new file mode 100644 index 0000000..b408624 --- /dev/null +++ b/test/artifacts.test.ts @@ -0,0 +1,249 @@ +/** + * Repository-artifact layer gates (#101, recalibrated to python PR #160 / the ratified + * 2026-08-27 spec): neutral artifact ids, rules-matched capture, flat evidence-tagged + * dependencies (npm kinds incl. the coined `peer`), lock backfill with `lockfile` prov, + * unresolved imports with the @types type-only rule, level-invariance, determinism, and the + * neutral :Artifact/:Package Neo4j projection. + */ +import { describe, expect, test } from "bun:test"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { analyze } from "../src/core"; +import { project } from "../src/build/neo4j"; +import type { AnalysisOptions } from "../src/options"; + +const FIXTURE = path.resolve(import.meta.dir, "fixtures/artifacts-app"); + +function options(over: Partial = {}): AnalysisOptions { + return { + input: FIXTURE, output: null, emit: "json", appName: "artifacts-app", neo4jUri: null, + neo4jUser: "neo4j", neo4jPassword: "", neo4jDatabase: null, analysisLevel: 1, graphs: [], + graphFieldDepth: 3, jobs: 1, targetFiles: null, skipTests: true, eager: true, noBuild: true, + phantoms: true, cacheDir: fs.mkdtempSync(path.join(os.tmpdir(), "cants-art-")), verbosity: 0, + ...over, + } as AnalysisOptions; +} + +const r1 = await analyze(options()); +const root = r1.application.application; +const arts = root.artifacts; +const deps = root.dependencies; +const byName = new Map(deps.map((d) => [d.name, d])); + +describe("artifact inventory — rules-matched, neutral ids (#101/PR-160)", () => { + test("rules-matched files are present; source files (ts/js/tsx/jsx) are absent", () => { + for (const key of [ + "package.json", "package-lock.json", "packages/web/package.json", "packages/web/bun.lock", + "yarn.lock", ".env", "tsconfig.json", "Dockerfile", ".github/workflows/ci.yml", "README.md", "LICENSE", + ]) { + expect(arts[key], key).toBeDefined(); + } + expect(Object.keys(arts).some((k) => k.endsWith(".ts"))).toBe(false); + }); + + test("never drops: unmatched files are unknown-role, binaries are hash-only", () => { + const unknown = arts["notes.dat"]; + expect(unknown?.roles).toEqual(["unknown"]); + expect(unknown?.format).toBe("text"); + expect(unknown?.source.length).toBeGreaterThan(0); + + const bin = arts["logo.bin"]; + expect(bin?.format).toBe("binary"); + expect(bin?.roles).toEqual(["unknown"]); + expect(bin?.source).toBe(""); + expect(bin?.sha256.length).toBe(64); + expect(bin?.size_bytes).toBeGreaterThan(0); + }); + + test("ids are LANGUAGE-NEUTRAL (can://artifact//); dotfiles keep the dot", () => { + expect(arts[".env"]?.id).toBe("can://artifact/artifacts-app/.env"); + expect(arts["packages/web/package.json"]?.id).toBe("can://artifact/artifacts-app/packages/web/package.json"); + }); + + test("roles and formats from the rules table; roles union across matches", () => { + expect(arts["package.json"]?.roles).toEqual(["dependency-manifest", "tool-config"]); + expect(arts["package-lock.json"]?.roles).toEqual(["dependency-manifest"]); + expect(arts[".env"]?.roles).toEqual(["env"]); + expect(arts["tsconfig.json"]?.roles).toEqual(["tool-config"]); + expect(arts["Dockerfile"]?.roles).toEqual(["container-image"]); + expect(arts[".github/workflows/ci.yml"]?.roles).toEqual(["ci"]); + expect(arts["README.md"]?.roles).toEqual(["docs"]); + expect(arts["LICENSE"]?.roles).toEqual(["legal"]); + expect(arts["packages/web/bun.lock"]?.format).toBe("jsonc"); + }); + + test("verbatim source + sha256 + extraction status", () => { + expect(arts["package.json"]?.source).toContain('"express"'); + expect(arts["package.json"]?.sha256?.length).toBe(64); + expect(arts["package.json"]?.extraction).toBe("full"); + expect(arts["yarn.lock"]?.extraction).toBe("none"); // inventory-only lock format + expect(arts["README.md"]?.extraction).toBe("none"); + }); +}); + +describe("dependencies — flat, evidence-tagged (#101/PR-160)", () => { + test("npm sections map to the shared kind vocabulary, `peer` included; prov declared", () => { + expect(byName.get("express")?.kind).toBe("runtime"); + expect(byName.get("typescript")?.kind).toBe("dev"); + expect(byName.get("fsevents")?.kind).toBe("optional"); + expect(byName.get("react")?.kind).toBe("peer"); + for (const d of deps.filter((d) => d.direct)) { + expect(d.prov).toContain("declared"); + expect(d.extras).toEqual([]); + } + }); + + test("declared_in is the manifest's neutral artifact id (workspace member keeps its own)", () => { + expect(byName.get("express")?.declared_in).toBe("can://artifact/artifacts-app/package.json"); + expect(byName.get("lodash")?.declared_in).toBe("can://artifact/artifacts-app/packages/web/package.json"); + }); + + test("locks backfill locked_version on declared records only, prov gains lockfile", () => { + expect(byName.get("express")?.locked_version).toBe("4.19.2"); + expect(byName.get("express")?.prov).toEqual(["declared", "lockfile"]); + expect(byName.get("lodash")?.locked_version).toBe("4.17.21"); // sibling bun.lock (JSONC) + expect(byName.get("react")?.locked_version).toBeUndefined(); + expect(byName.has("transitive-shadow")).toBe(false); // nested lock entries ignored + }); + + test("lock-only packages become direct:false records attributed to the lock", () => { + const t = deps.find((d) => d.name === "lockonly-transitive"); + expect(t).toBeDefined(); + expect(t?.direct).toBe(false); + expect(t?.kind).toBe("runtime"); + expect(t?.prov).toEqual(["lockfile"]); + expect(t?.locked_version).toBe("1.0.0"); + expect(t?.declared_in).toBe("can://artifact/artifacts-app/package-lock.json"); + // declared packages stay direct + expect(byName.get("express")?.direct).toBe(true); + // nested shadow entries are NOT records + expect(deps.some((d) => d.name === "transitive-shadow")).toBe(false); + }); + + test("provides_imports: the name itself; @types/x also provides x", () => { + expect(byName.get("express")?.provides_imports).toEqual(["express"]); + expect(byName.get("@types/typed-only-pkg")?.provides_imports).toEqual(["@types/typed-only-pkg", "typed-only-pkg"]); + }); +}); + +describe("unresolved imports — the hygiene signal (#101/PR-160)", () => { + test("an undeclared VALUE import surfaces; declared and type-only-via-@types do not", () => { + const mods = root.unresolved_imports.map((u) => u.module); + expect(mods).toContain("left-pad"); // imported, never declared + expect(mods).not.toContain("express"); // declared runtime + expect(mods).not.toContain("typed-only-pkg"); // import type + @types declared → satisfied + expect(mods).not.toContain("node:fs"); // builtin + }); + + test("--resolve-installed binds via node_modules metadata (prov installed-metadata)", async () => { + // The probe reads a real path, so plant the install the fixture is meant to have. + // node_modules is gitignored everywhere, so it cannot ship with the fixture. + const pkgDir = path.join(FIXTURE, "node_modules", "left-pad"); + fs.mkdirSync(pkgDir, { recursive: true }); + fs.writeFileSync(path.join(pkgDir, "package.json"), JSON.stringify({ name: "left-pad", version: "1.3.0" })); + + const r = await analyze(options({ resolveInstalled: true })); + const u = r.application.application.unresolved_imports.find((x) => x.module === "left-pad"); + expect(u?.bound_to).toBe("left-pad"); + expect(u?.prov).toEqual(["installed-metadata"]); + + // Clean up + fs.rmSync(path.join(FIXTURE, "node_modules"), { recursive: true, force: true }); + }); +}); + +describe("level-invariance + determinism (#101)", () => { + test("the three sections are identical at -a 1 and -a 4", async () => { + const r4 = await analyze(options({ analysisLevel: 4, graphs: ["cfg", "dfg", "pdg", "sdg"] })); + expect(r4.application.application.artifacts).toEqual(arts); + expect(r4.application.application.dependencies).toEqual(deps); + expect(r4.application.application.unresolved_imports).toEqual(root.unresolved_imports); + }); + + test("two consecutive default runs are byte-identical", async () => { + const a = JSON.stringify((await analyze(options())).application); + const b = JSON.stringify((await analyze(options())).application); + expect(a).toBe(b); + }); + + test("text capture: on by default, truncates under the cap, hash stays full-file", async () => { + const full = (await analyze(options())).application.application.artifacts["README.md"]; + expect(full?.source.length).toBeGreaterThan(0); + expect(full?.text_truncated).toBe(false); + + const capped = (await analyze(options({ artifactTextMaxBytes: 8 }))).application.application.artifacts["README.md"]; + expect(capped?.text_truncated).toBe(true); + expect(capped!.source.length).toBeLessThanOrEqual(8); + expect(capped?.sha256).toBe(full?.sha256); // hash is of the FULL file + expect(capped?.size_bytes).toBe(full?.size_bytes); + }); + + test("--no-artifact-text drops source but keeps inventory AND extraction", async () => { + const a = (await analyze(options({ artifactText: false }))).application.application; + expect(a.artifacts["package.json"]?.source).toBe(""); + expect(a.dependencies.find((d) => d.name === "express")?.locked_version).toBe("4.19.2"); + }); + + test("text cap is byte-accurate on multi-byte UTF-8 (not character-count)", async () => { + // Create a test .md file with multi-byte chars to verify cap is byte-accurate, not char-count. + // "Hi 🎉": H=1 byte, i=1 byte, space=1 byte, emoji=4 bytes (UTF-8) = 7 bytes total, 5 UTF-16 code units. + const testFile = path.join(FIXTURE, "multi-byte-test.md"); + const fullContent = "Hi 🎉"; + fs.writeFileSync(testFile, fullContent, "utf8"); + + // Test 1: cap=6 bytes. Discriminates: old char-slicing counts UTF-16 units (text.length=5, which is NOT > 6), + // so text.slice(0,6) incorrectly returns full "Hi 🎉" (7 bytes, FAILS <= 6). New byte-slicing correctly + // compares Buffer.byteLength (7 > 6), truncates to 6 bytes (replacement char boundary), and PASSES <= 6. + const r1 = await analyze(options({ artifactTextMaxBytes: 6 })); + const art1 = r1.application.application.artifacts["multi-byte-test.md"]; + expect(art1?.text_truncated).toBe(true); + expect(Buffer.byteLength(art1!.source, "utf8")).toBeLessThanOrEqual(6); + expect(art1!.source).not.toBe(fullContent); + + // Test 2: cap=7 bytes (exact full byte length). Verifies > boundary (not >=): + // text_truncated must be false and source must be intact at exact cap. + const r2 = await analyze(options({ artifactTextMaxBytes: 7 })); + const art2 = r2.application.application.artifacts["multi-byte-test.md"]; + expect(art2?.text_truncated).toBe(false); + expect(art2!.source).toBe(fullContent); + expect(art2?.sha256).toBeDefined(); // hash is always full-file + expect(art2?.size_bytes).toBe(7); // size is full-file (7 bytes) + + // Clean up + fs.unlinkSync(testFile); + }); +}); + +describe("Neo4j projection — neutral :Artifact/:Package (#101)", () => { + const rows = project(r1.application); + + test("neutral nodes with purl ids; TS-prefixed claims into the ghost space", () => { + const art = rows.nodes.find((n) => n.value === "can://artifact/artifacts-app/package.json"); + expect(art?.labels).toEqual(["Artifact"]); + expect(art?.props["roles"]).toEqual(["dependency-manifest", "tool-config"]); + expect(art?.props["source"]).toBeUndefined(); // text stays off the graph + const pkg = rows.nodes.find((n) => n.value === "pkg:npm/react"); + expect(pkg?.labels).toEqual(["Package"]); + const scoped = rows.nodes.find((n) => n.value === "pkg:npm/%40scope/util"); + expect(scoped, "scoped purl").toBeDefined(); + expect(rows.edges.some((e) => e.type === "HAS_ARTIFACT" && e.to.value === art?.value)).toBe(true); + const decl = rows.edges.find((e) => e.type === "DECLARES_DEPENDENCY" && e.to.value === "pkg:npm/react"); + expect(decl?.props["kind"]).toBe("peer"); + // direct:true (declared in package.json) vs. direct:false (lock-only transitive) — the + // recipe `MATCH (:Artifact)-[d:DECLARES_DEPENDENCY {direct: true}]->(p:Package)` separates + // the declared SURFACE from the full lockfile-inclusive SUPPLY CHAIN. + expect(decl?.props["direct"]).toBe(true); + const transitive = rows.edges.find((e) => e.type === "DECLARES_DEPENDENCY" && e.to.value === "pkg:npm/lockonly-transitive"); + expect(transitive?.props["direct"]).toBe(false); + expect(rows.edges.some((e) => e.type === "LOCKS" && e.to.value === "pkg:npm/express")).toBe(true); + expect( + rows.edges.some( + (e) => e.type === "TS_PROVIDES" && e.from.value === "pkg:npm/express" && String(e.to.value).endsWith("/@external/express"), + ), + ).toBe(true); + expect( + rows.edges.some((e) => e.type === "TS_UNRESOLVED_IMPORT" && String(e.to.value).endsWith("/@external/left-pad")), + ).toBe(true); + }); +}); diff --git a/test/checker-guard.test.ts b/test/checker-guard.test.ts new file mode 100644 index 0000000..aaf4ee0 --- /dev/null +++ b/test/checker-guard.test.ts @@ -0,0 +1,79 @@ +/** + * A `.js` file that no tsconfig `include` covers must still resolve — and a node the checker + * genuinely cannot resolve must cost its own edges, never the run (#103). + * + * JS source discovery (#98) adds a discovered `.js` file to the program that owns its PATH, which + * is not the same thing as the program whose `include` names it. A TypeScript project's tsconfig + * normally leaves `allowJs` unset — false — so that file has no valid checker state, and resolving + * ANY identifier in it throws inside tsc rather than returning undefined. On vscode a single such + * mock (`extensions/microsoft-authentication/packageMocks/dpapi/dpapi.js`, `throw new Error(...)`) + * aborted the whole 9,351-module analysis at -a 2, -a 3 and -a 4; -a 1 survived only because it + * never resolves callees. The fixture here is that file, minimized. + * + * Two independent guarantees, tested apart: `createProject` forces `allowJs` on so the file + * resolves properly, and `symbolAt` degrades a checker throw to undefined so any REMAINING + * unresolvable node cannot take the run down. + */ +import { describe, expect, test } from "bun:test"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { analyze } from "../src/core"; +import { checkerFailures, resetCheckerFailures, symbolAt } from "../src/schema/checker"; +import type { AnalysisOptions } from "../src/options"; + +const FIXTURE = path.resolve(import.meta.dir, "fixtures/unresolvable-js-app"); +const ID = "can://typescript/unresolvable-js-app"; + +function options(over: Partial = {}): AnalysisOptions { + return { + input: FIXTURE, output: null, emit: "json", appName: "unresolvable-js-app", neo4jUri: null, + neo4jUser: "neo4j", neo4jPassword: "", neo4jDatabase: null, analysisLevel: 2, graphs: [], + graphFieldDepth: 3, jobs: 1, targetFiles: null, skipTests: true, eager: true, noBuild: true, + phantoms: true, cacheDir: fs.mkdtempSync(path.join(os.tmpdir(), "cants-guard-")), verbosity: 0, + ...over, + } as AnalysisOptions; +} + +describe("a .js file outside tsconfig's include still resolves (#103)", () => { + test("-a 2 completes, and the JS file's own declarations land in the symbol table", async () => { + const app = (await analyze(options())).application.application; + + const js = app.symbol_table["mocks/dpapi.js"]; + expect(js).toBeDefined(); + expect(Object.keys(js!.types ?? {})).toContain("defaultDpapi"); + const methods = Object.values((js!.types ?? {})["defaultDpapi"]?.callables ?? {}).map((c) => c.id); + expect(methods).toContain(`${ID}/mocks/dpapi.js/defaultDpapi/protectData`); + expect(methods).toContain(`${ID}/mocks/dpapi.js/defaultDpapi/unprotectData`); + }); + + test("its call edges resolve — with allowJs off, the checker threw before reaching them", async () => { + const app = (await analyze(options())).application.application; + const edges = app.call_graph.map((e) => `${e.src} -> ${e.dst}`); + + // Module-scope `new defaultDpapi()`, attributed to the MODULE. This edge does not merely go + // missing without the fix — resolving it is what threw. + expect(edges).toContain(`${ID}/mocks/dpapi.js -> ${ID}/mocks/dpapi.js/defaultDpapi/constructor`); + // The healthy TypeScript file is unaffected either way. + expect(edges).toContain(`${ID}/src/index.ts/run -> ${ID}/src/index.ts/greet`); + }); + + test("nothing is skipped any more — the checker resolves the file cleanly", async () => { + await analyze(options()); + expect(checkerFailures()).toBe(0); + }); +}); + +describe("a checker throw degrades to unresolved, it does not abort (#103)", () => { + test("symbolAt returns undefined and counts the failure", () => { + resetCheckerFailures(); + const exploding = { + getSymbol(): never { + throw new TypeError("undefined is not an object (evaluating 'getSymbolOfDeclaration(location).members')"); + }, + }; + + expect(symbolAt(exploding as never)).toBeUndefined(); + expect(checkerFailures()).toBe(1); + }); +}); diff --git a/test/cli-options.test.ts b/test/cli-options.test.ts new file mode 100644 index 0000000..176c565 --- /dev/null +++ b/test/cli-options.test.ts @@ -0,0 +1,23 @@ +import { describe, expect, test } from "bun:test"; +import { parseArgs } from "../src/cli"; +import { DEFAULT_ARTIFACT_TEXT_MAX_BYTES } from "../src/options"; + +const cap = (...extra: string[]): number => + parseArgs(["--input", ".", ...extra]).artifactTextMaxBytes; + +describe("--artifact-text-max-bytes", () => { + test("defaults when absent", () => { + expect(cap()).toBe(DEFAULT_ARTIFACT_TEXT_MAX_BYTES); + }); + + // A bad value must not silently disable truncation: NaN loses every `> cap` + // comparison, and Number("") is 0, which would empty every artifact's source. + test.each([["abc"], [""], [" "], ["-1"]])("falls back on %p", (bad) => { + expect(cap("--artifact-text-max-bytes", bad)).toBe(DEFAULT_ARTIFACT_TEXT_MAX_BYTES); + }); + + test("honors a valid value, 0 included", () => { + expect(cap("--artifact-text-max-bytes", "4096")).toBe(4096); + expect(cap("--artifact-text-max-bytes", "0")).toBe(0); + }); +}); diff --git a/test/config-keys.test.ts b/test/config-keys.test.ts new file mode 100644 index 0000000..4ea3732 --- /dev/null +++ b/test/config-keys.test.ts @@ -0,0 +1,433 @@ +import { describe, expect, test } from "bun:test"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { analyze } from "../src/core"; +import { parseEnvKeys, parseIniKeys, parseJsonc } from "../src/artifacts/configKeys"; +import { parseYamlKeys } from "../src/artifacts/yamlKeys"; +import { parseDockerfileEnv, yamlEnvKeys } from "../src/artifacts/deployEnv"; +import type { AnalysisOptions } from "../src/options"; + +const FIXTURE = path.resolve(import.meta.dir, "fixtures/artifacts-app"); +const opts = { + input: FIXTURE, output: null, emit: "json", appName: "artifacts-app", neo4jUri: null, + neo4jUser: "neo4j", neo4jPassword: "", neo4jDatabase: null, analysisLevel: 1, graphs: [], + graphFieldDepth: 3, jobs: 1, targetFiles: null, skipTests: true, eager: true, noBuild: true, + phantoms: true, cacheDir: fs.mkdtempSync(path.join(os.tmpdir(), "cants-keys-")), verbosity: 0, +} as AnalysisOptions; + +const arts = (await analyze(opts)).application.application.artifacts; +const keysOf = (p: string): Record => + Object.fromEntries((arts[p]?.config_keys ?? []).map((k) => [k.key, k])); + +describe("config keys — flat and JSON (#101 unit B)", () => { + test(".env keys land in the env namespace with refs and stripped quotes", () => { + const k = keysOf(".env"); + expect(k["PAYMENT_HOST"]?.value).toBe("https://pay.example.com"); + expect(k["PAYMENT_HOST"]?.namespace).toBe("env"); + expect(k["DB_URL"]?.references).toEqual(["env:PAYMENT_HOST"]); + expect(k["NODE_OPTIONS"]?.value).toBe("--max-old-space-size=4096"); + }); + + test("JSONC tsconfig parses despite comments and trailing commas", () => { + const k = keysOf("tsconfig.json"); + expect(k["compilerOptions.strict"]?.value).toBe(true); + expect(k["compilerOptions.target"]?.value).toBe("ES2022"); + expect(k["include.0"]?.value).toBe("src"); // arrays get numeric segments + expect(arts["tsconfig.json"]?.extraction).toBe("full"); + }); + + test("key ids chain off the artifact id", () => { + expect(keysOf(".env")["PAYMENT_HOST"]?.id).toBe("can://artifact/artifacts-app/.env@key/PAYMENT_HOST"); + }); + + test("a dependency manifest is never also a config file — manifests/lockfiles yield zero keys", () => { + expect(arts["package.json"]?.config_keys.length).toBe(0); + expect(arts["package-lock.json"]?.config_keys.length).toBe(0); + expect(arts["packages/web/bun.lock"]?.config_keys.length).toBe(0); + // fix round 2: the gate is checked on `roles` before the format switch, so it must hold for + // YAML too — pnpm-lock.yaml is format "yaml" with role dependency-manifest; previously only + // exercised by probe, not a committed test. + expect(arts["pnpm-lock.yaml"]?.config_keys.length).toBe(0); + // the gate is role-scoped, not blanket — plain config files in the same formats still extract + expect(arts["tsconfig.json"]?.config_keys.length).toBeGreaterThan(0); + expect(arts[".env"]?.config_keys.length).toBeGreaterThan(0); + }); + + test("a genuinely malformed config file falls back to partial — node survives intact", () => { + const art = arts["tsconfig.broken.json"]; + expect(art?.extraction).toBe("partial"); + expect(art?.config_keys).toEqual([]); // nothing salvaged, but no throw escaped + expect(art?.sha256.length).toBe(64); // inventory fields untouched by the failed extraction + expect(art?.size_bytes).toBeGreaterThan(0); + }); +}); + +// Direct unit tests of the parser (fix round 2): faster and clearer than round-tripping +// every case through a full analyze() run, since these are pure-function properties of +// parseJsonc itself, not of the artifact pipeline around it. +describe("parseJsonc — comments then trailing commas, each pass string-aware", () => { + test("a string containing ', }' survives byte-for-byte (not mistaken for a trailing comma)", () => { + expect(parseJsonc(`{"note": "hi, }"}`)).toEqual({ note: "hi, }" }); + }); + + test("a string containing ',]' survives byte-for-byte", () => { + expect(parseJsonc(`{"pattern": "a,]b"}`)).toEqual({ pattern: "a,]b" }); + }); + + test("a string containing a trailing-comma-shaped glob token survives byte-for-byte", () => { + expect(parseJsonc(`{"files": "dist/{cjs,}"}`)).toEqual({ files: "dist/{cjs,}" }); + }); + + test("a string containing '//' survives (not mistaken for a line comment)", () => { + expect(parseJsonc(`{"homepage": "https://example.com"}`)).toEqual({ homepage: "https://example.com" }); + }); + + test("a string containing '/*' survives (not mistaken for a block comment)", () => { + expect(parseJsonc(`{"glob": "a/*b"}`)).toEqual({ glob: "a/*b" }); + }); + + test("an escaped quote inside a string doesn't end it early, even with a real comment right after", () => { + // Built with a single-quoted JS string (not a template literal) so `\\"` in the SOURCE + // collapses to a literal `\"` (backslash + quote) at RUNTIME — i.e. an actual escaped + // quote inside the JSON text, the way it would read straight off disk. + const doc = '{"note": "he said \\"hi\\"", "x": 1} // trailing comment'; + expect(parseJsonc(doc)).toEqual({ note: 'he said "hi"', x: 1 }); + }); + + test("a genuine trailing comma is still stripped before both } and ]", () => { + expect(parseJsonc(`{ "a": 1, "b": [1, 2, 3,], }`)).toEqual({ a: 1, b: [1, 2, 3] }); + }); + + // Fix round 3: a trailing comma separated from its closing bracket by a comment (not just + // whitespace) is an everyday tsconfig shape — comments removed in pass 1 make it reachable by + // pass 2. A non-empty result here is exactly what flips the caller's `extraction` to "full" + // (src/artifacts/index.ts: `if (keys.length) { ... extraction = "full" }`) — already pinned + // end-to-end by the "JSONC tsconfig parses..." test above. + test("a trailing comma followed by a line comment before the closing brace still parses", () => { + expect(parseJsonc(`{"a": 1, // note\n}`)).toEqual({ a: 1 }); + }); + + test("a trailing comma followed by a block comment before the closing brace still parses", () => { + expect(parseJsonc(`{"a": 1, /* note */ }`)).toEqual({ a: 1 }); + }); + + test("a genuinely malformed document still throws (caller marks the artifact partial)", () => { + expect(() => parseJsonc(`{"a": "unterminated}`)).toThrow(); + }); +}); + +describe("config keys — YAML (#101 unit B)", () => { + test("nested maps and sequences flatten with numeric segments and real spans", () => { + const k = keysOf("docker-compose.yml"); + expect(k["services.web.image"]?.value).toBe("node:22"); + expect(k["services.web.ports.0"]?.value).toBe("3000:3000"); + expect(k["services.web.image"]?.namespace).toBe("yaml"); + expect(k["services.web.image"]?.span?.start[0]).toBeGreaterThan(0); + }); + + // Fix round 1: a parse error is a different fact from "nothing to extract" — parseYamlKeys now + // throws when the document has errors, so the caller's existing catch (src/artifacts/index.ts) + // records "partial" the same way it already does for malformed JSONC. Mirrors the + // "tsconfig.broken.json" test above, one namespace over. + test("a genuinely malformed YAML file falls back to partial — node survives intact", () => { + const art = arts["docker-compose.broken.yml"]; + expect(art?.extraction).toBe("partial"); + expect(art?.config_keys).toEqual([]); // nothing salvaged, but no throw escaped past the caller + expect(art?.sha256.length).toBe(64); // inventory fields untouched by the failed extraction + expect(art?.source).toBe("services:\n web:\n ports: [1, 2\n"); // full on-disk text, verbatim + }); + + // Fix round 1: `---`-separated multi-document streams (the standard shape of a Kubernetes + // manifest) now parse via parseAllDocuments instead of being silently truncated to the first + // document. Each document's keys get a zero-based index prefix; a single document (the fixture + // above) stays unprefixed — already re-confirmed by the untouched test above still passing + // unmodified (services.web.image, not 0.services.web.image). + test("a multi-document stream (Kubernetes-style) prefixes keys by document index, with real spans in the SECOND document", () => { + const k = keysOf("k8s/multi.yaml"); + expect(k["0.services.web.image"]?.value).toBe("node:22"); + expect(k["1.spec.containers.0.env.0.name"]?.value).toBe("PAYMENT_HOST"); + expect(k["1.spec.containers.0.env.0.value"]?.value).toBe("https://pay.example.com"); + // Line 11 of the COMBINED file is where the second document's env name actually sits. One + // LineCounter spans the whole parse; a counter reset per document would report a smaller, + // wrong line here instead. + expect(k["1.spec.containers.0.env.0.name"]?.span?.start).toEqual([11, 17]); + expect(arts["k8s/multi.yaml"]?.extraction).toBe("full"); + }); + + // Fix round 2: "any document has errors" must mean any — not just the first. This fixture's + // FIRST document is well-formed; only the second has a syntax error, exercised through the + // full index.ts pipeline (not a direct parseYamlKeys call) so the assertion is on what the + // artifact node actually ends up looking like, the same shape as the single-document + // "falls back to partial" test above. + test("an error in a NON-first document still fails the whole file — partial, node intact", () => { + const art = arts["k8s/multi-broken.yaml"]; + expect(art?.extraction).toBe("partial"); + expect(art?.config_keys).toEqual([]); // the well-formed first document is not partially salvaged + expect(art?.sha256.length).toBe(64); + expect(art?.source).toBe("services:\n web:\n image: node:22\n---\nspec:\n containers: [1, 2\n"); + }); +}); + +// Direct unit tests of the parser (fix round 2): anchors/aliases/merge-keys/cycles are properties +// of parseYamlKeys itself, not of the artifact pipeline around it — same rationale as the +// parseJsonc block above. +describe("parseYamlKeys — anchors, aliases, and merge keys (fix round 2)", () => { + test("an aliased map recurses into its own keys; a merge key splices into the CURRENT prefix", () => { + const text = "defaults: &defaults\n timeout: 30\nweb:\n <<: *defaults\nplain:\n val: *defaults\n"; + const keys = Object.fromEntries(parseYamlKeys(text).map((k) => [k.key, k])); + expect(keys["defaults.timeout"]?.value).toBe(30); + expect(keys["web.timeout"]?.value).toBe(30); // merge key: no ".<<." segment — spliced into "web" + expect(keys["plain.val.timeout"]?.value).toBe(30); // aliased map recurses exactly like an inline one + // the span belongs to the ANCHOR's own scalar node — every alias of it points back at that + // one source location, not at the `*name` reference site. + expect(keys["web.timeout"]?.span?.start).toEqual(keys["defaults.timeout"]?.span?.start); + expect(keys["plain.val.timeout"]?.span?.start).toEqual(keys["defaults.timeout"]?.span?.start); + }); + + test("a merge-key sequence (<<: [*a, *b]) splices every source into the current prefix", () => { + const text = "a: &a\n x: 1\nb: &b\n y: 2\nc:\n <<: [*a, *b]\n"; + const keys = Object.fromEntries(parseYamlKeys(text).map((k) => [k.key, k.value])); + expect(keys["a.x"]).toBe(1); + expect(keys["b.y"]).toBe(2); + expect(keys["c.x"]).toBe(1); // merged from *a + expect(keys["c.y"]).toBe(2); // merged from *b + }); + + test("a cyclic alias (a map aliasing itself) terminates via the existing depth cap instead of hanging, and still keys the reachable scalar", () => { + const text = "node: &node\n value: 1\n self: *node\n"; + const start = Date.now(); + const keys = parseYamlKeys(text); + expect(Date.now() - start).toBeLessThan(1000); // terminates — the bug this guards is an infinite loop + const byKey = Object.fromEntries(keys.map((k) => [k.key, k.value])); + expect(byKey["node.value"]).toBe(1); // the one genuine scalar is still reachable and correct + expect(keys.length).toBeLessThan(30); // bounded by the depth cap (~24), not unbounded + }); + + test("a dangling alias (no matching anchor) resolves to undefined and is skipped, not thrown", () => { + expect(parseYamlKeys("a: *nope\n")).toEqual([]); + }); + + // Fix round 3: explicit keys must beat merged ones of the same name regardless of document + // order, and the winning entry's span must be the EXPLICIT site, not the anchor's — otherwise a + // last-wins consumer reads the value YAML says is overridden, and the two entries collide on id + // (configKeyIdOf derives the id from the key string alone, so a genuine duplicate would too). + test("an explicit key beats a merged one of the same name — one entry, explicit value and span", () => { + const text = "defaults: &defaults\n timeout: 30\nweb:\n timeout: 60\n <<: *defaults\n"; + const keys = parseYamlKeys(text).filter((k) => k.key === "web.timeout"); + expect(keys.length).toBe(1); + expect(keys[0]?.value).toBe(60); + expect(keys[0]?.span?.start).toEqual([4, 12]); // line 4: the explicit " timeout: 60" line + }); + + test("the same override with << written BEFORE the explicit key — identical result, proving order-independence", () => { + const text = "defaults: &defaults\n timeout: 30\nweb:\n <<: *defaults\n timeout: 60\n"; + const keys = parseYamlKeys(text).filter((k) => k.key === "web.timeout"); + expect(keys.length).toBe(1); + expect(keys[0]?.value).toBe(60); + expect(keys[0]?.span?.start).toEqual([5, 12]); // explicit key moved to line 5 — span follows it + }); + + test("in a merge sequence, an earlier source wins over a later one when both define the same key", () => { + const text = "a: &a\n x: 1\nb: &b\n x: 2\nc:\n <<: [*a, *b]\n"; + const keys = parseYamlKeys(text).filter((k) => k.key === "c.x"); + expect(keys.length).toBe(1); + expect(keys[0]?.value).toBe(1); // *a, not *b + }); + + test("no duplicate dotted keys across the whole anchor/merge fixture", () => { + const text = "defaults: &defaults\n timeout: 30\nweb:\n <<: *defaults\nplain:\n val: *defaults\n"; + const keys = parseYamlKeys(text); + expect(new Set(keys.map((k) => k.key)).size).toBe(keys.length); + }); +}); + +describe("deployment-env namespaces (#101 unit D)", () => { + test("Dockerfile ENV mints bindable env keys; ARG stays non-bindable dockerfile", () => { + const k = keysOf("Dockerfile"); + expect(k["PAYMENT_HOST"]?.namespace).toBe("env"); + expect(k["PAYMENT_HOST"]?.value).toBe("https://pay.example.com"); + expect(k["FEATURE_FLAG"]?.namespace).toBe("env"); + expect(k["FEATURE_FLAG"]?.value).toBe("on"); // legacy `ENV K "V"` form, quotes stripped + expect(k["BUILD_ID"]?.namespace).toBe("dockerfile"); // build-time only, never joins a read + expect(k["BUILD_ID"]?.value).toBe("local"); + expect(arts["Dockerfile"]?.extraction).toBe("full"); + }); + + test("compose environment blocks mint env keys ALONGSIDE the structural yaml keys", () => { + const k = keysOf("docker-compose.yml"); + expect(k["services.web.environment.PAYMENT_HOST"]?.namespace).toBe("yaml"); // structural + const envKeys = (arts["docker-compose.yml"]?.config_keys ?? []).filter((x) => x.namespace === "env"); + expect(envKeys.map((x) => x.key).sort()).toEqual(["FEATURE_FLAG", "PAYMENT_HOST"]); + const byName = Object.fromEntries(envKeys.map((x) => [x.key, x])); + expect(byName["PAYMENT_HOST"]?.value).toBe("https://pay.example.com"); + }); + + // Controller ruling: yamlKeys.ts prefixes every key in a multi-document stream with its + // zero-based document index ("1.spec.containers.0.env.0.name"). A compose/k8s matcher anchored + // at "^services" that ignores this prefix would silently match nothing on a real + // Kubernetes-shaped multi-document file. k8s/multi.yaml is exactly that shape: doc 0 is a + // compose-shaped services: map, doc 1 is a Deployment with spec.containers[0].env[0]. + test("multi-document files (k8s/multi.yaml) still mint env keys despite the doc-index prefix", () => { + const envKeys = (arts["k8s/multi.yaml"]?.config_keys ?? []).filter((x) => x.namespace === "env"); + expect(envKeys.map((x) => x.key)).toEqual(["PAYMENT_HOST"]); + expect(envKeys[0]?.value).toBe("https://pay.example.com"); + expect(arts["k8s/multi.yaml"]?.extraction).toBe("full"); + }); + + // extractConfigKeys already refuses a dependency-manifest; deploymentEnvKeys must not + // independently re-open that door. pnpm-lock.yaml (format "yaml", role dependency-manifest) is + // the one existing fixture that exercises this for the yaml path (a Dockerfile can never carry + // that role, so there is nothing to check on the dockerfile side of the same gate). + test("a dependency-manifest yaml file (pnpm-lock.yaml) gains zero env keys — same gate as structural keys", () => { + expect(arts["pnpm-lock.yaml"]?.config_keys.filter((x) => x.namespace === "env")).toEqual([]); + }); + + // Fix round 1 (coordinator ruling): configKeyIdOf(artifactId, key) ignores namespace, so a bare + // name minted into TWO namespaces on the SAME artifact collided on `.id` before this round — + // confirmed to actually happen via `ARG VERSION` + `ENV VERSION=$VERSION`, a routine Dockerfile + // idiom (promote a build arg into a runtime env var under the same name). The `key` FIELD stays + // the bare name in both cases (env-namespace resolution still joins on a plain `key ===` match); + // only the id gets an internal prefix — python v1.3.0 parity, applied in assignIds.ts. + test("ARG VERSION + ENV VERSION=$VERSION mint the same bare key in two namespaces with DISTINCT ids", () => { + const keys = arts["Dockerfile"]?.config_keys ?? []; + const argVersion = keys.find((k) => k.namespace === "dockerfile" && k.key === "VERSION"); + const envVersion = keys.find((k) => k.namespace === "env" && k.key === "VERSION"); + expect(argVersion?.key).toBe("VERSION"); // key field stays bare — only the id is disambiguated + expect(envVersion?.key).toBe("VERSION"); + expect(argVersion?.id).toBe("can://artifact/artifacts-app/Dockerfile@key/arg.VERSION"); + expect(envVersion?.id).toBe("can://artifact/artifacts-app/Dockerfile@key/VERSION"); // ENV: unprefixed + expect(argVersion?.id).not.toBe(envVersion?.id); + }); + + // Same collision class, one namespace over: a yaml artifact's TOP-LEVEL leaf and its "env" + // dual-mint (compose/k8s) can share a bare name too — docker-compose.yml now carries both a + // root PAYMENT_HOST: leaf and services.web.environment.PAYMENT_HOST (which dual-mints an "env" + // key named bare "PAYMENT_HOST"). Only the dual-mint's id gets the `env.` prefix. + test("a yaml artifact's top-level leaf and its env dual-mint share a bare name but get DISTINCT ids", () => { + const keys = arts["docker-compose.yml"]?.config_keys ?? []; + const structural = keys.find((k) => k.namespace === "yaml" && k.key === "PAYMENT_HOST"); + const deployEnv = keys.find((k) => k.namespace === "env" && k.key === "PAYMENT_HOST"); + expect(structural?.value).toBe("https://root-level.example.com"); + expect(deployEnv?.value).toBe("https://pay.example.com"); + expect(structural?.id).toBe("can://artifact/artifacts-app/docker-compose.yml@key/PAYMENT_HOST"); + expect(deployEnv?.id).toBe("can://artifact/artifacts-app/docker-compose.yml@key/env.PAYMENT_HOST"); + expect(structural?.id).not.toBe(deployEnv?.id); + }); + + // The regression guard that would have caught the original bug: every config-key id across the + // WHOLE fixture app, unique — not just within the two cases spelled out above. + test("no two config keys anywhere in the fixture app share an id", () => { + const ids = Object.values(arts).flatMap((a) => a.config_keys.map((k) => k.id)); + expect(new Set(ids).size).toBe(ids.length); + }); + + // docker-compose.broken.yml / k8s/multi-broken.yaml falling back to "partial" with empty + // config_keys is already pinned above (config keys — YAML block); nothing to re-assert here + // beyond confirming this task didn't reach for deploy keys on a file that never parsed. +}); + +// Direct unit tests (fixtures don't cover every shape without risking the line/span assertions +// pinned on them elsewhere): compose list form, k8s valueFrom, and the doc-index prefix, each in +// isolation — same rationale as the parseJsonc/parseYamlKeys direct-unit-test blocks above. +describe("yamlEnvKeys — compose list form and k8s valueFrom (#101 unit D)", () => { + test("compose list-form environment entries (`- KEY=value`) mint bindable env keys too", () => { + const text = + "services:\n web:\n environment:\n - FEATURE_FLAG=on\n - PAYMENT_HOST=https://pay.example.com\n"; + const keys = Object.fromEntries(yamlEnvKeys(parseYamlKeys(text)).map((k) => [k.key, k])); + expect(keys["FEATURE_FLAG"]?.namespace).toBe("env"); + expect(keys["FEATURE_FLAG"]?.value).toBe("on"); + expect(keys["PAYMENT_HOST"]?.value).toBe("https://pay.example.com"); + // the numeric list index must never itself be minted as a variable name + expect(keys["0"]).toBeUndefined(); + expect(keys["1"]).toBeUndefined(); + }); + + // Fix round 2 (coordinator ruling, python v1.3.0 parity): a bare list entry ("- KEY", no "=") + // is compose's own syntax for "inherit this variable from the host environment" — a real + // bindable declaration, not dropped convenience — and must mint a valueless key, same shape as + // the k8s valueFrom case below. Distinguished from "- KEY=" (an "=" present, empty right side), + // which mints an EMPTY-STRING value. A name that isn't a valid env var name stays dropped either way. + test("a bare compose list entry (`- KEY`, no `=`) mints a valueless env key; `- KEY=` mints an empty string; junk is dropped", () => { + const text = + "services:\n web:\n environment:\n - PASSTHROUGH_KEY\n - EMPTY_KEY=\n - 9INVALID\n"; + const keys = Object.fromEntries(yamlEnvKeys(parseYamlKeys(text)).map((k) => [k.key, k])); + expect(keys["PASSTHROUGH_KEY"]?.namespace).toBe("env"); + expect(keys["PASSTHROUGH_KEY"]?.value).toBeUndefined(); // no "=" at all — absent, not "" + expect("value" in (keys["PASSTHROUGH_KEY"] ?? {})).toBe(false); // truly absent, not present-as-undefined + expect(keys["EMPTY_KEY"]?.value).toBe(""); // "=" present, empty right side — distinct from bare + expect(keys["9INVALID"]).toBeUndefined(); // not a valid env var name (leading digit) — dropped + }); + + test("a k8s env entry using valueFrom (no literal value) still mints the key, with no value", () => { + const text = [ + "spec:", + " containers:", + " - name: app", + " env:", + " - name: DB_PASSWORD", + " valueFrom:", + " secretKeyRef:", + " name: db-secret", + " key: password", + "", + ].join("\n"); + const keys = Object.fromEntries(yamlEnvKeys(parseYamlKeys(text)).map((k) => [k.key, k])); + expect(keys["DB_PASSWORD"]?.namespace).toBe("env"); + expect(keys["DB_PASSWORD"]?.value).toBeUndefined(); // no literal — not dropped, just valueless + }); + + test("the doc-index prefix from a multi-document stream is tolerated by both compose and k8s matchers", () => { + const text = + 'services:\n web:\n environment:\n DEBUG: "true"\n---\nspec:\n containers:\n - name: app\n env:\n - name: DEBUG\n value: "false"\n'; + const flat = parseYamlKeys(text); + expect(flat.some((k) => k.key.startsWith("0."))).toBe(true); // sanity: this IS a multi-doc stream + const keys = yamlEnvKeys(flat); + // same bare name "DEBUG" from two different documents/containers — first occurrence in + // flattened (document) order wins: exactly one TSConfigKey, not two colliding on id. + const debug = keys.filter((k) => k.key === "DEBUG"); + expect(debug.length).toBe(1); + expect(debug[0]?.value).toBe("true"); + }); +}); + +describe("parseDockerfileEnv — redefinition and dedup (#101 unit D)", () => { + test("a key redefined on a later ENV line keeps the LAST value — one key, not two colliding ids", () => { + const keys = parseDockerfileEnv("FROM node:22\nENV FOO=1\nENV FOO=2\n"); + const foo = keys.filter((k) => k.key === "FOO"); + expect(foo.length).toBe(1); + expect(foo[0]?.value).toBe("2"); + }); + + test("ARG and ENV of the same name land in different namespaces and both survive", () => { + const keys = parseDockerfileEnv("ARG VERSION=1.0\nENV VERSION=$VERSION\n"); + const byNs = Object.fromEntries(keys.map((k) => [k.namespace, k])); + expect(byNs["dockerfile"]?.key).toBe("VERSION"); + expect(byNs["env"]?.key).toBe("VERSION"); + expect(keys.length).toBe(2); + }); + + test("never throws, even on garbage input", () => { + expect(() => parseDockerfileEnv("ENV \nARG\n\0\0binary garbage\nENV =nope\n")).not.toThrow(); + }); +}); + +// Review fix: parseEnvKeys/parseIniKeys pushed one TSConfigKey per line, so a key set twice in +// one file emitted two records sharing an id — Neo4j's id-dedup collapsed them to one node while +// analysis.json kept both, so the two projections disagreed. Same last-wins fix as +// parseDockerfileEnv above, applied to the other two line-based parsers. +describe("parseEnvKeys / parseIniKeys — same-file dedup (review fix)", () => { + test("a key set twice in one .env file keeps the LAST value — one key, not two colliding ids", () => { + const keys = parseEnvKeys("FOO=1\nFOO=2\n"); + const foo = keys.filter((k) => k.key === "FOO"); + expect(foo.length).toBe(1); + expect(foo[0]?.value).toBe("2"); + }); + + test("a key set twice in one .ini section keeps the LAST value — one key, not two colliding ids", () => { + const keys = parseIniKeys("[web]\ntimeout=30\ntimeout=60\n", "ini"); + const timeout = keys.filter((k) => k.key === "web.timeout"); + expect(timeout.length).toBe(1); + expect(timeout[0]?.value).toBe("60"); + }); +}); diff --git a/test/config-use.test.ts b/test/config-use.test.ts new file mode 100644 index 0000000..2525316 --- /dev/null +++ b/test/config-use.test.ts @@ -0,0 +1,214 @@ +import { describe, expect, test } from "bun:test"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { project as neoProject } from "../src/build/neo4j"; +import { analyze } from "../src/core"; +import type { AnalysisOptions } from "../src/options"; + +const FIXTURE = path.resolve(import.meta.dir, "fixtures/artifacts-app"); +function options(level: number): AnalysisOptions { + return { + input: FIXTURE, output: null, emit: "json", appName: "artifacts-app", neo4jUri: null, + neo4jUser: "neo4j", neo4jPassword: "", neo4jDatabase: null, analysisLevel: level, + graphs: level >= 3 ? ["cfg", "dfg", "pdg", "sdg"] : [], graphFieldDepth: 3, jobs: 1, + targetFiles: null, skipTests: true, eager: true, noBuild: true, phantoms: true, + cacheDir: fs.mkdtempSync(path.join(os.tmpdir(), "cants-cu-")), verbosity: 0, + } as AnalysisOptions; +} + +const r1 = await analyze(options(1)); +const mod = r1.application.application.symbol_table["src/config.ts"]; + +describe("config_access body nodes (#101 unit C1)", () => { + test("member, element, and destructured env reads all mint nodes with keys", () => { + const nodesOf = (fn: string) => Object.values(mod?.functions[fn]?.body ?? {}).filter((b) => b.kind === "config_access"); + expect(nodesOf("readHost").map((n) => n.key)).toEqual(["PAYMENT_HOST"]); + expect(nodesOf("readFlag").map((n) => n.key)).toEqual(["FEATURE_FLAG"]); + expect(nodesOf("readDestructured").map((n) => n.key)).toEqual(["NODE_OPTIONS"]); + expect(nodesOf("readHost")[0]?.root).toBe("process.env"); + expect(nodesOf("readHost")[0]?.callee).toBeUndefined(); // a read is not a call + }); + + test("a dynamic key mints a node with no key", () => { + const n = Object.values(mod?.functions["readVia"]?.body ?? {}).filter((b) => b.kind === "config_access"); + expect(n.length).toBe(1); + expect(n[0]?.key).toBeUndefined(); + }); + + test("a property-initializer env read attributes to the constructor, not module scope", () => { + const ctor = mod?.types["Client"]?.callables?.["constructor"]; + const nodes = Object.values(ctor?.body ?? {}).filter((b) => b.kind === "config_access"); + expect(nodes.map((n) => n.key)).toEqual(["PAYMENT_HOST"]); + }); + + test("a call and a config read sharing a start position get distinct body keys", () => { + const body = mod?.functions["readList"]?.body ?? {}; + const call = Object.entries(body).find(([, b]) => b.kind === "call"); + const access = Object.entries(body).find(([, b]) => b.kind === "config_access"); + expect(call).toBeDefined(); + expect(access).toBeDefined(); + const [callKey] = call!; + const [accessKey, accessNode] = access!; + expect(accessNode.key).toBe("LIST"); + expect(accessKey).not.toBe(callKey); + expect(accessKey).toBe(`${callKey}/2`); + }); +}); + +const r2 = await analyze(options(2)); +const app2 = r2.application.application; +const useDsts = (fnFragment: string): string[] => + app2.config_uses.filter((u) => u.src.includes(fnFragment)).map((u) => u.dst).sort(); + +describe("config_use literal tier (#101 unit C3)", () => { + test("a literal env read joins every declaring ConfigKey", () => { + const dsts = useDsts("readHost"); + expect(dsts).toContain("can://artifact/artifacts-app/.env@key/PAYMENT_HOST"); + expect(dsts).toContain("can://artifact/artifacts-app/Dockerfile@key/PAYMENT_HOST"); + expect(app2.config_uses.every((u) => u.prov.includes("literal"))).toBe(true); + }); + + test("src is a global ordinal body-node id", () => { + const u = app2.config_uses.find((x) => x.src.includes("readHost")); + expect(u?.src).toMatch(/^can:\/\/typescript\/artifacts-app\/src\/config\.ts\/readHost@\d+:\d+$/); + }); + + test("a literal with no declared key is an undefined-key read, not an edge", () => { + const read = app2.config_reads.find((r) => r.key === "NOT_DECLARED_ANYWHERE"); + expect(read?.reason).toBe("undefined-key"); + expect(read?.prov).toEqual(["literal"]); + expect(app2.config_uses.some((u) => u.src.includes("readUndeclared"))).toBe(false); + }); + + test("a dynamic key is a non-literal read at L2", () => { + const read = app2.config_reads.find((r) => r.site.includes("readVia")); + expect(read?.reason).toBe("non-literal"); + expect(read?.key).toBeUndefined(); + }); + + test("Dockerfile ARG is never bindable", () => { + // Assert the real invariant (namespace, resolved from the TSConfigKey) rather than sniffing + // the id's "arg." prefix — a dockerfile-namespace key's id is ALWAYS "arg." (assignIds), + // so `dst.endsWith("@key/BUILD_ID")` can never be true regardless of what the rule tables do. + // `readBuildId` reads BUILD_ID, which ONLY Dockerfile ARG declares — a real read exercises + // this invariant instead of leaving it vacuously true on a fixture with nothing to bind. + const namespaceOf = new Map(); + for (const art of Object.values(app2.artifacts)) { + for (const ck of art.config_keys) namespaceOf.set(ck.id, ck.namespace); + } + expect(app2.config_uses.some((u) => namespaceOf.get(u.dst) === "dockerfile")).toBe(false); + + // Positive half: the read was SEEN and deliberately left unbound, not silently dropped. + const read = app2.config_reads.find((r) => r.site.includes("readBuildId")); + expect(read?.reason).toBe("undefined-key"); + expect(read?.key).toBe("BUILD_ID"); + }); + + test("a CALL rule resolves through the resolved external callee", () => { + const dsts = useDsts("readViaLibrary"); + expect(dsts.some((d) => d.endsWith("@key/PAYMENT_HOST"))).toBe(true); + const u = app2.config_uses.find((x) => x.src.includes("readViaLibrary")); + expect(u?.src).toMatch(/@\d+:\d+$/); // the CALL node's ordinal id + }); + + test("an interpolated template-literal key is a non-literal read (not a bogus undefined-key)", () => { + const read = app2.config_reads.find((r) => r.site.includes("readTemplateInterpolated")); + expect(read?.reason).toBe("non-literal"); + expect(read?.key).toBeUndefined(); + expect(app2.config_uses.some((u) => u.src.includes("readTemplateInterpolated"))).toBe(false); + }); + + test("a non-interpolated template literal still resolves like a quoted string", () => { + const dsts = useDsts("readTemplateLiteral"); + expect(dsts.some((d) => d.endsWith("@key/PAYMENT_HOST"))).toBe(true); + }); +}); + +const r3 = await analyze(options(3)); +const app3 = r3.application.application; + +describe("config_use dataflow tiers (#101 unit C3)", () => { + test("an indirect key resolves at -a 3 and carries prov dataflow", () => { + const u = app3.config_uses.find((x) => x.src.includes("readIndirect")); + expect(u?.dst).toContain("@key/PAYMENT_HOST"); + expect(u?.prov).toContain("dataflow"); + }); + + test("config_uses is superset-monotonic L2 ⊆ L3", () => { + const key = (u: { src: string; dst: string }): string => `${u.src}|${u.dst}`; + const l3 = new Set(app3.config_uses.map(key)); + for (const u of app2.config_uses) expect(l3.has(key(u))).toBe(true); + }); + + test("config_reads shrinks as levels rise (the deliberate non-monotonic section)", () => { + expect(app3.config_reads.length).toBeLessThan(app2.config_reads.length); + // a read that never closes on a literal stays unresolved at every level + expect(app3.config_reads.some((r) => r.site.includes("readVia"))).toBe(true); + }); + + test("a reassigned local never widens, even though its initializer is a single literal", () => { + // isReassigned must block this: `key`'s initializer IS one literal, but it's also assigned to. + expect(app3.config_reads.some((r) => r.site.includes("readReassigned"))).toBe(true); + expect(app3.config_uses.some((u) => u.src.includes("readReassigned"))).toBe(false); + }); + + test("a destructuring reassignment (`({ key } = ...)`) never widens either (fix round 1)", () => { + // isReassigned's original identity check missed this: `key` is a binding target nested + // inside the assignment's left side, never the whole of it. + const read = app3.config_reads.find((r) => r.site.includes("readDestructuredKey")); + expect(read?.reason).toBe("non-literal"); + expect(app3.config_uses.some((u) => u.src.includes("readDestructuredKey"))).toBe(false); + }); +}); + +const r4 = await analyze(options(4)); +const app4 = r4.application.application; + +describe("config_use interproc tier (#101 unit C3, -a 4)", () => { + test("a parameter resolves once its one resolved caller passes a literal", () => { + const u = app4.config_uses.find((x) => x.src.includes("readVia@")); + expect(u?.dst).toContain("@key/PAYMENT_HOST"); + expect(u?.prov).toEqual(["dataflow"]); + expect(app4.config_reads.some((r) => r.site.includes("readVia@"))).toBe(false); + }); + + test("never at -a 3 — the interproc tier is level-gated, not just call-graph-gated", () => { + expect(app3.config_uses.some((u) => u.src.includes("readVia@"))).toBe(false); + }); + + test("disagreeing callers block the interproc tier — a missing edge, not a wrong one", () => { + expect(app4.config_uses.some((u) => u.src.includes("readAmbiguous"))).toBe(false); + expect(app4.config_reads.some((r) => r.site.includes("readAmbiguous"))).toBe(true); + }); + + test("a destructuring reassignment still never widens at -a 4 (fix round 1)", () => { + const read = app4.config_reads.find((r) => r.site.includes("readDestructuredKey")); + expect(read?.reason).toBe("non-literal"); + expect(app4.config_uses.some((u) => u.src.includes("readDestructuredKey"))).toBe(false); + }); + + test("config_uses stays superset-monotonic L3 ⊆ L4", () => { + const key = (u: { src: string; dst: string }): string => `${u.src}|${u.dst}`; + const l4 = new Set(app4.config_uses.map(key)); + for (const u of app3.config_uses) expect(l4.has(key(u))).toBe(true); + }); +}); + +describe("Neo4j projection of the config layer (#101)", () => { + const rows = neoProject(r2.application); + + test("ConfigKey nodes are neutral and hang off their artifact", () => { + const id = "can://artifact/artifacts-app/.env@key/PAYMENT_HOST"; + const n = rows.nodes.find((x) => x.value === id); + expect(n?.labels).toContain("ConfigKey"); + expect(n?.labels).not.toContain("TSConfigKey"); + expect(rows.edges.some((e) => e.type === "DEFINES_CONFIG" && e.to.value === id)).toBe(true); + }); + + test("TS_USES_CONFIG carries prov and points at a ConfigKey", () => { + const e = rows.edges.find((x) => x.type === "TS_USES_CONFIG"); + expect(e?.props["prov"]).toEqual(["literal"]); + expect(String(e?.to.value)).toContain("@key/"); + }); +}); diff --git a/test/dataflow.test.ts b/test/dataflow.test.ts index a539472..7e5a686 100644 --- a/test/dataflow.test.ts +++ b/test/dataflow.test.ts @@ -34,7 +34,6 @@ function options(level: 1 | 2 | 3, cacheDir: string, jobs: number): AnalysisOpti eager: true, noBuild: true, phantoms: true, - callGraphProvider: "tsc", cacheDir, verbosity: 0, }; diff --git a/test/external-resolution.test.ts b/test/external-resolution.test.ts index 86a957e..034d942 100644 --- a/test/external-resolution.test.ts +++ b/test/external-resolution.test.ts @@ -39,7 +39,6 @@ function options(): AnalysisOptions { eager: true, noBuild: true, phantoms: true, - callGraphProvider: "tsc", cacheDir: null, verbosity: 0, }; diff --git a/test/fixtures/artifacts-app/.env b/test/fixtures/artifacts-app/.env new file mode 100644 index 0000000..889d18f --- /dev/null +++ b/test/fixtures/artifacts-app/.env @@ -0,0 +1,4 @@ +# comment +PAYMENT_HOST=https://pay.example.com +DB_URL="postgres://u:p@${PAYMENT_HOST}/db" +export NODE_OPTIONS='--max-old-space-size=4096' diff --git a/test/fixtures/artifacts-app/.github/workflows/ci.yml b/test/fixtures/artifacts-app/.github/workflows/ci.yml new file mode 100644 index 0000000..4a79684 --- /dev/null +++ b/test/fixtures/artifacts-app/.github/workflows/ci.yml @@ -0,0 +1,2 @@ +name: ci +on: push diff --git a/test/fixtures/artifacts-app/Dockerfile b/test/fixtures/artifacts-app/Dockerfile new file mode 100644 index 0000000..de4d64f --- /dev/null +++ b/test/fixtures/artifacts-app/Dockerfile @@ -0,0 +1,7 @@ +FROM node:22 +ARG BUILD_ID=local +ENV PAYMENT_HOST=https://pay.example.com +ENV FEATURE_FLAG "on" +ARG VERSION=1.0 +ENV VERSION=$VERSION +COPY . . diff --git a/test/fixtures/artifacts-app/LICENSE b/test/fixtures/artifacts-app/LICENSE new file mode 100644 index 0000000..d1e1072 --- /dev/null +++ b/test/fixtures/artifacts-app/LICENSE @@ -0,0 +1 @@ +MIT License diff --git a/test/fixtures/artifacts-app/README.md b/test/fixtures/artifacts-app/README.md new file mode 100644 index 0000000..446f4a8 --- /dev/null +++ b/test/fixtures/artifacts-app/README.md @@ -0,0 +1 @@ +# artifacts-app diff --git a/test/fixtures/artifacts-app/docker-compose.broken.yml b/test/fixtures/artifacts-app/docker-compose.broken.yml new file mode 100644 index 0000000..2bd3566 --- /dev/null +++ b/test/fixtures/artifacts-app/docker-compose.broken.yml @@ -0,0 +1,3 @@ +services: + web: + ports: [1, 2 diff --git a/test/fixtures/artifacts-app/docker-compose.yml b/test/fixtures/artifacts-app/docker-compose.yml new file mode 100644 index 0000000..75720d1 --- /dev/null +++ b/test/fixtures/artifacts-app/docker-compose.yml @@ -0,0 +1,9 @@ +services: + web: + image: node:22 + ports: + - "3000:3000" + environment: + PAYMENT_HOST: https://pay.example.com + FEATURE_FLAG: "on" +PAYMENT_HOST: https://root-level.example.com diff --git a/test/fixtures/artifacts-app/k8s/multi-broken.yaml b/test/fixtures/artifacts-app/k8s/multi-broken.yaml new file mode 100644 index 0000000..8b8f402 --- /dev/null +++ b/test/fixtures/artifacts-app/k8s/multi-broken.yaml @@ -0,0 +1,6 @@ +services: + web: + image: node:22 +--- +spec: + containers: [1, 2 diff --git a/test/fixtures/artifacts-app/k8s/multi.yaml b/test/fixtures/artifacts-app/k8s/multi.yaml new file mode 100644 index 0000000..1700c38 --- /dev/null +++ b/test/fixtures/artifacts-app/k8s/multi.yaml @@ -0,0 +1,12 @@ +services: + web: + image: node:22 +--- +apiVersion: apps/v1 +kind: Deployment +spec: + containers: + - name: app + env: + - name: PAYMENT_HOST + value: https://pay.example.com diff --git a/test/fixtures/artifacts-app/logo.bin b/test/fixtures/artifacts-app/logo.bin new file mode 100644 index 0000000..e5132ea Binary files /dev/null and b/test/fixtures/artifacts-app/logo.bin differ diff --git a/test/fixtures/artifacts-app/notes.dat b/test/fixtures/artifacts-app/notes.dat new file mode 100644 index 0000000..6932b4d --- /dev/null +++ b/test/fixtures/artifacts-app/notes.dat @@ -0,0 +1 @@ +plain data, no rule matches this diff --git a/test/fixtures/artifacts-app/package-lock.json b/test/fixtures/artifacts-app/package-lock.json new file mode 100644 index 0000000..e2607b1 --- /dev/null +++ b/test/fixtures/artifacts-app/package-lock.json @@ -0,0 +1,12 @@ +{ + "name": "artifacts-app", + "lockfileVersion": 3, + "packages": { + "": { "name": "artifacts-app" }, + "node_modules/express": { "version": "4.19.2" }, + "node_modules/@scope/util": { "version": "2.1.5" }, + "node_modules/typescript": { "version": "5.5.4" }, + "node_modules/express/node_modules/transitive-shadow": { "version": "9.9.9" }, + "node_modules/lockonly-transitive": { "version": "1.0.0" } + } +} diff --git a/test/fixtures/artifacts-app/package.json b/test/fixtures/artifacts-app/package.json new file mode 100644 index 0000000..d161144 --- /dev/null +++ b/test/fixtures/artifacts-app/package.json @@ -0,0 +1,9 @@ +{ + "name": "artifacts-app", + "version": "1.0.0", + "workspaces": ["packages/*"], + "dependencies": { "express": "^4.19.0", "@scope/util": "~2.1.0" }, + "devDependencies": { "typescript": "^5.5.0", "@types/typed-only-pkg": "^1.0.0" }, + "optionalDependencies": { "fsevents": "^2.3.3" }, + "peerDependencies": { "react": ">=18" } +} diff --git a/test/fixtures/artifacts-app/packages/web/bun.lock b/test/fixtures/artifacts-app/packages/web/bun.lock new file mode 100644 index 0000000..72674e9 --- /dev/null +++ b/test/fixtures/artifacts-app/packages/web/bun.lock @@ -0,0 +1,6 @@ +{ + "lockfileVersion": 1, + "packages": { + "lodash": ["lodash@4.17.21", {},], + }, +} diff --git a/test/fixtures/artifacts-app/packages/web/package.json b/test/fixtures/artifacts-app/packages/web/package.json new file mode 100644 index 0000000..41162d3 --- /dev/null +++ b/test/fixtures/artifacts-app/packages/web/package.json @@ -0,0 +1,4 @@ +{ + "name": "@artifacts-app/web", + "dependencies": { "lodash": "^4.17.21" } +} diff --git a/test/fixtures/artifacts-app/pnpm-lock.yaml b/test/fixtures/artifacts-app/pnpm-lock.yaml new file mode 100644 index 0000000..c5333d8 --- /dev/null +++ b/test/fixtures/artifacts-app/pnpm-lock.yaml @@ -0,0 +1,6 @@ +lockfileVersion: '9.0' +importers: + .: + dependencies: + express: + specifier: ^4.18.0 diff --git a/test/fixtures/artifacts-app/src/config.ts b/test/fixtures/artifacts-app/src/config.ts new file mode 100644 index 0000000..56323a9 --- /dev/null +++ b/test/fixtures/artifacts-app/src/config.ts @@ -0,0 +1,73 @@ +export function readHost(): string | undefined { + return process.env.PAYMENT_HOST; +} +export function readFlag(): string | undefined { + return process.env["FEATURE_FLAG"]; +} +export function readDestructured(): string | undefined { + const { NODE_OPTIONS } = process.env; + return NODE_OPTIONS; +} +export function readVia(name: string): string | undefined { + return process.env[name]; +} +// The lone internal caller (#101 unit C3): gives the interproc tier a resolved call site with a +// literal argument, so readVia's parameter closes on "PAYMENT_HOST" at -a 4 (never at -a 3). +export function readViaResolved(): string | undefined { + return readVia("PAYMENT_HOST"); +} +export function readIndirect(): string | undefined { + const key = "PAYMENT_HOST"; + return process.env[key]; +} +// Two callers disagreeing on the literal (#101 unit C3): the interproc tier must NOT widen — +// "every resolved internal call site passes the same string literal" fails here on purpose. +export function readAmbiguous(name: string): string | undefined { + return process.env[name]; +} +export function callAmbiguousA(): string | undefined { + return readAmbiguous("PAYMENT_HOST"); +} +export function callAmbiguousB(): string | undefined { + return readAmbiguous("FEATURE_FLAG"); +} +// A reassigned local (#101 unit C3): isReassigned must block the intra tier even though the +// initializer alone is a single literal. +export function readReassigned(): string | undefined { + let key = "PAYMENT_HOST"; + if (process.env.NODE_ENV === "test") key = "FEATURE_FLAG"; + return process.env[key]; +} +// Destructuring reassignment (#101 unit C3 fix round 1): `({ key } = ...)` rebinds `key` just as +// much as `key = ...` does. isReassigned must catch this even though the tracked identifier never +// appears as the WHOLE left side of an assignment, only nested inside one. +export function readDestructuredKey(): string | undefined { + let key = "PAYMENT_HOST"; + ({ key } = { key: "FEATURE_FLAG" }); + return process.env[key]; +} +export function readUndeclared(): string | undefined { + return process.env.NOT_DECLARED_ANYWHERE; +} +export function readList(): string[] { + return process.env.LIST.split(","); +} +export class Client { + private readonly host = process.env.PAYMENT_HOST; + url(): string | undefined { + return this.host; + } +} +import nconf from "nconf"; +export function readViaLibrary(): string | undefined { + return nconf.get("PAYMENT_HOST"); +} +export function readTemplateInterpolated(x: string): string | undefined { + return nconf.get(`PAYMENT_${x}`); +} +export function readTemplateLiteral(): string | undefined { + return nconf.get(`PAYMENT_HOST`); +} +export function readBuildId(): string | undefined { + return process.env.BUILD_ID; +} diff --git a/test/fixtures/artifacts-app/src/index.ts b/test/fixtures/artifacts-app/src/index.ts new file mode 100644 index 0000000..1e02624 --- /dev/null +++ b/test/fixtures/artifacts-app/src/index.ts @@ -0,0 +1,5 @@ +import express from "express"; +import leftPad from "left-pad"; +import type { SomeType } from "typed-only-pkg"; +import * as fs from "node:fs"; +export function main(): void { void express; void leftPad; void fs; const t: SomeType | null = null; void t; } diff --git a/test/fixtures/artifacts-app/tsconfig.broken.json b/test/fixtures/artifacts-app/tsconfig.broken.json new file mode 100644 index 0000000..b8f263f --- /dev/null +++ b/test/fixtures/artifacts-app/tsconfig.broken.json @@ -0,0 +1 @@ +{ "compilerOptions": { "strict": "unterminated } diff --git a/test/fixtures/artifacts-app/tsconfig.json b/test/fixtures/artifacts-app/tsconfig.json new file mode 100644 index 0000000..3ab0a74 --- /dev/null +++ b/test/fixtures/artifacts-app/tsconfig.json @@ -0,0 +1,5 @@ +{ + // JSONC: comments and trailing commas are legal in tsconfig + "compilerOptions": { "strict": true, "target": "ES2022", }, + "include": ["src"], +} diff --git a/test/fixtures/artifacts-app/yarn.lock b/test/fixtures/artifacts-app/yarn.lock new file mode 100644 index 0000000..db6bda8 --- /dev/null +++ b/test/fixtures/artifacts-app/yarn.lock @@ -0,0 +1,3 @@ +yarn lockfile v1 +lodash@^4.17.21: + version "4.17.21" diff --git a/test/fixtures/unresolvable-js-app/mocks/dpapi.js b/test/fixtures/unresolvable-js-app/mocks/dpapi.js new file mode 100644 index 0000000..bbad1de --- /dev/null +++ b/test/fixtures/unresolvable-js-app/mocks/dpapi.js @@ -0,0 +1,13 @@ +// Deliberately OUTSIDE tsconfig's `include`, so it is discovered as source but lands in a program +// with no default lib. Resolving the global `Error` here makes the TypeScript checker throw +// rather than return undefined — the vscode crash this fixture pins down. +class defaultDpapi { + protectData() { + throw new Error('Dpapi bindings unavailable'); + } + unprotectData() { + throw new Error('Dpapi bindings unavailable'); + } +} +const Dpapi = new defaultDpapi(); +export { Dpapi }; diff --git a/test/fixtures/unresolvable-js-app/package.json b/test/fixtures/unresolvable-js-app/package.json new file mode 100644 index 0000000..9f03fdf --- /dev/null +++ b/test/fixtures/unresolvable-js-app/package.json @@ -0,0 +1,4 @@ +{ + "name": "unresolvable-js-app", + "version": "1.0.0" +} diff --git a/test/fixtures/unresolvable-js-app/src/index.ts b/test/fixtures/unresolvable-js-app/src/index.ts new file mode 100644 index 0000000..a325f76 --- /dev/null +++ b/test/fixtures/unresolvable-js-app/src/index.ts @@ -0,0 +1,7 @@ +export function greet(name: string): string { + return `hello ${name}`; +} + +export function run(): string { + return greet("world"); +} diff --git a/test/fixtures/unresolvable-js-app/tsconfig.json b/test/fixtures/unresolvable-js-app/tsconfig.json new file mode 100644 index 0000000..85f2107 --- /dev/null +++ b/test/fixtures/unresolvable-js-app/tsconfig.json @@ -0,0 +1,4 @@ +{ + "compilerOptions": { "target": "ES2020", "module": "ESNext" }, + "include": ["src"] +} diff --git a/test/l1-body-cache-shape.test.ts b/test/l1-body-cache-shape.test.ts new file mode 100644 index 0000000..f27cb90 --- /dev/null +++ b/test/l1-body-cache-shape.test.ts @@ -0,0 +1,31 @@ +import { describe, expect, test } from "bun:test"; +import { populateL1Body } from "../src/schema/l1Body"; +import type { AnalysisInternal, TSCallable, TSModule } from "../src/schema"; + +describe("l1Body tolerates a stale-cache callable (#101 fix round 1)", () => { + test("callable missing config_accesses (call_sites present) does not throw", () => { + // Simulates a TSCallable deserialized from a .codeanalyzer cache written before Task 7 added + // `config_accesses` — loadCache only invalidates on analyzer_version change, not shape, so a + // same-version warm cache can hand resetCallable an object narrower than today's TSCallable + // contract. `call_sites` present, `config_accesses` deliberately OMITTED; `as unknown as` + // documents this as an intentional stand-in for stale cached data, not an oversight. + const stale = { + body: {}, + call_sites: [ + { + start_line: 5, start_column: 3, end_line: 5, end_column: 10, bytes: [40, 47], + method_name: "foo", argument_types: [], type_arguments: [], + is_constructor_call: false, is_optional_chain: false, + }, + ], + } as unknown as TSCallable; + const mod = { functions: { stale }, types: {} } as unknown as TSModule; + const app = { symbol_table: { "x.ts": mod } } as unknown as AnalysisInternal; + + expect(() => populateL1Body(app)).not.toThrow(); + + const kinds = Object.values(stale.body).map((n) => n.kind); + expect(kinds).toEqual(["call"]); // call_sites still processed normally + expect(kinds).not.toContain("config_access"); // nothing to materialize; no crash either + }); +}); diff --git a/test/multi-tsconfig.test.ts b/test/multi-tsconfig.test.ts index 7084748..2d630f9 100644 --- a/test/multi-tsconfig.test.ts +++ b/test/multi-tsconfig.test.ts @@ -42,7 +42,6 @@ function options(): AnalysisOptions { eager: true, noBuild: true, phantoms: true, - callGraphProvider: "tsc", cacheDir: null, verbosity: 0, }; diff --git a/test/neo4j-bolt.test.ts b/test/neo4j-bolt.test.ts index 684f7c3..9015a52 100644 --- a/test/neo4j-bolt.test.ts +++ b/test/neo4j-bolt.test.ts @@ -48,7 +48,6 @@ function optsFor(overrides: Partial = {}): AnalysisOptions { eager: true, noBuild: true, phantoms: true, - callGraphProvider: "tsc", cacheDir: path.join(TMP, "cache"), verbosity: 0, ...overrides, diff --git a/test/neo4j-schema.test.ts b/test/neo4j-schema.test.ts index 02417b7..8872c94 100644 --- a/test/neo4j-schema.test.ts +++ b/test/neo4j-schema.test.ts @@ -30,7 +30,7 @@ async function fixtureRows() { neo4jUri: null, neo4jUser: "neo4j", neo4jPassword: "", neo4jDatabase: null, analysisLevel: 4, graphs: ["cfg", "dfg", "pdg", "sdg"], graphFieldDepth: 3, jobs: 1, targetFiles: null, skipTests: true, eager: true, - noBuild: true, phantoms: true, callGraphProvider: "union", cacheDir, verbosity: 0, + noBuild: true, phantoms: true, cacheDir, verbosity: 0, }; try { return project((await analyze(opts)).application); @@ -100,14 +100,21 @@ describe("neo4j schema conformance", () => { } }); - test("2.0.0 emits only TS-prefixed specific labels and TS_ rel types (#66)", () => { + test("TS-prefixed labels/rels, with the artifact layer's sanctioned NEUTRAL exception (#66, #101)", () => { + // :Artifact/:Package (+ HAS_ARTIFACT/DECLARES_DEPENDENCY/LOCKS) are deliberately + // language-neutral so sibling analyzers MERGE onto the same nodes (python PR #160's rule); + // edges that stay this analyzer's own claim (TS_PROVIDES, TS_UNRESOLVED_IMPORT) keep TS_. + const NEUTRAL_LABELS = new Set(["Artifact", "Package", "ConfigKey"]); + const NEUTRAL_RELS = new Set(["HAS_ARTIFACT", "DECLARES_DEPENDENCY", "LOCKS", "DEFINES_CONFIG"]); for (const node of rows.nodes) { for (const l of node.labels) { - const ok = l === "CanNode" || l === "Application" || l.startsWith("TS"); + const ok = l === "CanNode" || l === "Application" || l.startsWith("TS") || NEUTRAL_LABELS.has(l); expect(ok, `bare label leaked: ${l}`).toBe(true); } } - for (const edge of rows.edges) expect(edge.type.startsWith("TS_"), `bare rel leaked: ${edge.type}`).toBe(true); + for (const edge of rows.edges) { + expect(edge.type.startsWith("TS_") || NEUTRAL_RELS.has(edge.type), `bare rel leaked: ${edge.type}`).toBe(true); + } }); }); diff --git a/test/schema-v2.test.ts b/test/schema-v2.test.ts index eda00ce..ac7f528 100644 --- a/test/schema-v2.test.ts +++ b/test/schema-v2.test.ts @@ -38,7 +38,6 @@ function options(): AnalysisOptions { eager: true, noBuild: true, phantoms: true, - callGraphProvider: "tsc", cacheDir: null, verbosity: 0, }; @@ -88,7 +87,19 @@ describe("schema v2 — L1 envelope", () => { expect(v2.schema_version).toBe("2.1.0"); expect(v2.language).toBe("typescript"); expect(v2.max_level).toBe(1); - expect(Object.keys(root).sort()).toEqual(["call_graph", "id", "kind", "param_in", "param_out", "symbol_table"]); + expect(Object.keys(root).sort()).toEqual([ + "artifacts", + "call_graph", + "config_reads", + "config_uses", + "dependencies", + "id", + "kind", + "param_in", + "param_out", + "symbol_table", + "unresolved_imports", + ]); expect(root.id).toBe("can://typescript/sample-app"); expect(root.kind).toBe("application"); }); @@ -246,7 +257,7 @@ describe("schema v2 — L1 skips the call-graph solve (issue #31)", () => { const spy = spyOn(tscProvider, "build"); const cacheDir = fs.mkdtempSync(path.join(os.tmpdir(), "cants-v2-l1-guard-")); try { - const v1L1 = (await analyze({ ...options(), analysisLevel: 1, callGraphProvider: "tsc", cacheDir })).internal; + const v1L1 = (await analyze({ ...options(), analysisLevel: 1, cacheDir })).internal; expect(spy).not.toHaveBeenCalled(); expect(v1L1.call_graph).toEqual([]); expect(Object.keys(v1L1.external_symbols)).toEqual([]); @@ -261,7 +272,7 @@ describe("schema v2 — L1 skips the call-graph solve (issue #31)", () => { const spy = spyOn(tscProvider, "build"); const cacheDir = fs.mkdtempSync(path.join(os.tmpdir(), "cants-v2-l1-guard-l2-")); try { - const v1L2guard = (await analyze({ ...options(), analysisLevel: 2, callGraphProvider: "tsc", cacheDir })).internal; + const v1L2guard = (await analyze({ ...options(), analysisLevel: 2, cacheDir })).internal; expect(spy).toHaveBeenCalledTimes(1); expect(v1L2guard.call_graph.length).toBeGreaterThan(0); } finally { @@ -701,6 +712,15 @@ function canNodeIds(app: TSAnalysis): Set { } for (const id of Object.keys(app.application.external_symbols ?? {})) ids.add(id); for (const id of Object.keys(app.application.synthesized_callables ?? {})) ids.add(id); + // Repository-artifact layer (#101/PR-160 shape): artifacts/packages are NOT CanNodes (own + // neutral merge labels) — but TS_PROVIDES / TS_UNRESOLVED_IMPORT mint module-level + // :TSExternal ghosts in the CanNode id space. + for (const d of app.application.dependencies ?? []) { + for (const top of d.provides_imports) ids.add(`${app.application.id}/@external/${top}`); + } + for (const u of app.application.unresolved_imports ?? []) { + ids.add(`${app.application.id}/@external/${u.module}`); + } return ids; } @@ -714,8 +734,13 @@ function resolvesToCount(app: TSAnalysis): number { } describe("neo4j ↔ json count parity — full depth (issue #27)", () => { - test("node count: 1 :Application row + every :CanNode id", () => { - expect(monoRows.nodes.length).toBe(1 + canNodeIds(monoApp4).size); + test("node count: 1 :Application row + every :CanNode id + neutral Artifact/Package/ConfigKey rows", () => { + const artifactCount = Object.keys(monoApp4.application.artifacts ?? {}).length; + const packageCount = new Set((monoApp4.application.dependencies ?? []).map((d) => d.name)).size; + const configKeyCount = new Set( + Object.values(monoApp4.application.artifacts ?? {}).flatMap((a) => a.config_keys.map((ck) => ck.id)), + ).size; + expect(monoRows.nodes.length).toBe(1 + canNodeIds(monoApp4).size + artifactCount + packageCount + configKeyCount); }); test("typed overlay relationships match their JSON edge-list length 1:1", () => { @@ -742,7 +767,11 @@ describe("neo4j ↔ json count parity — full depth (issue #27)", () => { const containmentEdges = containment.reduce((n, t) => n + relCount(monoRows, t), 0); const externalCount = Object.keys(monoApp4.application.external_symbols ?? {}).length; const synthCount = Object.keys(monoApp4.application.synthesized_callables ?? {}).length; - expect(containmentEdges).toBe(canNodeIds(monoApp4).size - externalCount - synthCount); + // minted provides/unresolved ghosts are off-tree CanNodes too (like externals) + const ghostIds = new Set(); + for (const d of monoApp4.application.dependencies ?? []) for (const t of d.provides_imports) ghostIds.add(t); + for (const u of monoApp4.application.unresolved_imports ?? []) ghostIds.add(u.module); + expect(containmentEdges).toBe(canNodeIds(monoApp4).size - externalCount - synthCount - ghostIds.size); }); test("EXTENDS/IMPLEMENTS have no JSON edge-list (extends_ids/implements_ids node props are the source of truth); counts still match 1:1", () => { @@ -774,10 +803,14 @@ describe("neo4j ↔ json count parity — full depth (issue #27)", () => { (n, t) => n + relCount(monoRows, t), 0, ); + const artifactLayer = [ + "HAS_ARTIFACT", "DECLARES_DEPENDENCY", "LOCKS", "TS_PROVIDES", "TS_UNRESOLVED_IMPORT", + "DEFINES_CONFIG", "TS_USES_CONFIG", + ].reduce((n, t) => n + relCount(monoRows, t), 0); const resolvesTo = relCount(monoRows, "TS_RESOLVES_TO"); const heritage = relCount(monoRows, "TS_EXTENDS") + relCount(monoRows, "TS_IMPLEMENTS"); expect(resolvesTo).toBe(resolvesToCount(monoApp4)); - expect(typedOverlay + containment + resolvesTo + heritage).toBe(monoRows.edges.length); + expect(typedOverlay + containment + artifactLayer + resolvesTo + heritage).toBe(monoRows.edges.length); }); test("DDG/CFG_NEXT parity survives the writers: every row keyed, keys fully discriminate (issue #70)", () => { diff --git a/test/synthesized-nodes.test.ts b/test/synthesized-nodes.test.ts index 018cb88..b639260 100644 --- a/test/synthesized-nodes.test.ts +++ b/test/synthesized-nodes.test.ts @@ -13,7 +13,10 @@ const ANON = "src/x.foo:<3:10>"; const SPAN: TSSpan = { start: [1, 1], end: [5, 1], bytes: [0, 10] }; const callable = (signature: string, name: string): TSCallable => - ({ signature, name, kind: "function", span: SPAN, parameters: [], call_sites: [], inner_callables: {}, inner_classes: {} }) as unknown as TSCallable; + ({ + signature, name, kind: "function", span: SPAN, parameters: [], + call_sites: [], config_accesses: [], inner_callables: {}, inner_classes: {}, + }) as unknown as TSCallable; const app: AnalysisInternal = { symbol_table: { @@ -23,7 +26,7 @@ const app: AnalysisInternal = { classes: {}, interfaces: {}, enums: {}, type_aliases: {}, namespaces: {}, variables: [], } as unknown as TSModule, }, - call_graph: [{ source: "src/x.foo", target: ANON, type: CALL_DEP, weight: 1, provenance: ["jelly"], tags: {} }], + call_graph: [{ source: "src/x.foo", target: ANON, type: CALL_DEP, weight: 1, provenance: ["defuse"], tags: {} }], external_symbols: {}, synthesized_callables: { [ANON]: { name: "", path: "src/x.ts", start_line: 3, start_column: 10 } }, }; diff --git a/test/tagged-templates.test.ts b/test/tagged-templates.test.ts new file mode 100644 index 0000000..0ea2452 --- /dev/null +++ b/test/tagged-templates.test.ts @@ -0,0 +1,56 @@ +/** + * Tagged template expressions are call sites (#98): `inline\`url(...)\`` must record a `call` + * body node and resolve a call edge to the tag — found missing by the vscode Joern ledger + * (cssValue.ts's `inline` idiom), then crash-guarded (tagged templates have no arguments list). + */ +import { describe, expect, test } from "bun:test"; +import * as fs from "node:fs"; +import * as os from "node:os"; +import * as path from "node:path"; +import { analyze } from "../src/core"; +import type { AnalysisOptions } from "../src/options"; + +const dir = fs.mkdtempSync(path.join(os.tmpdir(), "cants-tagged-")); +fs.mkdirSync(path.join(dir, "src")); +fs.writeFileSync( + path.join(dir, "src", "x.ts"), + [ + "export function inline(strings: TemplateStringsArray, ...v: string[]): string { return ''; }", + "export function asCSSUrl(): string { return inline`url('x')`; }", + "export const top = inline`module-scope`;", + "declare const unknownTag: any;", + "export function throughLinker(): void { unknownTag`unresolved-tag`; }", + "export function mkSheet(): number { return 1; }", + "export function createRule(sel: string, sheet = mkSheet()): number { return sheet; }", + ].join("\n"), +); + +const opts = { + input: dir, output: null, emit: "json", appName: "tagged", neo4jUri: null, neo4jUser: "neo4j", + neo4jPassword: "", neo4jDatabase: null, analysisLevel: 2, graphs: [], graphFieldDepth: 3, + jobs: 1, targetFiles: null, skipTests: true, eager: true, noBuild: true, phantoms: true, + cacheDir: fs.mkdtempSync(path.join(os.tmpdir(), "cants-tagged-cache-")), verbosity: 0, +} as AnalysisOptions; +const result = await analyze(opts); +fs.rmSync(dir, { recursive: true, force: true }); + +describe("tagged template calls (#98)", () => { + test("a tagged template resolves a call edge to its tag", () => { + expect(result.internal.call_graph.some((e) => e.source === "src/x.asCSSUrl" && e.target === "src/x.inline")).toBe(true); + }); + + test("a module-scope tagged template is attributed to the module", () => { + expect(result.internal.call_graph.some((e) => e.source === "src/x" && e.target === "src/x.inline")).toBe(true); + }); + + test("a parameter-default initializer call is attributed to the callable (#98)", () => { + expect(result.internal.call_graph.some((e) => e.source === "src/x.createRule" && e.target === "src/x.mkSheet")).toBe(true); + }); + + test("the tagged call is a body call node with a refined callee", () => { + const fn = result.application.application.symbol_table["src/x.ts"]?.functions["asCSSUrl"]; + const calls = Object.values(fn?.body ?? {}).filter((b) => b.kind === "call"); + expect(calls.length).toBe(1); + expect(calls[0]?.callee).toBe("can://typescript/tagged/src/x.ts/inline"); + }); +}); diff --git a/test/union-provider.test.ts b/test/union-provider.test.ts deleted file mode 100644 index d9f0cc9..0000000 --- a/test/union-provider.test.ts +++ /dev/null @@ -1,73 +0,0 @@ -/** - * Unit tests for the union merge (issue #11): tsc + jelly edges and external symbols must be - * combined — not discarded — with provenance preserved so consumers can still tell them apart. - */ -import { describe, expect, test } from "bun:test"; -import type { CallGraphResult } from "../src/semantic_analysis"; -import { mergeCallGraphs } from "../src/semantic_analysis"; -import { CALL_DEP, type TSCallEdge } from "../src/schema"; - -const edge = (source: string, target: string, provenance: string[], extra: Partial = {}): TSCallEdge => ({ - source, - target, - type: CALL_DEP, - weight: 1, - provenance, - tags: {}, - ...extra, -}); - -const result = ( - edges: TSCallEdge[], - external: CallGraphResult["external_symbols"] = {}, - synthesized: CallGraphResult["synthesized_callables"] = {}, -): CallGraphResult => ({ - edges, - external_symbols: external, - synthesized_callables: synthesized, -}); - -describe("mergeCallGraphs", () => { - test("keeps jelly-only edges (the bug: they used to be dropped)", () => { - const tsc = result([edge("a", "b", ["tsc"])]); - const jelly = result([edge("c", "d", ["jelly"])]); - const merged = mergeCallGraphs(tsc, jelly); - const keys = merged.edges.map((e) => `${e.source}->${e.target}`).sort(); - expect(keys).toEqual(["a->b", "c->d"]); - }); - - test("an edge found by both carries both provenances and summed weight", () => { - const tsc = result([edge("a", "b", ["tsc"], { weight: 2 })]); - const jelly = result([edge("a", "b", ["jelly"], { weight: 3 })]); - const merged = mergeCallGraphs(tsc, jelly); - expect(merged.edges).toHaveLength(1); - expect(merged.edges[0].provenance.sort()).toEqual(["jelly", "tsc"]); - expect(merged.edges[0].weight).toBe(5); - }); - - test("merges external symbols from both, tsc winning on conflict", () => { - const tsc = result([], { "pkg.foo": { name: "foo", module: "pkg" } }); - const jelly = result([], { - "pkg.foo": { name: "FOO-jelly", module: "pkg" }, - "pkg.bar": { name: "bar", module: "pkg" }, - }); - const merged = mergeCallGraphs(tsc, jelly); - expect(Object.keys(merged.external_symbols).sort()).toEqual(["pkg.bar", "pkg.foo"]); - expect(merged.external_symbols["pkg.foo"].name).toBe("foo"); // base (tsc) wins - }); - - test("unions synthesized (anonymous-callback) callables from both", () => { - const tsc = result([]); - const jelly = result([], {}, { "src/x.foo:<3:10>": { name: "", path: "src/x.ts", start_line: 3, start_column: 10 } }); - const merged = mergeCallGraphs(tsc, jelly); - expect(Object.keys(merged.synthesized_callables)).toEqual(["src/x.foo:<3:10>"]); - }); - - test("does not mutate the input results", () => { - const tsc = result([edge("a", "b", ["tsc"])]); - const jelly = result([edge("a", "b", ["jelly"])]); - mergeCallGraphs(tsc, jelly); - expect(tsc.edges[0].provenance).toEqual(["tsc"]); - expect(tsc.edges[0].weight).toBe(1); - }); -});