diff --git a/README.md b/README.md index 10ea9f9..3d681bd 100644 --- a/README.md +++ b/README.md @@ -190,9 +190,6 @@ Options: binding (default: repo files only) --no-artifact-text keep the artifact inventory but drop captured raw text - --artifact-text-max-bytes per-file byte cap for captured artifact text; - larger files are truncated and flagged - (default: "262144") -c, --cache-dir cache/intermediate directory -v, --verbose increase verbosity (repeatable) -h, --help display help for command diff --git a/docs/design/specs/2026-08-30-artifact-layer-v130-parity.md b/docs/design/specs/2026-08-30-artifact-layer-v130-parity.md index c4d6491..18819f1 100644 --- a/docs/design/specs/2026-08-30-artifact-layer-v130-parity.md +++ b/docs/design/specs/2026-08-30-artifact-layer-v130-parity.md @@ -40,7 +40,7 @@ application.config_uses: TSConfigUse[] // C application.config_reads: TSConfigRead[] // C TSArtifact { id: `can://artifact//`, kind: "artifact", path, format, roles[], - size_bytes, sha256, source, text_truncated, extraction: none|partial|full, + size_bytes, sha256, source, extraction: none|partial|full, config_keys: TSConfigKey[] } TSConfigKey { id: `${artifactId}@key/${dotted}`, key, namespace, value?, span?, references[] } TSDependency { name, ecosystem: "npm", spec, kind: runtime|dev|optional|peer|build, extras[], @@ -64,7 +64,7 @@ code-only. but decodable → `roles: ["unknown"]`; undecodable → `format: "binary"`, `source: ""`, hash and size only. (Today's branch skips unmatched files — the largest divergence from shipped python.) 2. **Text policy.** Capture on by default; `--artifact-text` / `--no-artifact-text`; - `--artifact-text-max-bytes` (default 256 KiB). `text_truncated` marks a stored prefix. + No byte cap: `source` is the whole file, or `""` under `--no-artifact-text` (superseded by #116). `sha256` and `size_bytes` are always the **full file**. Extraction (dependencies, config keys) parses the **full on-disk text**, never the stored copy — truncation can never change extracted meaning. @@ -236,8 +236,8 @@ record why a package is present. - **`--app-name` is the cross-analyzer join precondition** — artifact ids are language-neutral so a TS and a python analysis of one monorepo MERGE onto the same `:Artifact`; they only do so if both runs pinned the same app name. Analyzers pointed at different subdirectories disagree. -- **`sha256` is always the full file; `source` may not be.** Under `--artifact-text-max-bytes`, - `text_truncated: true` means the stored text is a prefix — hash-compare on `sha256`, never on +- **`sha256` is always the full file, and so is `source`** (the byte cap was removed in #116) — + hash-compare on `sha256`, never on `source`, and never re-derive meaning from a truncated copy (the analyzer itself parses the full on-disk text). - **`value` is present by default** and absent under `--no-artifact-text` — an absent `value` is a diff --git a/docs/skills/analyzing-cants-graphs/SKILL.md b/docs/skills/analyzing-cants-graphs/SKILL.md index 16f5e9b..b70cbc9 100644 --- a/docs/skills/analyzing-cants-graphs/SKILL.md +++ b/docs/skills/analyzing-cants-graphs/SKILL.md @@ -86,15 +86,11 @@ specifically so a TS and a Python analysis of one repository MERGE onto the same subdirectories of one monorepo, or run with mismatched app names, will silently disagree on every artifact and package id and never merge. -**`sha256` is always the full file; `source` may not be — and in Neo4j, `Artifact` has no -`source` at all.** Under `--artifact-text-max-bytes`, `text_truncated: true` in `analysis.json` -means the stored `source` is a prefix — hash-compare on `sha256`, never on `source`, and never -re-derive meaning from a truncated copy (the analyzer itself always parses the full on-disk text -for extraction). In the graph, this is moot for a different reason: `Artifact` carries `sha256` + -`size_bytes` but **no `source` property at all** — verbatim text lives only in `analysis.json`, by -design (`src/build/neo4j/project.ts`: "`source` text stays off the graph — hash and size -dereference to it"). A query like `WHERE f.source CONTAINS "..."` will not error; it will silently -match nothing. +**`source` is the whole file, in both projections.** There is no byte cap: an artifact's text is +captured complete or, under `--no-artifact-text`, not at all (`source: ""`). `Artifact` carries +`source` in Neo4j as well as in `analysis.json`, matching codeanalyzer-python — so +`WHERE a.source CONTAINS "..."` works against the graph. Compare on `sha256` when you want +identity rather than content; it is always the hash of the full file. **`value` is present by default and absent under `--no-artifact-text`** — an absent `ConfigKey.value` is a capture setting, not an empty or unset config key. Check `references/vocabulary.md`'s diff --git a/docs/skills/analyzing-cants-graphs/references/vocabulary.md b/docs/skills/analyzing-cants-graphs/references/vocabulary.md index e417973..7bbcd00 100644 --- a/docs/skills/analyzing-cants-graphs/references/vocabulary.md +++ b/docs/skills/analyzing-cants-graphs/references/vocabulary.md @@ -10,7 +10,7 @@ assuming the graph is empty. | label | merge key | properties | notes | | --- | --- | --- | --- | | `TSApplication` | `id` | id, schema_version, language, max_level, k_limit, analyzer_name, analyzer_version | one per run; the `:Application` anchor | -| `Artifact` | `id` (`can://artifact//`) | id, kind, path, format, roles[], size_bytes, sha256, extraction | **language-neutral, no TS prefix by design** — sibling analyzers MERGE onto the same node. No `source`/`text_truncated`/`config_keys` here (see "No verbatim text in the graph" below) | +| `Artifact` | `id` (`can://artifact//`) | id, kind, path, format, roles[], size_bytes, sha256, source, extraction | **language-neutral, no TS prefix by design** — sibling analyzers MERGE onto the same node. `source` is the WHOLE file (no byte cap, #116) or `""` under `--no-artifact-text`; `config_keys` are separate `ConfigKey` nodes | | `Package` | `id` (purl `pkg:npm/`, scoped `pkg:npm/%40scope/`) | id, ecosystem, name | language-neutral | | `ConfigKey` | `id` (`@key/`) | id, key, namespace, value, references[] | language-neutral; `key` is always the bare dotted name even when `id` carries an internal `arg.`/`env.` disambiguation prefix (see SKILL.md's identity section) | | `TSModule` | `id` | _module, content_hash, id, is_declaration_file, is_tsx, kind, name, start_line, end_line | `name` is the file key (e.g. `"src/config.ts"`, WITH extension) — same value as `_module` | @@ -36,14 +36,16 @@ matching rule): `dependency-manifest`, `tool-config`, `container-image`, `servic \| `full`. `ConfigKey.namespace`: `env` \| `json` \| `yaml` \| `toml` \| `ini` \| `properties` \| `dockerfile`. -### No verbatim text in the graph +### Source text in the graph -Neither `TSModule` nor `TSCallable` nor `Artifact` carries source text, a file path, or column -positions in Neo4j — only `_module`/`path` (the file key) and `start_line`/`end_line`. This is a -deliberate design line (`src/build/neo4j/project.ts`: "`source` text stays off the graph — hash -and size dereference to it"), not an omission. To read exact text: re-open the file at -`start_line`/`end_line`, or read `analysis.json`, where every module's `source` is stored once and -every node's exact text is `source.slice(...span.bytes)`. +`Artifact` carries `source` — the whole file, matching codeanalyzer-python, so +`WHERE a.source CONTAINS "..."` works. There is no byte cap; the only way `source` is empty is +`--no-artifact-text`. + +Code nodes are different: neither `TSModule` nor `TSCallable` carries source text, a file path, or +column positions — only `_module`/`path` (the file key) and `start_line`/`end_line`. To read exact +code text, re-open the file at those lines, or read `analysis.json`, where every module's `source` +is stored once and every node's exact text is `source.slice(...span.bytes)`. ### External ghosts (`TSExternal`) — two grains, one label diff --git a/schema.neo4j.json b/schema.neo4j.json index 6584e49..29f1aef 100644 --- a/schema.neo4j.json +++ b/schema.neo4j.json @@ -29,7 +29,8 @@ "roles": "string[]", "size_bytes": "integer", "sha256": "string", - "extraction": "string" + "extraction": "string", + "source": "string" } }, { diff --git a/src/artifacts/index.ts b/src/artifacts/index.ts index 61be983..ed5d801 100644 --- a/src/artifacts/index.ts +++ b/src/artifacts/index.ts @@ -14,7 +14,6 @@ */ import * as fs from "node:fs"; import * as path from "node:path"; -import { DEFAULT_ARTIFACT_TEXT_MAX_BYTES } from "../options"; import type { AnalysisOptions } from "../options"; import type { TSArtifact, TSDependency, TSImportBinding, TSModule } from "../schema"; import { sha256 } from "../utils"; @@ -74,15 +73,11 @@ export function inventoryArtifacts( } } const text = decodeLossy(raw); + // Captured whole or not at all -- there is no byte cap. A truncated `source` is a prefix that + // reads like a complete file, and every consumer then needs a flag to tell the two apart; + // `--no-artifact-text` remains the way to opt out of the payload entirely. const capture = opts.artifactText ?? true; - const cap = opts.artifactTextMaxBytes ?? DEFAULT_ARTIFACT_TEXT_MAX_BYTES; - const textByteLength = text === undefined ? 0 : Buffer.byteLength(text, "utf8"); - const stored = - !capture || text === undefined - ? "" - : textByteLength > cap - ? Buffer.from(text, "utf8").subarray(0, cap).toString("utf8") - : text; + const stored = !capture || text === undefined ? "" : text; const node: TSArtifact = { id: "", kind: "artifact", @@ -92,7 +87,6 @@ export function inventoryArtifacts( size_bytes: raw.length, sha256: sha256(raw), source: stored, - text_truncated: capture && text !== undefined && textByteLength > cap, extraction: "none", config_keys: [], }; diff --git a/src/build/neo4j/bolt.ts b/src/build/neo4j/bolt.ts index 2e1246b..ebc1f68 100644 --- a/src/build/neo4j/bolt.ts +++ b/src/build/neo4j/bolt.ts @@ -40,32 +40,28 @@ export function shouldForceFullUpsert(dbVersion: string | null, producerVersion: } /** - * #46/#68 migration: wipe the pre-2.0.0 (schema 1.x) residue when the version gate forces a full - * upsert. Pushing 2.0.0 onto a 1.1.0 DB otherwise orphans the whole 1.x subgraph AND poisons v2 - * queries — 1.x nodes carry twin TS labels (`:Module:TSModule`, `:Symbol:TSCallable`) so they match - * v2 patterns, and a second `:Application` (keyed on name, no `id`) makes the version read - * nondeterministic. These run ONCE, before the per-module loop; they are ordered, idempotent, and - * no-op on a fresh DB. + * `--eager` purge (#116): delete THIS APPLICATION's own nodes, then repopulate from scratch. * - * They are intentionally UNANCHORED (no `:CanNode` guard on the MATCH) — that is the whole point: - * they must match the *legacy* nodes, which never carry `:CanNode`. Every v2 project-owned node DOES - * carry `:CanNode`, so the `AND NOT n:CanNode` predicate spares everything current. RETURN count so - * the caller can log what each statement removed. + * Scoped two ways at once, and both matter. `id STARTS WITH ` keeps it to this + * application, so a second app in the same database survives. `:CanNode` keeps it to nodes this + * analyzer wrote, so a sibling analyzer's graph survives — codeanalyzer-python and + * codeanalyzer-java both tag nodes with `_module` and neither applies `:CanNode`, so a predicate + * like "has _module but no :CanNode" is an exact description of THEIR nodes, not of stale ours. + * That is what the pre-2.0.0 wipe this replaces got wrong: it deleted foreign graphs. + * + * Batched, because deleting a whole application in one transaction exhausts + * `dbms.memory.transaction.total.max` on a modestly-sized server (#116, measured at 2.7 GiB). */ -export const LEGACY_WIPE_STATEMENTS: readonly string[] = [ - // 1.x project-owned nodes carried `_module` under twin labels (:Module:TSModule, …) but not :CanNode. - "MATCH (n) WHERE n._module IS NOT NULL AND NOT n:CanNode DETACH DELETE n RETURN count(n) AS wiped", - // 1.x shared nodes (externals / packages / decorators) had no `_module`; keyed on name, not id. - "MATCH (n) WHERE (n:External OR n:Package OR n:Decorator) AND NOT n:CanNode DETACH DELETE n RETURN count(n) AS wiped", - // The 1.x :Application node was keyed on `name`, so it has no `id`. - "MATCH (a:Application) WHERE a.id IS NULL DETACH DELETE a RETURN count(a) AS wiped", -]; +export const EAGER_PURGE = + "MATCH (n:CanNode) WHERE n.id STARTS WITH $prefix " + + "CALL { WITH n DETACH DELETE n } IN TRANSACTIONS OF 5000 ROWS"; export async function boltWriter( rows: GraphRows, cfg: BoltConfig, log: Logger, fullRun: boolean, + eager: boolean, ): Promise { // eslint-disable-next-line @typescript-eslint/no-explicit-any const neo4j: any = (await import("neo4j-driver")).default; @@ -107,28 +103,20 @@ export async function boltWriter( dbSchemaVersion = res.records[0]?.get("v") ?? null; }); } + // Version mismatch no longer deletes anything: a schema change forces a full re-UPSERT, and + // MERGE overwrites in place. Removing nodes is `--eager`'s job alone (#116). const forceAll = shouldForceFullUpsert(dbSchemaVersion, SCHEMA_VERSION); if (forceAll) { log.info( `neo4j(bolt): schema ${dbSchemaVersion ?? "(none)"} → ${SCHEMA_VERSION}, full upsert forced`, ); - // Detect-and-wipe the pre-2.0.0 subgraph in one step, BEFORE any new write, so stale twin-label - // nodes can't survive to poison v2 queries or the content-hash diff. Idempotent on fresh DBs. - const counts: number[] = []; - await withSession(session, async (s) => { - for (const stmt of LEGACY_WIPE_STATEMENTS) { - const res = await s.run(stmt); - const c = res.records[0]?.get("wiped"); - counts.push(typeof c?.toNumber === "function" ? c.toNumber() : Number(c ?? 0)); - } - }); - const wiped = counts.reduce((a, b) => a + b, 0); - if (wiped > 0) { - log.info( - `neo4j(bolt): wiped ${wiped} legacy (pre-2.0.0) nodes ` + - `(module=${counts[0]}, shared=${counts[1]}, app=${counts[2]})`, - ); - } + } + + // --eager: drop this application's own nodes and rebuild. Without it the push only ever adds + // and updates -- managing the database's lifetime is the operator's call, not the analyzer's. + if (eager && appId !== null) { + await withSession(session, (s) => s.run(EAGER_PURGE, { prefix: appId })); + log.info(`neo4j(bolt): --eager, purged the existing graph for ${appId}`); } // 3. diff content_hash. @@ -154,18 +142,21 @@ export async function boltWriter( for (const m of changed) { const nodes = byModule.get(m)!; const keys = nodes.map((n) => n.value); - await withSession(session, async (s) => { - await s.executeWrite(async (tx: any) => { - // Anchor on :CanNode so these seek the module's index slice instead of scanning the whole - // store (and never touch non-CanNode nodes). The `x.id IS NULL` guard defends the sweep - // against a null key — three-valued logic would otherwise drop the row from `NOT x.id IN`. - await tx.run(`MATCH (x:CanNode {_module: $m})-[r]->() DELETE r`, { m }); - await tx.run( - `MATCH (x:CanNode {_module: $m}) WHERE x.id IS NULL OR NOT x.id IN $keys DETACH DELETE x`, - { m, keys }, - ); + // Only --eager removes a module's vanished declarations. A default push MERGEs current + // nodes over the old ones and leaves anything no longer emitted in place: deleting is the + // operator's call (#116). Anchored on :CanNode either way, so a sibling analyzer's nodes + // sharing this `_module` key are never in scope. + if (eager) { + await withSession(session, async (s) => { + await s.executeWrite(async (tx: any) => { + await tx.run(`MATCH (x:CanNode {_module: $m})-[r]->() DELETE r`, { m }); + await tx.run( + `MATCH (x:CanNode {_module: $m}) WHERE x.id IS NULL OR NOT x.id IN $keys DETACH DELETE x`, + { m, keys }, + ); + }); }); - }); + } await upsertNodes(session, neo4j, nodes); } @@ -177,19 +168,23 @@ export async function boltWriter( await upsertEdges(session, neo4j, edges); // 7. orphan prune — only safe on a full run (a targeted run can't tell deleted from untargeted). - if (fullRun) { + // appId === null would make `STARTS WITH ""` match every node in the store. + if (fullRun && eager && appId !== null) { const present = [...byModule.keys()]; await withSession(session, async (s) => { + // Anchored on :CanNode AND this app's id prefix, same as EAGER_PURGE. `MATCH (m:TSModule)` + // alone would reach a SECOND TypeScript application in the same database -- every one of + // its modules is "not in this app's $present" -- and any 1.x twin-labelled node too (#116). const res = await s.run( - `MATCH (m:TSModule) WHERE NOT m._module IN $present ` + - `OPTIONAL MATCH (m)-${DESCENDANTS}->(x) DETACH DELETE x, m RETURN count(m) AS pruned`, - { present }, + `MATCH (m:TSModule:CanNode) WHERE m.id STARTS WITH $prefix AND NOT m._module IN $present ` + + `OPTIONAL MATCH (m)-${DESCENDANTS}->(x) DETACH DELETE x, m RETURN count(DISTINCT m) AS pruned`, + { present, prefix: appId }, ); const pruned = res.records[0]?.get("pruned") ?? 0; log.info(`neo4j(bolt): pruned ${pruned} vanished module(s)`); }); } else { - log.info("neo4j(bolt): targeted run — orphan pruning skipped (deleted files not removed)"); + log.info("neo4j(bolt): orphan pruning skipped (use --eager to remove vanished modules)"); } } finally { await driver.close(); diff --git a/src/build/neo4j/project.ts b/src/build/neo4j/project.ts index 6e16c78..a779c6d 100644 --- a/src/build/neo4j/project.ts +++ b/src/build/neo4j/project.ts @@ -81,6 +81,10 @@ export function project(app: TSAnalysis, _appName?: string): GraphRows { id: art.id, kind: "artifact", path: art.path, format: art.format, roles: art.roles.length ? art.roles : null, size_bytes: art.size_bytes, sha256: art.sha256, extraction: art.extraction, + // `source` belongs on the graph: python has carried it on :Artifact since it shipped the + // layer, and a consumer reading the same neutral :Artifact node from two analyzers must not + // get the text from one and nothing from the other. `--no-artifact-text` still empties it. + source: art.source, })); b.edge("HAS_ARTIFACT", appRef, aRef); for (const ck of art.config_keys) { diff --git a/src/build/neo4j/schema.ts b/src/build/neo4j/schema.ts index 9f32c2a..984ddfe 100644 --- a/src/build/neo4j/schema.ts +++ b/src/build/neo4j/schema.ts @@ -76,6 +76,7 @@ export const NODE_LABELS: NodeLabel[] = [ properties: { id: "string", kind: "string", path: "string", format: "string", roles: "string[]", size_bytes: "integer", sha256: "string", extraction: "string", + source: "string", }, }, { diff --git a/src/cli.ts b/src/cli.ts index f0de69c..dfce972 100644 --- a/src/cli.ts +++ b/src/cli.ts @@ -1,7 +1,6 @@ import * as path from "node:path"; import { Command, Option } from "commander"; import type { AnalysisOptions, EmitTarget } from "./options"; -import { DEFAULT_ARTIFACT_TEXT_MAX_BYTES } from "./options"; import { ALL_GRAPHS, type GraphSelector } from "./schema"; /** @@ -60,11 +59,6 @@ export function buildProgram(): Command { .option("--no-phantoms", "disable phantom (external) nodes for imported/required library calls") .option("--resolve-installed", "probe node_modules metadata for import→package binding (default: repo files only)") .option("--no-artifact-text", "keep the artifact inventory but drop captured raw text") - .option( - "--artifact-text-max-bytes ", - "per-file byte cap for captured artifact text; larger files are truncated and flagged", - String(DEFAULT_ARTIFACT_TEXT_MAX_BYTES), - ) .option("-c, --cache-dir ", "cache/intermediate directory") .option("-v, --verbose", "increase verbosity (repeatable)", (_v: string, prev: number) => prev + 1, 0) .allowExcessArguments(true); @@ -160,15 +154,6 @@ export function parseArgs(argv: string[]): AnalysisOptions { phantoms: o.phantoms !== false, resolveInstalled: Boolean(o.resolveInstalled), artifactText: o.artifactText !== false, - // Malformed input (e.g. "abc") must fall back, not silently disable truncation via NaN -- - // every `> cap` comparison against NaN is false. - artifactTextMaxBytes: (() => { - // An empty value is malformed too -- Number("") is 0, which would cap every - // artifact's text at zero bytes. An explicit 0 still means exactly that. - const raw = String(o.artifactTextMaxBytes ?? "").trim(); - const n = raw === "" ? NaN : Number(raw); - return Number.isFinite(n) && n >= 0 ? n : DEFAULT_ARTIFACT_TEXT_MAX_BYTES; - })(), cacheDir: o.cacheDir ? path.resolve(String(o.cacheDir)) : null, verbosity: typeof o.verbose === "number" ? o.verbose : 0, }; diff --git a/src/options/options.ts b/src/options/options.ts index 7829245..2ee9c81 100644 --- a/src/options/options.ts +++ b/src/options/options.ts @@ -2,7 +2,6 @@ import type { GraphSelector } from "../schema"; export type EmitTarget = "json" | "neo4j" | "schema"; /** Default per-file byte cap for captured artifact text (256 KiB, python v1.3.0 parity). */ -export const DEFAULT_ARTIFACT_TEXT_MAX_BYTES = 256 * 1024; /** Normalized analysis options (produced by the CLI layer, consumed by core). */ export interface AnalysisOptions { diff --git a/src/schema/schema.ts b/src/schema/schema.ts index d280181..5ec034e 100644 --- a/src/schema/schema.ts +++ b/src/schema/schema.ts @@ -375,7 +375,6 @@ export interface TSArtifact { size_bytes: number; sha256: string; source: string; // verbatim, unbounded by decision (spec §3) - text_truncated: boolean; // true when `source` is a prefix, not the full file extraction: "none" | "partial" | "full"; config_keys: TSConfigKey[]; // contained children; containment mirrors DEFINES_CONFIG } diff --git a/src/utils/serialize.ts b/src/utils/serialize.ts index 95b9989..5f685ae 100644 --- a/src/utils/serialize.ts +++ b/src/utils/serialize.ts @@ -58,6 +58,7 @@ async function emitNeo4j(application: TSAnalysis, opts: AnalysisOptions): Promis }, log, opts.targetFiles === null, // full run ⇒ orphan pruning is safe + opts.eager, // --eager ⇒ purge this app's graph and rebuild; otherwise never delete ); return; } diff --git a/test/artifacts.test.ts b/test/artifacts.test.ts index b408624..c709854 100644 --- a/test/artifacts.test.ts +++ b/test/artifacts.test.ts @@ -167,16 +167,16 @@ describe("level-invariance + determinism (#101)", () => { expect(a).toBe(b); }); - test("text capture: on by default, truncates under the cap, hash stays full-file", async () => { - const full = (await analyze(options())).application.application.artifacts["README.md"]; - expect(full?.source.length).toBeGreaterThan(0); - expect(full?.text_truncated).toBe(false); - - const capped = (await analyze(options({ artifactTextMaxBytes: 8 }))).application.application.artifacts["README.md"]; - expect(capped?.text_truncated).toBe(true); - expect(capped!.source.length).toBeLessThanOrEqual(8); - expect(capped?.sha256).toBe(full?.sha256); // hash is of the FULL file - expect(capped?.size_bytes).toBe(full?.size_bytes); + // No byte cap (#116): a file is captured whole or not at all. A truncated `source` is a prefix + // that reads like a complete file, so every consumer then needs a flag to tell the two apart. + test("text capture: on by default, always the WHOLE file, hash matches", async () => { + const app = (await analyze(options())).application.application; + const readme = app.artifacts["README.md"]; + expect(readme?.source.length).toBeGreaterThan(0); + // The stored text is the entire file, byte for byte -- not a prefix. + const onDisk = fs.readFileSync(path.join(FIXTURE, "README.md"), "utf8"); + expect(readme!.source).toBe(onDisk); + expect(Buffer.byteLength(readme!.source, "utf8")).toBe(readme!.size_bytes); }); test("--no-artifact-text drops source but keeps inventory AND extraction", async () => { @@ -185,34 +185,6 @@ describe("level-invariance + determinism (#101)", () => { expect(a.dependencies.find((d) => d.name === "express")?.locked_version).toBe("4.19.2"); }); - test("text cap is byte-accurate on multi-byte UTF-8 (not character-count)", async () => { - // Create a test .md file with multi-byte chars to verify cap is byte-accurate, not char-count. - // "Hi 🎉": H=1 byte, i=1 byte, space=1 byte, emoji=4 bytes (UTF-8) = 7 bytes total, 5 UTF-16 code units. - const testFile = path.join(FIXTURE, "multi-byte-test.md"); - const fullContent = "Hi 🎉"; - fs.writeFileSync(testFile, fullContent, "utf8"); - - // Test 1: cap=6 bytes. Discriminates: old char-slicing counts UTF-16 units (text.length=5, which is NOT > 6), - // so text.slice(0,6) incorrectly returns full "Hi 🎉" (7 bytes, FAILS <= 6). New byte-slicing correctly - // compares Buffer.byteLength (7 > 6), truncates to 6 bytes (replacement char boundary), and PASSES <= 6. - const r1 = await analyze(options({ artifactTextMaxBytes: 6 })); - const art1 = r1.application.application.artifacts["multi-byte-test.md"]; - expect(art1?.text_truncated).toBe(true); - expect(Buffer.byteLength(art1!.source, "utf8")).toBeLessThanOrEqual(6); - expect(art1!.source).not.toBe(fullContent); - - // Test 2: cap=7 bytes (exact full byte length). Verifies > boundary (not >=): - // text_truncated must be false and source must be intact at exact cap. - const r2 = await analyze(options({ artifactTextMaxBytes: 7 })); - const art2 = r2.application.application.artifacts["multi-byte-test.md"]; - expect(art2?.text_truncated).toBe(false); - expect(art2!.source).toBe(fullContent); - expect(art2?.sha256).toBeDefined(); // hash is always full-file - expect(art2?.size_bytes).toBe(7); // size is full-file (7 bytes) - - // Clean up - fs.unlinkSync(testFile); - }); }); describe("Neo4j projection — neutral :Artifact/:Package (#101)", () => { @@ -222,7 +194,10 @@ describe("Neo4j projection — neutral :Artifact/:Package (#101)", () => { const art = rows.nodes.find((n) => n.value === "can://artifact/artifacts-app/package.json"); expect(art?.labels).toEqual(["Artifact"]); expect(art?.props["roles"]).toEqual(["dependency-manifest", "tool-config"]); - expect(art?.props["source"]).toBeUndefined(); // text stays off the graph + // Artifact text belongs on the graph: python has carried `source` on :Artifact since it + // shipped the layer, so a consumer reading the same neutral node from two analyzers must not + // get text from one and nothing from the other (#116). + expect(art?.props["source"]).toBeDefined(); const pkg = rows.nodes.find((n) => n.value === "pkg:npm/react"); expect(pkg?.labels).toEqual(["Package"]); const scoped = rows.nodes.find((n) => n.value === "pkg:npm/%40scope/util"); diff --git a/test/bolt-version-gate.test.ts b/test/bolt-version-gate.test.ts index c1f3507..1cbe3ab 100644 --- a/test/bolt-version-gate.test.ts +++ b/test/bolt-version-gate.test.ts @@ -1,5 +1,5 @@ import { describe, expect, test } from "bun:test"; -import { LEGACY_WIPE_STATEMENTS, shouldForceFullUpsert } from "../src/build/neo4j/bolt"; +import { EAGER_PURGE, shouldForceFullUpsert } from "../src/build/neo4j/bolt"; describe("bolt version gate (#68)", () => { test("mismatch or absent stored version forces a full upsert", () => { @@ -9,29 +9,25 @@ describe("bolt version gate (#68)", () => { }); }); -describe("legacy (pre-2.0.0) wipe (#46)", () => { - test("wipes the three legacy node classes, in order, unanchored but CanNode-guarded", () => { - expect(LEGACY_WIPE_STATEMENTS).toHaveLength(3); +describe("--eager purge is scoped to this analyzer AND this app (#116)", () => { + // The statement this replaces was `MATCH (n) WHERE n._module IS NOT NULL AND NOT n:CanNode + // DETACH DELETE n`, deliberately unanchored so it would reach pre-2.0.0 nodes. But + // codeanalyzer-python and codeanalyzer-java both set `_module` and neither applies `:CanNode`, + // so that predicate described THEIR nodes exactly: pointing cants at a shared database deleted + // the python and java graphs. It only failed loudly because the delete exhausted transaction + // memory and rolled back. + test("anchors on :CanNode, so a sibling analyzer's nodes can never match", () => { + expect(EAGER_PURGE).toContain("MATCH (n:CanNode)"); + // The lethal shape: reaching nodes by the ABSENCE of our own marker. + expect(EAGER_PURGE).not.toContain("NOT n:CanNode"); + expect(EAGER_PURGE).not.toContain("_module IS NOT NULL"); + }); - // (1) project-owned twin-label nodes: matched by `_module`, spared iff :CanNode. - expect(LEGACY_WIPE_STATEMENTS[0]).toBe( - "MATCH (n) WHERE n._module IS NOT NULL AND NOT n:CanNode DETACH DELETE n RETURN count(n) AS wiped", - ); - // (2) 1.x shared nodes (no `_module`): externals / packages / decorators, spared iff :CanNode. - expect(LEGACY_WIPE_STATEMENTS[1]).toBe( - "MATCH (n) WHERE (n:External OR n:Package OR n:Decorator) AND NOT n:CanNode DETACH DELETE n RETURN count(n) AS wiped", - ); - // (3) the 1.x :Application node, keyed on name → no `id`. - expect(LEGACY_WIPE_STATEMENTS[2]).toBe( - "MATCH (a:Application) WHERE a.id IS NULL DETACH DELETE a RETURN count(a) AS wiped", - ); + test("anchors on the application id, so another app in the same database survives", () => { + expect(EAGER_PURGE).toContain("n.id STARTS WITH $prefix"); + }); - // Each is intentionally UNANCHORED on the MATCH (no :CanNode label) — it must reach legacy nodes — - // yet every one guards current v2 data with an explicit `NOT n:CanNode` / `a.id IS NULL` predicate. - for (const stmt of LEGACY_WIPE_STATEMENTS) { - expect(stmt).not.toContain("MATCH (n:CanNode"); - expect(stmt).toContain("DETACH DELETE"); - expect(stmt).toContain("RETURN count("); - } + test("deletes in batches — one transaction over a whole app exhausts the memory cap", () => { + expect(EAGER_PURGE).toContain("IN TRANSACTIONS OF"); }); }); diff --git a/test/cli-options.test.ts b/test/cli-options.test.ts deleted file mode 100644 index 176c565..0000000 --- a/test/cli-options.test.ts +++ /dev/null @@ -1,23 +0,0 @@ -import { describe, expect, test } from "bun:test"; -import { parseArgs } from "../src/cli"; -import { DEFAULT_ARTIFACT_TEXT_MAX_BYTES } from "../src/options"; - -const cap = (...extra: string[]): number => - parseArgs(["--input", ".", ...extra]).artifactTextMaxBytes; - -describe("--artifact-text-max-bytes", () => { - test("defaults when absent", () => { - expect(cap()).toBe(DEFAULT_ARTIFACT_TEXT_MAX_BYTES); - }); - - // A bad value must not silently disable truncation: NaN loses every `> cap` - // comparison, and Number("") is 0, which would empty every artifact's source. - test.each([["abc"], [""], [" "], ["-1"]])("falls back on %p", (bad) => { - expect(cap("--artifact-text-max-bytes", bad)).toBe(DEFAULT_ARTIFACT_TEXT_MAX_BYTES); - }); - - test("honors a valid value, 0 included", () => { - expect(cap("--artifact-text-max-bytes", "4096")).toBe(4096); - expect(cap("--artifact-text-max-bytes", "0")).toBe(0); - }); -}); diff --git a/test/neo4j-bolt.test.ts b/test/neo4j-bolt.test.ts index 9015a52..9c33493 100644 --- a/test/neo4j-bolt.test.ts +++ b/test/neo4j-bolt.test.ts @@ -93,7 +93,7 @@ containerSuite("neo4j bolt writer", () => { async () => { const opts = optsFor(); const rows = project((await analyze(opts)).application); - await boltWriter(rows, cfg, log, true); + await boltWriter(rows, cfg, log, true, false); // Every projected node/edge lands (the fixture has no library deps, so endpoints all resolve). expect(await num("MATCH (n) RETURN count(n)")).toBe(rows.nodes.length); @@ -133,7 +133,7 @@ containerSuite("neo4j bolt writer", () => { async () => { const opts = optsFor(); const rows = project((await analyze(opts)).application); - await boltWriter(rows, cfg, log, true); + await boltWriter(rows, cfg, log, true, false); expect(await num("MATCH (n) RETURN count(n)")).toBe(rows.nodes.length); expect(await num("MATCH ()-[r]->() RETURN count(r)")).toBe(rows.edges.length); }, @@ -141,7 +141,7 @@ containerSuite("neo4j bolt writer", () => { ); test( - "a full run prunes a module whose source vanished", + "a vanished module is pruned only under --eager (#116)", async () => { const opts = optsFor(); const result = await analyze(opts); @@ -150,9 +150,13 @@ containerSuite("neo4j bolt writer", () => { delete app.symbol_table[victim]; const rows = project(finalizeAnalysis(app, result.program_graphs ?? null, opts).application); - await boltWriter(rows, cfg, log, true); - // The victim's nodes are gone. + // Default push: deletion is the operator's call, so the vanished module's nodes stay. + await boltWriter(rows, cfg, log, true, false); + expect(await num("MATCH (n {_module:$m}) RETURN count(n)", { m: victim })).toBeGreaterThan(0); + + // --eager: purge this application and rebuild, so the vanished module goes. + await boltWriter(rows, cfg, log, true, true); expect(await num("MATCH (n {_module:$m}) RETURN count(n)", { m: victim })).toBe(0); // The surviving module-scoped graph matches the reduced projection. (Shared :TSExternal @@ -164,7 +168,7 @@ containerSuite("neo4j bolt writer", () => { ); test( - "migrates a 1.1.0-shaped graph to the current schema, wiping legacy residue (#46)", + "a 1.x graph in the same store is left alone, not wiped (#116)", async () => { // Seed a minimal schema-1.1.0 graph on a clean store: twin labels, the old // name/file_key/signature keys, and an :Application keyed on `name` (no `id`). @@ -180,28 +184,31 @@ containerSuite("neo4j bolt writer", () => { await seed.close(); } - // A full current-version push against the same DB must detect the mismatch and wipe the residue. + // The seed stands in for ANY foreign data: nodes carrying `_module` without `:CanNode`. That + // is also an exact description of a codeanalyzer-python or codeanalyzer-java graph, which is + // why the old "wipe the residue" behaviour deleted sibling analyzers' work (#116). const opts = optsFor(); const rows = project((await analyze(opts)).application); - await boltWriter(rows, cfg, log, true); + await boltWriter(rows, cfg, log, true, false); - // Exactly one :Application survives — the fresh v2 one (id set, version bumped). The 1.x app, - // keyed on name with no id, was wiped, so the version read is no longer nondeterministic. - expect(await num("MATCH (a:Application) RETURN count(a)")).toBe(1); + // The 1.x nodes survive. We do not delete what we cannot prove we wrote. + expect( + await num("MATCH (n) WHERE n._module IS NOT NULL AND NOT n:CanNode RETURN count(n)"), + ).toBe(2); + + // Two :Application nodes now coexist, and that is fine: the version read is scoped by id, + // so it is deterministic regardless of what else shares the store. expect( await num( `MATCH (a:Application) WHERE a.id IS NOT NULL AND a.schema_version = '${SCHEMA_VERSION}' RETURN count(a)`, ), ).toBe(1); - // No legacy twin-label residue remains (would-be poison for v2 label queries). + // Even --eager spares them: the purge is anchored on :CanNode AND this app's id prefix. + await boltWriter(rows, cfg, log, true, true); expect( await num("MATCH (n) WHERE n._module IS NOT NULL AND NOT n:CanNode RETURN count(n)"), - ).toBe(0); - - // The stale 'x.ts' :TSModule seed did not survive as a duplicate — exactly the fixture's modules. - const fixtureModules = rows.nodes.filter((n) => n.labels.includes("TSModule")).length; - expect(await num("MATCH (m:TSModule) RETURN count(m)")).toBe(fixtureModules); + ).toBe(2); }, 120_000, );