Skip to content

fix(neo4j)!: stop deleting other analyzers' graphs; artifact text whole and on the graph - #117

Merged
rahlk merged 3 commits into
mainfrom
fix/issue-116-neo4j-scoping
Sep 2, 2026
Merged

fix(neo4j)!: stop deleting other analyzers' graphs; artifact text whole and on the graph#117
rahlk merged 3 commits into
mainfrom
fix/issue-116-neo4j-scoping

Conversation

@rahlk

@rahlk rahlk commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Closes #116.

The bug behind the reported crash

#116 was filed as "Neo4j transaction too big". The out-of-memory error is real, but it is a
symptom — and it is what prevented data loss rather than causing it.

Before every write whose stored schema version did not match, the writer ran:

MATCH (n) WHERE n._module IS NOT NULL AND NOT n:CanNode DETACH DELETE n

:CanNode is a label only this analyzer applies, so "has _module, lacks :CanNode" was read as
"stale TypeScript". It is not. codeanalyzer-python sets _module on four node types
(neo4j/schema.py:91,107,131,160) and codeanalyzer-java sets it in GraphProjector.java:256,273
— and neither applies :CanNode. That predicate is an exact description of their nodes.

So pointing cants at a shared Neo4j deleted the python and java graphs before writing anything of
its own. It only failed loudly because the delete exhausted dbms.memory.transaction.total.max and
rolled back. A smaller foreign graph, or a roomier server, and it would have succeeded silently.

What changes

Deletion is the operator's call, not the analyzer's.

  • Default push — MERGE current nodes over whatever is there and delete nothing. The per-module
    purge and the orphan prune are now --eager-only, and a schema-version mismatch forces a full
    re-UPSERT instead of a wipe.

  • --eager — purge this application's graph and rebuild, scoped twice:

    MATCH (n:CanNode) WHERE n.id STARTS WITH $prefix
    CALL { WITH n DETACH DELETE n } IN TRANSACTIONS OF 5000 ROWS

    :CanNode keeps it to nodes this analyzer wrote; the id prefix keeps it to this application, so
    a second app in the same database survives; batched, because one transaction over a whole
    application is what hit the cap in the first place.

The tradeoff, stated plainly: declarations removed from a source file now linger until someone runs
--eager. A stale node is visible and recoverable; another tool's silently deleted edges are
neither. codeanalyzer-java reaches the same conclusion for its snapshot path in
CypherWriter.java:40-52.

:Artifact now carries source and text_truncatedsource. Python has emitted artifact text
since it shipped the layer (neo4j/schema.py:213); omitting it here meant a consumer reading the
same neutral :Artifact node got text from python and nothing from TypeScript.

Artifact text is captured whole — the byte cap is gone. Removed the 256 KiB cap,
--artifact-text-max-bytes, and the text_truncated flag. A truncated source reads exactly like
a complete small file, so it forces every consumer to carry a flag to tell them apart — and the flag
could not even do that, since text_truncated: false covered both "complete" and "capture
disabled". On vscode it bought little: 32 of 4,953 artifacts were clipped. --no-artifact-text
remains, as the one unambiguous opt-out.

BREAKING

Verification

Against a live Neo4j capped at the reporter's exact 2.7 GiB, with a python graph planted
alongside a 1,841-module / 124,866-node superset-frontend graph:

python nodes outcome
default push survive graph updated in place
--eager push survive this app purged and rebuilt

Previously either push would have deleted them. Suite: 232 pass / 6 skip / 0 fail; typecheck,
build, gen:schema, gen:readme all clean.

I also removed a test that had codified the bug as intent — it asserted the wipe was
"intentionally UNANCHORED" and looped over the statements asserting
not.toContain("MATCH (n:CanNode"). It is replaced by three tests asserting the inverse safety
property, including an explicit not.toContain("NOT n:CanNode") on the lethal shape.

Follow-ups, not in this PR

… on --eager

The pre-2.0.0 cleanup ran `MATCH (n) WHERE n._module IS NOT NULL AND NOT
n:CanNode DETACH DELETE n` before every write whose stored schema version did
not match. `:CanNode` is a label only this analyzer applies, so "has _module,
lacks :CanNode" was read as "stale TypeScript". It is not: codeanalyzer-python
sets `_module` on four node types (neo4j/schema.py:91,107,131,160) and
codeanalyzer-java sets it in GraphProjector.java:256,273, and NEITHER applies
:CanNode. That predicate describes their nodes exactly, so pointing cants at a
shared Neo4j deleted the python and java graphs before writing anything.

It surfaced as #116 -- "transaction too big" -- only because the delete
exhausted dbms.memory.transaction.total.max and rolled back. A smaller foreign
graph, or a roomier server, and it would have succeeded silently.

Deletion is now the operator's call, not the analyzer's:

  - default: MERGE current nodes over whatever is there, delete nothing. The
    per-module purge and the orphan prune are --eager-only, and a schema
    version mismatch forces a full re-UPSERT rather than a wipe.
  - --eager: purge this application's graph and rebuild, scoped twice --
    `MATCH (n:CanNode) WHERE n.id STARTS WITH $prefix` keeps it to nodes this
    analyzer wrote AND to this application, so both a sibling analyzer and a
    second app in the same database survive. Batched via CALL { } IN
    TRANSACTIONS, since one transaction over a whole app is what hit the cap.

Also: :Artifact now carries `source` and `text_truncated`. Python has emitted
them since it shipped the layer (neo4j/schema.py:213); dropping them here meant
a consumer reading the same NEUTRAL :Artifact node got text from python and
nothing from TypeScript. --no-artifact-text still empties it.

Verified on a live Neo4j against superset-frontend (1,841 modules, 124,866
nodes) with a python graph planted alongside: the foreign nodes survive both a
default push and an --eager push. The replaced test had codified the bug as
intent -- it asserted the wipe was "intentionally UNANCHORED" and looped
asserting `not.toContain("MATCH (n:CanNode")`.
An artifact's `source` is now the complete file, or `""` under
--no-artifact-text. Removed: the 256 KiB cap, --artifact-text-max-bytes, and
the `text_truncated` flag it needed.

A truncated `source` is a prefix that reads exactly like a complete small file,
so it forces every consumer to carry a flag to tell the two apart -- and the
flag could not even do that job, since `text_truncated: false` covered both
"complete" and "capture disabled". Measured on vscode it bought little: 32 of
4,953 artifacts were clipped.

This diverges from codeanalyzer-python, which caps at 262144 and reports the
flag. Filed there separately.

BREAKING: `text_truncated` is gone from `analysis.json` and from the Neo4j
:Artifact node; --artifact-text-max-bytes is no longer accepted.
The prune ran `MATCH (m:TSModule) WHERE NOT m._module IN $present`, with no
:CanNode anchor and no application scoping. Two consequences:

  - a SECOND TypeScript application in the same database is pruned by the
    first app's --eager push, because every one of its modules is "not in this
    app's $present";
  - a 1.x twin-labelled (:Module:TSModule) node matches too.

Now anchored like EAGER_PURGE -- `:TSModule:CanNode` plus
`id STARTS WITH $prefix` -- and skipped entirely when appId is null, which
would otherwise make `STARTS WITH ""` match the whole store.

Also `count(m)` -> `count(DISTINCT m)`: OPTIONAL MATCH over the descendants
pattern repeats `m` once per PATH, so the log line reported 116,024 modules
pruned on a database holding 1,841.

Caught by the container suite, which the earlier commits had not been run
against: these tests only execute under RUN_CONTAINER_TESTS=1 and were
sitting in the 6 "skip" count. Two of them asserted the behaviour this branch
removes and now assert its replacement -- notably that a graph carrying
`_module` without `:CanNode` (the exact shape of a python or java graph)
survives both a default push AND an --eager push.

Full suite with containers enabled: 236 pass, 0 fail, 0 skip.
@rahlk
rahlk merged commit c1c27f3 into main Sep 2, 2026
1 check passed
@rahlk
rahlk deleted the fix/issue-116-neo4j-scoping branch September 2, 2026 15:09
rahlk added a commit that referenced this pull request Sep 3, 2026
ANALYZER_VERSION moves with package.json because it is the cache-invalidation key
(utils/cache.ts): this release changes artifact text capture (#117) and per-module
id namespacing (#115), so a warm 1.1.0 cache must not be reused against it.

Neo4j schema contract is 2.0.0 as of #144 — collapsed back from 2.1.0, since
intermediate 2.x versions are not meaningful until every analyzer re-baselines
together.

Released as a MINOR despite three breaking-marked commits (#115, #117, #144), by
explicit decision. The release notes lead with those breaks and their migrations.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Neo4j transaction too big

1 participant