fix(neo4j)!: stop deleting other analyzers' graphs; artifact text whole and on the graph - #117
Merged
Conversation
… on --eager The pre-2.0.0 cleanup ran `MATCH (n) WHERE n._module IS NOT NULL AND NOT n:CanNode DETACH DELETE n` before every write whose stored schema version did not match. `:CanNode` is a label only this analyzer applies, so "has _module, lacks :CanNode" was read as "stale TypeScript". It is not: codeanalyzer-python sets `_module` on four node types (neo4j/schema.py:91,107,131,160) and codeanalyzer-java sets it in GraphProjector.java:256,273, and NEITHER applies :CanNode. That predicate describes their nodes exactly, so pointing cants at a shared Neo4j deleted the python and java graphs before writing anything. It surfaced as #116 -- "transaction too big" -- only because the delete exhausted dbms.memory.transaction.total.max and rolled back. A smaller foreign graph, or a roomier server, and it would have succeeded silently. Deletion is now the operator's call, not the analyzer's: - default: MERGE current nodes over whatever is there, delete nothing. The per-module purge and the orphan prune are --eager-only, and a schema version mismatch forces a full re-UPSERT rather than a wipe. - --eager: purge this application's graph and rebuild, scoped twice -- `MATCH (n:CanNode) WHERE n.id STARTS WITH $prefix` keeps it to nodes this analyzer wrote AND to this application, so both a sibling analyzer and a second app in the same database survive. Batched via CALL { } IN TRANSACTIONS, since one transaction over a whole app is what hit the cap. Also: :Artifact now carries `source` and `text_truncated`. Python has emitted them since it shipped the layer (neo4j/schema.py:213); dropping them here meant a consumer reading the same NEUTRAL :Artifact node got text from python and nothing from TypeScript. --no-artifact-text still empties it. Verified on a live Neo4j against superset-frontend (1,841 modules, 124,866 nodes) with a python graph planted alongside: the foreign nodes survive both a default push and an --eager push. The replaced test had codified the bug as intent -- it asserted the wipe was "intentionally UNANCHORED" and looped asserting `not.toContain("MATCH (n:CanNode")`.
An artifact's `source` is now the complete file, or `""` under --no-artifact-text. Removed: the 256 KiB cap, --artifact-text-max-bytes, and the `text_truncated` flag it needed. A truncated `source` is a prefix that reads exactly like a complete small file, so it forces every consumer to carry a flag to tell the two apart -- and the flag could not even do that job, since `text_truncated: false` covered both "complete" and "capture disabled". Measured on vscode it bought little: 32 of 4,953 artifacts were clipped. This diverges from codeanalyzer-python, which caps at 262144 and reports the flag. Filed there separately. BREAKING: `text_truncated` is gone from `analysis.json` and from the Neo4j :Artifact node; --artifact-text-max-bytes is no longer accepted.
The prune ran `MATCH (m:TSModule) WHERE NOT m._module IN $present`, with no
:CanNode anchor and no application scoping. Two consequences:
- a SECOND TypeScript application in the same database is pruned by the
first app's --eager push, because every one of its modules is "not in this
app's $present";
- a 1.x twin-labelled (:Module:TSModule) node matches too.
Now anchored like EAGER_PURGE -- `:TSModule:CanNode` plus
`id STARTS WITH $prefix` -- and skipped entirely when appId is null, which
would otherwise make `STARTS WITH ""` match the whole store.
Also `count(m)` -> `count(DISTINCT m)`: OPTIONAL MATCH over the descendants
pattern repeats `m` once per PATH, so the log line reported 116,024 modules
pruned on a database holding 1,841.
Caught by the container suite, which the earlier commits had not been run
against: these tests only execute under RUN_CONTAINER_TESTS=1 and were
sitting in the 6 "skip" count. Two of them asserted the behaviour this branch
removes and now assert its replacement -- notably that a graph carrying
`_module` without `:CanNode` (the exact shape of a python or java graph)
survives both a default push AND an --eager push.
Full suite with containers enabled: 236 pass, 0 fail, 0 skip.
This was referenced Sep 2, 2026
This was referenced Sep 2, 2026
Merged
rahlk
added a commit
that referenced
this pull request
Sep 3, 2026
ANALYZER_VERSION moves with package.json because it is the cache-invalidation key (utils/cache.ts): this release changes artifact text capture (#117) and per-module id namespacing (#115), so a warm 1.1.0 cache must not be reused against it. Neo4j schema contract is 2.0.0 as of #144 — collapsed back from 2.1.0, since intermediate 2.x versions are not meaningful until every analyzer re-baselines together. Released as a MINOR despite three breaking-marked commits (#115, #117, #144), by explicit decision. The release notes lead with those breaks and their migrations.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #116.
The bug behind the reported crash
#116 was filed as "Neo4j transaction too big". The out-of-memory error is real, but it is a
symptom — and it is what prevented data loss rather than causing it.
Before every write whose stored schema version did not match, the writer ran:
:CanNodeis a label only this analyzer applies, so "has_module, lacks:CanNode" was read as"stale TypeScript". It is not. codeanalyzer-python sets
_moduleon four node types(
neo4j/schema.py:91,107,131,160) and codeanalyzer-java sets it inGraphProjector.java:256,273— and neither applies
:CanNode. That predicate is an exact description of their nodes.So pointing
cantsat a shared Neo4j deleted the python and java graphs before writing anything ofits own. It only failed loudly because the delete exhausted
dbms.memory.transaction.total.maxandrolled back. A smaller foreign graph, or a roomier server, and it would have succeeded silently.
What changes
Deletion is the operator's call, not the analyzer's.
Default push — MERGE current nodes over whatever is there and delete nothing. The per-module
purge and the orphan prune are now
--eager-only, and a schema-version mismatch forces a fullre-UPSERT instead of a wipe.
--eager— purge this application's graph and rebuild, scoped twice::CanNodekeeps it to nodes this analyzer wrote; the id prefix keeps it to this application, soa second app in the same database survives; batched, because one transaction over a whole
application is what hit the cap in the first place.
The tradeoff, stated plainly: declarations removed from a source file now linger until someone runs
--eager. A stale node is visible and recoverable; another tool's silently deleted edges areneither. codeanalyzer-java reaches the same conclusion for its snapshot path in
CypherWriter.java:40-52.:Artifactnow carriessourceandtext_truncated→source. Python has emitted artifact textsince it shipped the layer (
neo4j/schema.py:213); omitting it here meant a consumer reading thesame neutral
:Artifactnode got text from python and nothing from TypeScript.Artifact text is captured whole — the byte cap is gone. Removed the 256 KiB cap,
--artifact-text-max-bytes, and thetext_truncatedflag. A truncatedsourcereads exactly likea complete small file, so it forces every consumer to carry a flag to tell them apart — and the flag
could not even do that, since
text_truncated: falsecovered both "complete" and "capturedisabled". On vscode it bought little: 32 of 4,953 artifacts were clipped.
--no-artifact-textremains, as the one unambiguous opt-out.
BREAKING
--artifact-text-max-bytesis no longer accepted;text_truncatedis gone fromanalysis.jsonand from the
:Artifactnode.--eager.(Artifact text: drop the byte cap and text_truncated, to match typescript codeanalyzer-python#172, Artifact text: drop the byte cap and text_truncated, to match typescript codeanalyzer-java#214) so all three land
on one answer.
Verification
Against a live Neo4j capped at the reporter's exact 2.7 GiB, with a python graph planted
alongside a 1,841-module / 124,866-node superset-frontend graph:
--eagerpushPreviously either push would have deleted them. Suite: 232 pass / 6 skip / 0 fail; typecheck,
build,
gen:schema,gen:readmeall clean.I also removed a test that had codified the bug as intent — it asserted the wipe was
"intentionally UNANCHORED" and looped over the statements asserting
not.toContain("MATCH (n:CanNode"). It is replaced by three tests asserting the inverse safetyproperty, including an explicit
not.toContain("NOT n:CanNode")on the lethal shape.Follow-ups, not in this PR
content_hashis never written to the graph (project.tsomits it,bolt.tsreads it), so theincremental diff always concludes "everything changed" and every push is a full re-upsert.
Neo4j: never delete by default; gate purge on --eager and scope it to this analyzer codeanalyzer-python#171, Neo4j: never delete by default; gate purge on --eager and scope it to this analyzer codeanalyzer-java#213.