Skip to content

fix(neo4j): write content_hash, so the incremental push actually diffs - #119

Closed
rahlk wants to merge 1 commit into
fix/issue-116-neo4j-scopingfrom
fix/issue-118-content-hash
Closed

fix(neo4j): write content_hash, so the incremental push actually diffs#119
rahlk wants to merge 1 commit into
fix/issue-116-neo4j-scopingfrom
fix/issue-118-content-hash

Conversation

@rahlk

@rahlk rahlk commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Closes #118. Stacked on #117 — review that first; this PR's diff is the two-file change on top.

The bug

bolt.ts reads each module's stored content_hash to decide which modules changed:

const res = await s.run("MATCH (m:TSModule) RETURN m._module AS k, m.content_hash AS h");

Nothing wrote it. finalizeAnalysis stripped the field from the wire envelope, and the Neo4j
projection — built from that same envelope — omitted it, with project.ts even naming the strip as
the reason. So the diff compared against NULL for every module and concluded everything had
changed. Every push was a full re-upsert.

Three parts of the system disagreed:

schema.ts:97 declares content_hash: "string" on :TSModule
bolt.ts:137 reads it for the diff
project.ts never wrote it

The Neo4j contract needed no change — the property was already declared, only the writer was
missing, so schema.neo4j.json is untouched.

Parity

codeanalyzer-python does this correctly and is the reference: content_hash is wire payload
(schema/py_schema.py:450), projected onto the node (neo4j/project.py:689), and read back
(neo4j/bolt.py:105). Its incremental path works; ours never has.

Verification

Live Neo4j, superset-frontend (1,841 modules, 124,866 nodes):

result
push 1 1841 modules (1841 changed) — 55s
push 2, unchanged source 1841 modules (0 changed) — 25s
content_hash stored 1841 / 1841 (was 0)

Also checked on sample-app that editing exactly one file yields (1 changed), not 0 and not all.

Full suite with RUN_CONTAINER_TESTS=1: 237 pass, 0 fail, 0 skip.

About the test

The regression test asserts every projected :TSModule carries a non-null content_hash, and it
is break-checked — reverting the projection line fails it.

That shape is deliberate. The existing bolt test hand-seeded content_hash:'stale' into a fixture
node, so it exercised the diff against data the real projection could never produce. A test that
supplies the value it is meant to be verifying cannot catch this class of bug, which is why it sat
here undetected.

Risk worth naming

This enables a code path that has never run in production. A permanent full re-upsert was
masking whatever latent issues exist in the incremental route — edge ownership, the per-module
purge. The superset double-push above is the evidence it holds at real scale, but it is a
first-time execution and worth watching on the next real workload.

bolt.ts reads each module's stored `content_hash` to find what changed, but
nothing wrote it: finalizeAnalysis stripped the field from the wire envelope,
and the Neo4j projection -- built from that envelope -- omitted it, with a
comment in project.ts noting the strip as the reason. So the diff compared
against NULL for every module and concluded everything had changed. Every push
was a full re-upsert.

Three parts of the system disagreed: schema.ts DECLARED `content_hash` on
:TSModule, bolt.ts READ it, project.ts never WROTE it. The contract needed no
change -- the property was already declared, only the writer was missing.

codeanalyzer-python is the reference and does this correctly: content_hash is
wire payload (schema/py_schema.py:450), projected (neo4j/project.py:689), and
read back (neo4j/bolt.py:105).

Verified on a live Neo4j against superset-frontend:

  push 1              1841 modules (1841 changed)   55s
  push 2, unchanged   1841 modules (   0 changed)   25s
  content_hash stored 1841 / 1841 (was 0)

The regression test asserts every projected :TSModule carries a non-null hash,
and is break-checked. That shape matters here: the existing bolt test seeded
`content_hash:'stale'` into a fixture node by hand, so it exercised the diff
against data the projection could not produce, which is how this survived.

Note this enables a path that has never run in anger. A permanent full
re-upsert was masking any latent bug in the incremental route; the superset
double-push above is the evidence that it holds at scale.

Closes #118.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant