Skip to content

analysis.json stores declaration text per callable; python and typescript store it once per module #215

Description

@rahlk

Problem

The three analyzers disagree on where a declaration's text lives in analysis.json.

JSON Neo4j code
python module source once + span.bytes per node; no per-callable code derived at projection (neo4j/project.py::_span_code)
typescript module source once + span.bytes per node; no per-callable code derived at projection (matching python)
java Callable.code stored per callable (entities/Callable.java:64) carried straight through

Python and typescript store a module's text once and address every node into it by UTF-8 byte span.
Java stores each declaration's text again on the declaration. For a file with N declarations, java's
JSON carries the overlapping text N+1 times.

Python's projection documents the model (neo4j/project.py:697-704):

"A declaration's text: the owning module's source sliced by the node's utf-8 byte span. Schema
v2 stores source once per module, so the graph's code property … is derived here at projection
time (#104)."

typescript has just adopted the same approach for its graph, so java is now the only analyzer whose
JSON duplicates declaration text.

Scope boundary

Where declaration text lives in analysis.json. Not in scope: the Neo4j code property on
:JCallable, which should stay — all three analyzers agree the graph carries code directly, and
this issue would only change where java derives it from.

Goals

  • Decide whether java adopts store-once-plus-span, or python/typescript adopt store-per-node
  • If java adopts it: JModule (or the java equivalent) carries source; nodes carry
    span.bytes; Callable.code is dropped from the JSON model
  • The Neo4j projection derives code by byte-slicing, as python and typescript do
  • Measure the payload difference on a large repository before and after

Caveats and known risks

  • Breaking for analysis.json consumers. Anything reading Callable.code would need to slice
    instead. The java SDK and any downstream tooling are affected.
  • Byte spans, not character spans. Python and typescript slice UTF-8 bytes
    (source.encode()[lo:hi]). Java strings are UTF-16, so a naive substring would cut multi-byte
    characters. The span contract has to be explicitly byte-based.
  • Java may already carry line/column spans but not byte offsets — if so, the analyzer has to start
    emitting them, which is the bulk of the work.
  • This may not be worth doing. Storing text per node is redundant but simple and directly
    readable. The counter-argument is only that three analyzers emitting the same schema should not
    disagree about where text lives; if the duplication is acceptable, the better fix might be for
    python and typescript to document the divergence rather than for java to change.

Definition of done

Either all three analyzers store declaration text the same way in analysis.json, or the divergence
is recorded as a deliberate, documented decision in the canonical schema rather than an accident.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions