Skip to content

Neo4j projection drops module source and span byte offsets, so no span narrower than a callable resolves to text #202

Closed

Description

@rahlk

Problem

This analyzer's two projections disagree about the primary text, and the graph is the one that is wrong. codeanalyzer-java had the identical defect and fixed it in codellm-devkit/codeanalyzer-java#255; this is the sibling half.

The canonical model makes the module's whole-file source the primary text and every narrower node's text a byte-slice of it. The JSON projection carries source. The Neo4j projection drops it: PYMODULE_PROPS are id, file_key, module_name, content_hash, last_modified, file_size (codeanalyzer/neo4j/schema.py:86-96) — no source — while PyCallable keeps code (schema.py:106), which is a derivation of the source the graph no longer has.

The flattened span is the other half of the same defect. _SPAN = {"start_line": "integer", "end_line": "integer"} (schema.py:64) drops the columns and the UTF-8 byte offsets. Carrying source without the byte offsets does not make anything sliceable, so the two must move together.

Net effect: nothing narrower than a callable resolves to text on the graph backend. python-sdk documents living with it — see the "Projection-lossy fields" note at cldk/analysis/python/neo4j/neo4j_backend.py:78. A required pydantic source: str cannot express "the projection did not carry this", so it reports "" and the lossiness is invisible at the type level.

The invariant being restored: the JSON payload and the graph carry the same facts. Re-expressing a JSON field as an edge or a flattened property is fine; silently dropping one is not.

Scope boundary

Restores the text-resolution facts on this analyzer's Neo4j projection only: whole-file source on the module node, and columns plus byte offsets on the flattened span.

Does NOT drop PyCallable.code, even though this change makes it derivable — python-sdk reconstructs from it, so removing it is a separate breaking change. Does NOT touch the JSON projection, which is already correct. Does NOT move schema_version, which stays at the held 2.0.0 baseline (codellm-devkit/.github#50). Does NOT change python-sdk.

Goals

  • the module node carries source, the whole file, matching what the JSON projection carries
  • source is always present, never absent — an empty file yields "", so a consumer can never confuse "not carried" with "empty"
  • _SPAN gains start_column, end_column, start_byte, end_byte beside the existing line pair
  • the four new property names are exactly these spellings — codeanalyzer-java coined them in fix(neo4j): carry module.source and span byte offsets so the graph can resolve text codeanalyzer-java#255 and a term coined twice is permanently wrong under the cross-language parity clause
  • the checked-in graph schema snapshot is regenerated
  • a conformance test asserts every module node's source equals the JSON module source, per module rather than by presence, so a truncated or placeholder value fails

Caveats

  • Property names are not this repo's to choose. codeanalyzer-java shipped start_column / end_column / start_byte / end_byte first. Adopt them verbatim.
  • Node payloads grow by the size of the source tree. For scale: codeanalyzer-java measured 653 KiB across 141 modules on daytrader8, against 295 KiB the graph was already storing redundantly as callable code slices. The marginal cost is smaller than the headline and goes negative if code is later dropped.
  • schema_version does not move, so the addition is not detectable from the payload. Detection is by presence until the coordinated re-baseline.
  • Until this lands the grammar is split: a database holding both languages has sliceable Java nodes and unsliceable ones from this analyzer, so a cross-language "give me the text at this span" query cannot be written.

Definition of done

  • Every module node carries source, and each one's hash matches the content_hash the same node declares — this is what proves the value survived serialization intact rather than merely being present.
  • source[start_byte:end_byte] equals the code property already on the node, byte-for-byte, for every callable where both are present, with zero mismatches. This is the check that the byte offsets are real rather than merely declared.
  • Emitted schema matches the checked-in snapshot.
  • Existing suite green.

Activity

  1. rahlk commented on Sep 10, 2026

    @rahlk
    ContributorAuthor

    Design landed. Spec: docs/design/specs/2026-09-10-neo4j-json-parity.md (branch fix/issue-202-203-graph-json-parity). One PR closes this issue and its sibling — same three files, one snapshot regeneration, one pass over the conformance test.

    Correction to this issue's text. It cites codellm-devkit/.github#50 for "schema_version stays at the held 2.0.0 baseline". #50 is "scope every destructive Neo4j statement on the can:// id prefix; retire _module", and it mandates the opposite — "MAJOR graph-contract bump: a property is removed" — for that work, whose python leg is #173 (closed). The hold is real, but its source is the standing 2026-09-07 ruling: payload schema_version and graph SCHEMA_VERSION both stay 2.0.0 until the 2.0.0 line leaves release-candidate, across all three analyzers. Consumers gate on the analyzer-version floor, which moves with 1.5.2.

    Because the version holds while properties are added, neo4j/schema.py:27's promise of "MINOR on an additive change" becomes false. The same PR amends that note to record the hold — a file should not document a rule it breaks.

    One shape note. start_column/end_column/start_byte/end_byte are adopted verbatim from codellm-devkit/codeanalyzer-java#255, which coined them. For the labels whose JSON model carries flat line/column only and no Span — :PyAttribute, :PyVariable, :ConfigKey — the byte pair is computed at projection time with the existing byte_offsets() (schema/py_schema.py:104), rather than declaring four properties that get populated on only half the labels. All six _SPAN properties are then present everywhere _SPAN is spread, so the "position slices back to the text" check runs on every one of them.

  2. added a commit that references this issue on Sep 10, 2026
    f58d4b5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions