From d4cb2c876596b0981e6d53bf8c5c27c3d36237d3 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 10 Sep 2026 13:15:27 -0400 Subject: [PATCH 1/2] docs(design): spec the JSON-vs-graph parity sweep (#202, #203) The Neo4j projection drops facts analysis.json carries: module source, span columns and byte offsets, decorator positions, call-site callee_signature, literal-evaluated variable values, and import positions. Nothing narrower than a callable resolves to text on the graph backend as a result. Ten locked decisions, the non-obvious ones being: - the four span spellings are adopted verbatim from codeanalyzer-java#255, which coined them, rather than invented here - byte offsets are computed at projection time for the labels whose JSON model carries flat line/column only, so _SPAN keeps meaning one thing - callee_signature arrives by a positional join from call_sites, keeping the JSON projection out of scope; traversal to the callee loses the 20-28% of call sites that never resolve - argument_types is deliberately not carried: it is the legacy field #86 split into PyCallArgument, already on the graph as arguments_json - the decorator span rides PY_DECORATED_BY, since :PyDecorator is never pruned - PY_IMPORTS positions key on the spelling, because the edge pre-aggregates and emits spellings sorted, so index alignment is already lost Comment parity is dropped rather than deferred, per codeanalyzer-java#257 closing NOT_PLANNED: comments are globally ignored across the analyzers and docstring is where the comment model stops. Neither version moves, per the standing 2026-09-07 hold. Both issues cite .github#50 for that hold, which is a miscitation -- #50 mandates a MAJOR for the _module removal instead. The hold is real; only its source was wrong. --- .claude/SCHEMA_DECISIONS.md | 46 +++++ .../specs/2026-09-10-neo4j-json-parity.md | 179 ++++++++++++++++++ 2 files changed, 225 insertions(+) create mode 100644 docs/design/specs/2026-09-10-neo4j-json-parity.md diff --git a/.claude/SCHEMA_DECISIONS.md b/.claude/SCHEMA_DECISIONS.md index a36cbd0..2f47c83 100644 --- a/.claude/SCHEMA_DECISIONS.md +++ b/.claude/SCHEMA_DECISIONS.md @@ -557,3 +557,49 @@ codeanalyzer-typescript. Terms coined once and shared: `SCHEME`, `LANG`, - Additive field, `schema_version` unchanged. A 1.5.0 `analysis.json` still parses (`var` defaults to `None`); consumers should tolerate its absence for one generation of cached graphs. + +## 2026-09-10 — The Neo4j projection carries the JSON's facts (issues #202, #203) + +Spec: `docs/design/specs/2026-09-10-neo4j-json-parity.md`. +Sibling halves: codeanalyzer-java#255/#256, codeanalyzer-typescript#201/#202. + +- **Graph-only change.** `:PyModule` gains `source`; `_SPAN` gains + `start_column`/`end_column`/`start_byte`/`end_byte` on all six labels that spread + it. The four spellings are **adopted verbatim from codeanalyzer-java#255**, which + coined them — java has not landed them yet, and adoption follows the decision, not + the merge order. +- **Bytes are computed where the model has none.** Attribute, variable and + `ConfigKey` carry flat line/column, so the projector calls the existing + `byte_offsets()` rather than declaring four properties it populates on only half + the labels. `_SPAN` keeps meaning one thing. +- **`callee_signature` by projection-time join**, `PyCallable.call_sites` → body node + on `(start_line, start_column)`. `BodyNode` has no such field in JSON and the JSON + projection is out of scope; the join is asserted per call site, since a positional + join can miss silently. Traversal to the callee's signature was rejected: 20-28% of + call sites never resolve, which is where the field is the only source. +- **`argument_types` deliberately not carried.** It is the legacy field #86 split + into `PyCallArgument{ast_kind, inferred_type}`, already on the graph as + `arguments_json`. Carrying it would store one fact twice in two vocabularies, one + of them deprecated. #203's goal is satisfied by the successor. +- **`value_json`, always encoded.** `PyVariableDeclaration.value` is `Optional[Any]`; + Neo4j takes scalars and scalar arrays. One shape for every value, per the + `arguments_json` precedent. `initializer` stays raw source text. +- **Decorator span on the relationship, not the node.** `:PyDecorator` is merged on + `qualified_name`, carries no `_module` and is never pruned, so per-application + facts on it accumulate across every project in the database. Same reason + `expression` and the arguments already ride `PY_DECORATED_BY`. +- **`PY_IMPORTS.positions_json` keyed by spelling.** The edge pre-aggregates per + `(module, target)` and emits `spellings` `sorted()`, so parallel position arrays + have already lost index alignment. The spelling is the only stable key. +- **Neither version moves**, per the standing 2026-09-07 hold (payload + `schema_version` and graph `SCHEMA_VERSION` both `2.0.0` until the 2.0.0 line + leaves release-candidate). Consumers gate on the analyzer-version floor, which + moves with 1.5.2. `neo4j/schema.py`'s "MINOR on an additive change" note is amended + to record the hold — the file must not document a rule it breaks. Both issues cite + `.github#50` for the hold; that is a miscitation, #50 mandates a MAJOR for the + `_module` removal (#173, closed, bump waived by the same ruling). +- **Comments are dropped, not deferred.** codeanalyzer-java#257 closed `NOT_PLANNED` + on 2026-09-10: comments are globally ignored across the analyzers and `docstring` + is where the comment model stops. Its closing comment names #203 as dropping its + comment goals for the same reason. `python-sdk`'s `get_all_comments` raising on the + Neo4j backend is now the permanent answer, not a workaround. diff --git a/docs/design/specs/2026-09-10-neo4j-json-parity.md b/docs/design/specs/2026-09-10-neo4j-json-parity.md new file mode 100644 index 0000000..baf3783 --- /dev/null +++ b/docs/design/specs/2026-09-10-neo4j-json-parity.md @@ -0,0 +1,179 @@ +# JSON == Graph: the Neo4j projection carries the same facts as the payload + +**Date:** 2026-09-10 +**Status:** Approved (design dialogue in-session) +**Scope:** codeanalyzer-python, Neo4j projection only, schema v2 additive; closes #202 and #203 +**Builds on:** `can://` prefix scoping and `_module` retirement (#173, `.github#50`); +call-site/body convergence (`call-site-body-convergence.md`, #120) +**Sibling halves:** codeanalyzer-java#255/#256, codeanalyzer-typescript#201/#202 — same +defect class, each on its own clock + +## Problem + +The two projections disagree about what they carry, and the graph is the one that +is wrong. `analysis.json` carries the module's whole-file `source`, byte offsets on +every span, decorator positions, call-site `callee_signature`, literal-evaluated +variable `value`, and import positions. The Neo4j projection drops each of them, +while keeping `code` on `:PyClass`/`:PyCallable` — a *derivation* of the source the +graph does not have. + +Net effect: nothing narrower than a callable resolves to text on the graph backend. +`python-sdk` documents living with it (`cldk/analysis/python/neo4j/neo4j_backend.py`, +"Projection-lossy fields"). + +The invariant being restored: **re-expressing a JSON field as an edge or a flattened +property is fine — silently dropping one is not.** `arguments_json` and +`keyword_arguments_json` are the good precedent; a joined `docstring` string is the +bad one. + +## Locked decisions + +1. **`:PyModule` gains `source`**, the whole file, always present — an empty file + yields `""`, so "not carried" is never confusable with "empty". The projector + already holds the module source at this point (`_span_code`, `project.py:763`). + +2. **`_SPAN` gains `start_column`, `end_column`, `start_byte`, `end_byte`** beside + the existing line pair, spread on all six labels that use it (`:PyClass`, + `:PyCallable`, `:PyAttribute`, `:PyVariable`, `:PyBodyNode`, `:ConfigKey`). + **These four spellings are adopted verbatim** from codeanalyzer-java#255, which + coined them; a term coined twice is permanently wrong under the parity clause. + Java has not landed them yet (`start_byte` appears nowhere in that repo at time + of writing) — adoption follows the decision, not the merge order. + +3. **Byte offsets are computed at projection time where the JSON model has no + `Span`.** `PyClassAttribute`, `PyVariableDeclaration` and `PyConfigKey` carry + flat line/column only. `byte_offsets()` (`schema/py_schema.py:104`) already + converts ast positions to utf-8 offsets and the module source is in hand, so all + four properties are populated uniformly rather than present on some labels and + pruned on others. `_SPAN` keeps meaning one thing. + +4. **`callee_signature` reaches `:PyBodyNode` by a projection-time join** from + `PyCallable.call_sites`, keyed on `(start_line, start_column)`. `BodyNode` does + not carry the field in JSON — `PyCallsite` does — and neither issue's scope + admits a JSON change. The join is asserted per call site by the conformance + test, not by presence, because a positional join can silently miss. + Traversal (`PY_RESOLVES_TO` → `:PyCallable.signature`) is not sufficient: 20-28% + of call sites never resolve a callee (`py_schema.py:130`), which is exactly where + the field carries information nothing else has. + +5. **`argument_types` is not carried, by decision.** `py_schema.py:311` records it + as the legacy field that "mixed these two vocabularies in one list", superseded + by `PyCallArgument{ast_kind, inferred_type}` (#86) — which the graph already + carries as `arguments_json`. Carrying it would re-import a vocabulary this repo + deliberately split and store the same fact twice in two shapes. The goal in #203 + is satisfied by the successor field; recorded here so it is not rediscovered as + a gap. + +6. **`:PyVariable` gains `value_json`**, always JSON-encoded. `value` is + `Optional[Any]` and Neo4j properties are scalars or scalar arrays, so a dict or + list needs a serialization rather than a raw put. One shape for every value + (`json.loads` to recover), following the `arguments_json` precedent. `initializer` + stays the raw source text; `value_json` is the literal-evaluated result. + +7. **The decorator's span rides on `PY_DECORATED_BY`, not on `:PyDecorator`.** + Forced, not chosen: `:PyDecorator` is merged on the resolved `qualified_name` and + carries no `_module`, so it is never pruned — anything application-specific on it + accumulates across every project in the database (`project.py:712-724`). The span + joins `expression` and the argument properties, which already ride the + relationship for that reason. `PyDecorator.span` is a real `Span`, so all six + properties are available. + +8. **`PY_IMPORTS` gains `positions_json`, keyed by spelling.** The edge is + pre-aggregated — one edge per `(module, target)`, many statements collapsed onto + `spellings` — because the writer MERGEs on `(type, from, to)` and a second row + would overwrite rather than add. Parallel position arrays are therefore unsafe: + `spellings` is emitted `sorted()`, so index alignment is already lost and any + future edit to the sort would break it silently. Keying on the spelling survives + both aggregation and sorting. + +9. **Neither version moves.** Payload `schema_version` and graph `SCHEMA_VERSION` + both stay `2.0.0`, per the standing 2026-09-07 ruling (all three analyzers) that + holds them until the 2.0.0 line leaves release-candidate. Consumers gate on the + **analyzer version** — the python-sdk Neo4j backends carry an analyzer floor and + refuse anything below it at attach — and that floor moves with this release. + + `neo4j/schema.py:27` promises "MINOR on an additive change (new label/rel/property)". + This change is additive and does not bump, so that note is amended in the same PR + to record the hold and its scope. A file must not document a rule it breaks. + + Both issues cite `.github#50` for this hold. That is a miscitation — #50 mandates + the *opposite* ("MAJOR graph-contract bump: a property is removed") for the + `_module` retirement, whose python leg is #173, already closed, and whose bump the + same 2026-09-07 ruling waived. The hold is real; only its source was wrong. + +10. **One snapshot, two copies, both regenerated.** `schema.neo4j.json` is + regenerated by `canpy --emit schema` and asserted current by + `test/test_neo4j_schema.py`. `docs/handoff/schema.neo4j.json` is a second copy + that no test guards and that **already differs from the root snapshot before + this change**. Both are regenerated and a test asserts they match, so the + handoff bundle stays self-contained without drifting again. + +## Excluded: comment parity + +#203's comment goals are **dropped, not deferred.** codeanalyzer-java#257 closed +`NOT_PLANNED` on 2026-09-10: comments are globally ignored across the analyzers, the +ceiling is accepted rather than deferred, and `docstring` on the graph is where the +comment model stops. That issue's closing comment names this repo's #203 explicitly +as dropping its comment goals for the same reason. + +Consequences, recorded so they are not rediscovered: + +- The `JSON == Graph` invariant has exactly one carved-out exception, made + explicitly rather than by omission. +- `python-sdk`'s `get_all_comments` / `get_comment_in_file` raising on the Neo4j + backend is the permanent answer for that backend, not a workaround awaiting this + work. The in-memory backend remains the way to reach comments. +- Module-level comments, non-docstring comments and comment positions stay absent + from the graph. Reopen only if a consumer turns up that needs them. + +## Scope boundary + +Restores facts on this analyzer's Neo4j projection only. + +Does **not** touch the JSON projection, which is already correct — decision 4 is a +projection-time join precisely to keep that true. Does **not** drop `PyCallable.code` +or `PyClass.code`, which become derivable once `source` lands but which `python-sdk` +reads; removing them is a separate breaking change. Does **not** change +`python-sdk`. Does **not** move either version (decision 9). Does **not** carry +comments (above). + +## Tracking and release plan + +- **One PR closes both #202 and #203.** Same three files (`neo4j/schema.py`, + `neo4j/project.py`, `neo4j/rows.py`), one snapshot regeneration, one pass over the + conformance test. Split across two PRs, the snapshot is regenerated twice and the + same test file edited twice for one sweep. Granularity follows PR granularity, so + the tracking record is the two existing issues — no epic. +- **#203's comment goals are struck on the issue** with the java#257 citation, and + the `.github#50` miscitation is corrected on both. +- **Ships as 1.5.2, alone.** No lockstep, matching #50's own precedent that each + analyzer changes its own writers and cuts its own release. +- **`python-sdk` follow-up is filed just-in-time** after 1.5.2 is on PyPI: its + "projection-lossy fields" note becomes wrong for everything except comments, and + its analyzer floor moves. Detection is by analyzer version, not schema version + (decision 9). +- **Siblings adopt decisions 6 and 8 as new shared vocabulary** (`value_json`, + `positions_json`); the four span names of decision 2 are already java's. Recorded + as comments on codeanalyzer-java#256 and codeanalyzer-typescript#202 rather than + coordinated in an epic, since no analyzer reads another's nodes. + +## Definition of done + +- Every module node carries `source`, and each one's hash matches the + `content_hash` that same node declares — proof the value survived serialization + rather than merely being present. +- `source[start_byte:end_byte]` equals the node's `code` property byte-for-byte for + every callable where both exist, zero mismatches — proof the offsets are real and + not merely declared. +- For every `_SPAN`-bearing label, all six properties are present on every node, + and the byte pair slices the module source back to that node's own text. +- Every resolved call site's `callee_signature` on `:PyBodyNode` equals the value on + the matching `PyCallsite`, asserted per call site. +- `:PyVariable.value_json` round-trips through `json.loads` to the JSON `value` for + every variable that has one. +- `PY_DECORATED_BY` carries the span of the decorator application it represents; + `PY_IMPORTS.positions_json` carries a position for every spelling on the edge. +- Existing `docstring` values are unchanged on every node that had one. +- Emitted schema matches both checked-in snapshots, and the two snapshots match + each other. +- Existing suite green. From f58d4b52a5084f7b4bedcaf6c0a3302677d45526 Mon Sep 17 00:00:00 2001 From: Rahul Krishna Date: Thu, 10 Sep 2026 15:54:07 -0400 Subject: [PATCH 2/2] fix(neo4j): the graph projection carries the facts analysis.json carries MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both projections are first-class: anything one carries, the other re-expresses. Re-shaping a JSON field into an edge or a flattened property is fine; dropping it is not. Seven facts were only in the payload, so a consumer that attaches to the graph instead of reading `analysis.json` silently saw less. Carried now: - `:PyModule.source` — the whole-file text, always present ("" for an empty file, so "not carried" is never confusable with "empty"). Every narrower node's text is a byte slice of it, so without it the spans below have nothing to slice. - `_SPAN` gains `start_column`/`end_column`/`start_byte`/`end_byte`, spelled verbatim as codeanalyzer-java#255 coined them. Labels whose JSON model has a real `Span` read them off it; `:PyVariable`'s are computed with the existing `byte_offsets()`, so the dict means one thing on every label that spreads it. - `:PyVariable.value_json` — the literal-evaluated value, always JSON-encoded: it is `Optional[Any]` and a Neo4j property is a scalar or an array of scalars. `initializer` stays the raw source text. - `:PyBodyNode.callee_signature` — joined from `PyCallable.call_sites` on the call site's `(start_line, start_column)`, since `BodyNode` has no such field in JSON. `argument_types` is deliberately not carried: it is the legacy field #86 split into `PyCallArgument`, already on the graph as `arguments_json`. - `PY_IMPORTS.positions_json` — a position per import spelling, keyed by spelling rather than by index because the edge pre-aggregates per (module, target) and emits `spellings` sorted, so parallel arrays have already lost their alignment. - `PY_DECORATED_BY` gains the span, on the relationship rather than on `:PyDecorator`: that node is merged on `qualified_name`, carries no `_module` and is never pruned, so a per-application fact on it would accumulate across every project in the database. Same reason `expression` and the arguments already ride the relationship. `:PyAttribute` is the one `_SPAN` exception. `PyClassAttribute` carries `start_line`/`end_line` and no columns at all, so no byte offsets are derivable, and a fabricated column 0 would claim a sliceable span that is not one. The label declares the line pair honestly, the conformance test asserts exactly that so the exception cannot widen unnoticed, and giving the JSON model columns is follow-on work (a JSON-contract change). Comment parity is out of scope — won't fix, per codeanalyzer-java#257's NOT_PLANNED closure, which names this issue. Both `schema_version` and the graph's `SCHEMA_VERSION` hold at `2.0.0` per the 2026-09-07 ruling for all three analyzers; these additive properties are detectable by presence, and consumers gate on the analyzer version floor instead. `neo4j/schema.py`'s "MINOR on additive" note now records the suspension, so the file no longer documents a rule it breaks. Both schema snapshots are regenerated, and a new test asserts the handoff copy matches the root one — nothing guarded it and it had already drifted two node labels behind. Spec: docs/design/specs/2026-09-10-neo4j-json-parity.md Closes #202 Closes #203 --- .claude/SCHEMA_DECISIONS.md | 14 +- codeanalyzer/neo4j/project.py | 93 +++++- codeanalyzer/neo4j/schema.py | 55 +++- .../specs/2026-09-10-neo4j-json-parity.md | 20 +- docs/handoff/schema.neo4j.json | 290 ++++++++++++++---- schema.neo4j.json | 40 ++- test/test_neo4j_config_keys.py | 4 + test/test_neo4j_schema.py | 12 + test/test_v2_two_projection_agreement.py | 165 ++++++++++ 9 files changed, 593 insertions(+), 100 deletions(-) diff --git a/.claude/SCHEMA_DECISIONS.md b/.claude/SCHEMA_DECISIONS.md index 2f47c83..2dd5dbf 100644 --- a/.claude/SCHEMA_DECISIONS.md +++ b/.claude/SCHEMA_DECISIONS.md @@ -568,10 +568,16 @@ Sibling halves: codeanalyzer-java#255/#256, codeanalyzer-typescript#201/#202. it. The four spellings are **adopted verbatim from codeanalyzer-java#255**, which coined them — java has not landed them yet, and adoption follows the decision, not the merge order. -- **Bytes are computed where the model has none.** Attribute, variable and - `ConfigKey` carry flat line/column, so the projector calls the existing - `byte_offsets()` rather than declaring four properties it populates on only half - the labels. `_SPAN` keeps meaning one thing. +- **Bytes are computed where the model has none.** `PyVariableDeclaration` carries + flat line/column, so the projector calls the existing `byte_offsets()` rather than + declaring four properties it populates on only half the labels. `PyConfigKey` + already carries a real `Span`. +- **`:PyAttribute` is the one `_SPAN` exception** (found in implementation, spec + amended). `PyClassAttribute` carries `start_line`/`end_line` and no columns, so no + byte offsets are derivable; a fabricated column 0 would claim a sliceable span that + is not one. The label declares the line pair explicitly and the conformance test + asserts exactly that, so the exception cannot widen unnoticed. Giving the JSON model + columns is follow-on work. - **`callee_signature` by projection-time join**, `PyCallable.call_sites` → body node on `(start_line, start_column)`. `BodyNode` has no such field in JSON and the JSON projection is out of scope; the join is asserted per call site, since a positional diff --git a/codeanalyzer/neo4j/project.py b/codeanalyzer/neo4j/project.py index cf3f88e..efc0c38 100644 --- a/codeanalyzer/neo4j/project.py +++ b/codeanalyzer/neo4j/project.py @@ -50,7 +50,7 @@ ) from codeanalyzer.schema import model_dump from codeanalyzer.schema.ids import application_id, external_id, global_ordinal, purl_pypi -from codeanalyzer.schema.py_schema import PyDecorator +from codeanalyzer.schema.py_schema import PyDecorator, byte_offsets def project(app: PyApplication, app_name: str, sig_to_id: dict, @@ -182,6 +182,15 @@ def _project_program_graphs( if not c.id: continue # unstamped callable — assign_ids must run first owner = _sym(c.id) # the :PyCallable node, keyed by its can:// id + # ``callee_signature`` lives on ``PyCallable.call_sites``, not on the body + # node, so the graph joins the two on the call site's position (#203). + # ``argument_types`` is deliberately not joined: it is the legacy field #86 + # split into ``PyCallArgument``, already carried as ``arguments_json``. + sig_by_pos = { + (cs.start_line, cs.start_column): cs.callee_signature + for cs in (c.call_sites or []) + if cs.callee_signature + } for local_key, node in (c.body or {}).items(): span = node.span # L4 param vertices carry the variable they model (``of``) and @@ -194,8 +203,11 @@ def _project_program_graphs( prune( { "kind": node.kind, - "start_line": span.start[0] if span else None, - "end_line": span.end[0] if span else None, + **_span_props(span), + "callee_signature": ( + sig_by_pos.get((span.start[0], span.start[1])) + if span else None + ), "var": node.of, "call_node": node.parent, # Call-site detail (#120). The JSON emits one node per @@ -346,8 +358,7 @@ def _project_artifacts(b: RowBuilder, app: PyApplication, app_name: str, app_ref "namespace": ck.namespace, "value": ck.value, "references": list(ck.references or []), - "start_line": ck.span.start[0] if ck.span else None, - "end_line": ck.span.end[0] if ck.span else None, + **_span_props(ck.span), } ), ) @@ -569,7 +580,7 @@ def _project_module_body( _project_class(b, file_key, mod_ref, "PY_DECLARES", cl, externals, sig_to_id, mod.source, base_ref) for v in mod.variables or []: - _project_variable(b, file_key, mod_ref, v) + _project_variable(b, file_key, mod_ref, v, mod.source or "") _project_imports(b, mod_ref, mod, module_id_by_key) @@ -597,9 +608,15 @@ def _project_imports(b: RowBuilder, mod_ref: NodeRef, mod: PyModule, continue key = im.resolved_module or im.module a = agg.setdefault( - key, {"spellings": set(), "names": set(), "aliases": set(), "resolved": im.resolved_module} + key, + {"spellings": set(), "names": set(), "aliases": set(), "positions": {}, + "resolved": im.resolved_module}, ) a["spellings"].add(im.module) + # Keyed by spelling, never by index: ``spellings`` is emitted sorted, so a + # parallel position array has already lost its alignment (#203). + a["positions"][im.module] = [im.start_line, im.start_column, + im.end_line, im.end_column] if im.name: a["names"].add(im.name) if im.alias: @@ -623,6 +640,7 @@ def _project_imports(b: RowBuilder, mod_ref: NodeRef, mod: PyModule, "spellings": sorted(a["spellings"]), "imported_names": sorted(a["names"]) or None, "aliases": sorted(a["aliases"]) or None, + "positions_json": json.dumps(a["positions"], sort_keys=True), } ), ) @@ -675,7 +693,7 @@ def _project_callable( _project_decorator(b, ref, d) for v in c.local_variables or []: - _project_variable(b, file_key, ref, v) + _project_variable(b, file_key, ref, v, source) for ic in (c.callables or {}).values(): _project_callable(b, file_key, ref, "PY_DECLARES", ic, externals, sig_to_id, source, base_ref) @@ -700,12 +718,14 @@ def _project_variable( file_key: str, owner: NodeRef, v: PyVariableDeclaration, + source: str, ) -> None: # ``/@`` (#173) — the owner is the module or the # callable, so a module-level variable sits under ``/`` like every # other declaration and the module's prefix purge reaches it. var_id = f"{owner.value}/{v.name}@{v.start_line}" - ref = b.node(["PyVariable"], "id", var_id, _variable_props(v, var_id, file_key)) + ref = b.node(["PyVariable"], "id", var_id, + _variable_props(v, var_id, file_key, source)) b.edge("PY_DECLARES_VAR", owner, ref) @@ -737,6 +757,7 @@ def _project_decorator(b: RowBuilder, on: NodeRef, decorator: PyDecorator) -> No "keyword_arguments_json": json.dumps( dict(decorator.keyword_arguments or {}), sort_keys=True ), + **_span_props(decorator.span), }, ) @@ -752,6 +773,9 @@ def _module_props(mod: PyModule, file_key: str) -> Props: "id": mod.id, "file_key": file_key, "module_name": mod.module_name, + # Always present, never pruned: "" for an empty file, so a consumer can + # never confuse "not carried" with "empty" (#202). + "source": mod.source or "", "content_hash": mod.content_hash, "last_modified": mod.last_modified, "file_size": mod.file_size, @@ -760,6 +784,36 @@ def _module_props(mod: PyModule, file_key: str) -> Props: ) +def _span_props(span) -> Props: + """The six flattened span properties from a v2 ``Span`` (#202). Lines and columns + come straight off the model; ``bytes`` is already utf-8 and is what makes the span + sliceable out of ``:PyModule.source``.""" + if span is None: + return {} + return { + "start_line": span.start[0], "start_column": span.start[1], + "end_line": span.end[0], "end_column": span.end[1], + "start_byte": span.bytes[0], "end_byte": span.bytes[1], + } + + +def _flat_span_props(source: str, start_line: int, start_column: int, + end_line: int, end_column: int) -> Props: + """The same six for a model carrying flat ast positions and no ``Span`` + (:PyAttribute, :PyVariable). The byte pair is computed here rather than left + absent, so ``_SPAN`` means one thing on every label that spreads it (#202).""" + if start_line < 0 or end_line < 0 or not source: + return {} + start_column = max(start_column, 0) + end_column = max(end_column, 0) + lo, hi = byte_offsets(source, start_line, start_column, end_line, end_column) + return { + "start_line": start_line, "start_column": start_column, + "end_line": end_line, "end_column": end_column, + "start_byte": lo, "end_byte": hi, + } + + def _span_code(source: str, span) -> str | None: """A declaration's text: the owning module's ``source`` sliced by the node's utf-8 byte span. Schema v2 stores source once per module, so the graph's @@ -781,8 +835,8 @@ def _class_props(cl: PyClass, file_key: str, source: str) -> Props: "base_classes": list(cl.base_classes or []), "decorators": [d.qualified_name or d.name for d in (cl.decorators or [])], "docstring": _docstring_of(cl.comments), - "start_line": cl.start_line, - "end_line": cl.end_line, + **(_span_props(cl.span) or {"start_line": cl.start_line, + "end_line": cl.end_line}), "_module": file_key, "is_entrypoint": bool(cl.entrypoints), "entrypoint_frameworks": sorted({e.framework for e in (cl.entrypoints or [])}), @@ -801,8 +855,8 @@ def _callable_props(c: PyCallable, file_key: str, source: str) -> Props: "cyclomatic_complexity": c.cyclomatic_complexity, "code": _span_code(source, c.span), "code_start_line": c.code_start_line, - "start_line": c.start_line, - "end_line": c.end_line, + **(_span_props(c.span) or {"start_line": c.start_line, + "end_line": c.end_line}), "docstring": _docstring_of(c.comments), "decorators": [d.qualified_name or d.name for d in (c.decorators or [])], "modifiers": list(c.modifiers or []), @@ -823,6 +877,8 @@ def _attribute_props(a: PyClassAttribute, attr_id: str, file_key: str) -> Props: "type": a.type, "initializer": a.initializer, "docstring": _docstring_of(a.comments), + # Line-only: PyClassAttribute has no columns, so there is nothing to + # derive bytes from. See the note on :PyAttribute in schema.py. "start_line": a.start_line, "end_line": a.end_line, "_module": file_key, @@ -830,16 +886,21 @@ def _attribute_props(a: PyClassAttribute, attr_id: str, file_key: str) -> Props: ) -def _variable_props(v: PyVariableDeclaration, var_id: str, file_key: str) -> Props: +def _variable_props(v: PyVariableDeclaration, var_id: str, file_key: str, + source: str) -> Props: return prune( { "id": var_id, "name": v.name, "type": v.type, "initializer": v.initializer, + # ``value`` is Optional[Any] and a Neo4j property is a scalar or an array + # of scalars, so a dict/list value needs a serialization rather than a raw + # put -- always encoded, one shape for every value (#203). + "value_json": _stringify_if(v.value) if v.value is not None else None, "scope": v.scope, - "start_line": v.start_line, - "end_line": v.end_line, + **_flat_span_props(source, v.start_line, v.start_column, + v.end_line, v.end_column), "_module": file_key, } ) diff --git a/codeanalyzer/neo4j/schema.py b/codeanalyzer/neo4j/schema.py index 0bd9d50..9b455da 100644 --- a/codeanalyzer/neo4j/schema.py +++ b/codeanalyzer/neo4j/schema.py @@ -28,6 +28,13 @@ relationship or key), MINOR on an additive change (new label/rel/property). It is stamped onto the :PyApplication node of every emitted graph so any consumer can detect a producer/consumer mismatch at runtime. + +**The additive-MINOR rule is suspended for the 2.0.0 line.** Per the 2026-09-07 ruling (all three +analyzers), payload ``schema_version`` and this version both hold at ``2.0.0`` until the 2.0.0 line +leaves release-candidate, so the additive properties of #202/#203 ship without a bump and are +detectable only by presence. Consumers gate on the **analyzer version** instead -- the python-sdk +Neo4j backends carry an analyzer floor and refuse anything below it at attach. The rule above +resumes at the coordinated re-baseline. """ from __future__ import annotations @@ -61,7 +68,18 @@ class RelType: # anchor for the prefix-scoped destructive statements (see ``rows.CAN_NODE``). MARKER_LABELS: List[str] = ["PyCanNode"] -_SPAN = {"start_line": "integer", "end_line": "integer"} +# The flattened span. ``start_column``/``end_column``/``start_byte``/``end_byte`` +# are adopted **verbatim** from codeanalyzer-java#255, which coined them: a term +# coined twice is permanently wrong under the cross-language parity clause. The +# byte pair makes every span sliceable out of :PyModule.source -- for the labels +# whose JSON model carries flat ast positions and no ``Span`` (:PyAttribute, +# :PyVariable) the projector computes it, so this dict means one thing on every +# label that spreads it (#202). +_SPAN = { + "start_line": "integer", "end_line": "integer", + "start_column": "integer", "end_column": "integer", + "start_byte": "integer", "end_byte": "integer", +} NODE_LABELS: List[NodeLabel] = [ @@ -90,6 +108,10 @@ class RelType: "id": "string", "file_key": "string", "module_name": "string", + # The primary text: schema v2 stores source once per module and every + # narrower node's text is a byte slice of it. Always present -- an empty + # file yields "", so "not carried" is never confusable with "empty" (#202). + "source": "string", "content_hash": "string", "last_modified": "float", "file_size": "integer", @@ -158,7 +180,13 @@ class RelType: "type": "string", "initializer": "string", "docstring": "string", - **_SPAN, + # The one _SPAN exception: ``PyClassAttribute`` carries start/end LINE only + # -- no columns, hence no derivable byte offsets. Emitting a fabricated + # column 0 would make the span unsliceable while claiming otherwise, so the + # line pair is declared honestly instead. Filed for the JSON model to gain + # columns; until then this label is line-granular (#203). + "start_line": "integer", + "end_line": "integer", }, ), NodeLabel( @@ -170,6 +198,11 @@ class RelType: "name": "string", "type": "string", "initializer": "string", + # ``PyVariableDeclaration.value`` -- the literal-evaluated result, always + # JSON-encoded because it is ``Optional[Any]`` and a Neo4j property is a + # scalar or an array of scalars (the ``arguments_json`` precedent, #203). + # ``initializer`` stays the raw source text. + "value_json": "string", "scope": "string", **_SPAN, }, @@ -197,6 +230,12 @@ class RelType: "return_type": "string", "is_constructor_call": "boolean", "arguments_json": "string", + # What distinguishes overload targets at a resolved call site. Joined + # from ``PyCallable.call_sites`` at projection time, since ``BodyNode`` + # does not carry it in JSON (#203). ``argument_types`` is deliberately + # NOT here: it is the legacy field #86 split into ``PyCallArgument``, + # already carried as ``arguments_json``. + "callee_signature": "string", **_SPAN, }, ), @@ -247,16 +286,26 @@ class RelType: "PY_IMPORTS", ["PyModule"], ["PyModule", "PyPackage"], - {"spellings": "string[]", "imported_names": "string[]", "aliases": "string[]"}, + # ``positions_json`` keys on the spelling, not on an index: this edge + # pre-aggregates per (module, target) and emits ``spellings`` sorted, so + # parallel position arrays have already lost their alignment (#203). + {"spellings": "string[]", "imported_names": "string[]", "aliases": "string[]", + "positions_json": "string"}, ), RelType( "PY_DECORATED_BY", ["PyCallable", "PyClass"], ["PyDecorator"], + # The span rides here, not on :PyDecorator: that node is merged on the + # resolved ``qualified_name`` and carries no ``_module``, so it is never + # pruned and any per-application fact on it would accumulate across every + # project in the database -- the same reason ``expression`` and the + # arguments already ride the relationship (#203). { "expression": "string", "positional_arguments": "string[]", "keyword_arguments_json": "string", + **_SPAN, }, ), # Level-3 CPG overlay (-a 3 only): the cross-language dataflow vocabulary, diff --git a/docs/design/specs/2026-09-10-neo4j-json-parity.md b/docs/design/specs/2026-09-10-neo4j-json-parity.md index baf3783..0283538 100644 --- a/docs/design/specs/2026-09-10-neo4j-json-parity.md +++ b/docs/design/specs/2026-09-10-neo4j-json-parity.md @@ -41,11 +41,21 @@ bad one. of writing) — adoption follows the decision, not the merge order. 3. **Byte offsets are computed at projection time where the JSON model has no - `Span`.** `PyClassAttribute`, `PyVariableDeclaration` and `PyConfigKey` carry - flat line/column only. `byte_offsets()` (`schema/py_schema.py:104`) already - converts ast positions to utf-8 offsets and the module source is in hand, so all - four properties are populated uniformly rather than present on some labels and - pruned on others. `_SPAN` keeps meaning one thing. + `Span`.** `PyVariableDeclaration` carries flat line/column and no `Span`; + `byte_offsets()` (`schema/py_schema.py:104`) already converts ast positions to + utf-8 offsets and the module source is in hand, so `:PyVariable` gets all six + properties like every other label. `PyConfigKey` needs nothing — it carries a real + `Span` already. + + **Amended during implementation: `:PyAttribute` is the one exception.** + `PyClassAttribute` (`py_schema.py:409`) carries `start_line`/`end_line` and *no + columns at all* — this spec originally lumped it with `PyVariableDeclaration`, + which was wrong. With no columns there is nothing to derive byte offsets from, and + emitting a fabricated column 0 would make the span unsliceable while claiming + otherwise. So `:PyAttribute` declares the line pair honestly instead of spreading + `_SPAN`, the exception is asserted in the conformance test so it cannot widen + silently, and giving `PyClassAttribute` columns in the JSON model is left as + follow-on work (a JSON-contract change, out of scope here). 4. **`callee_signature` reaches `:PyBodyNode` by a projection-time join** from `PyCallable.call_sites`, keyed on `(start_line, start_column)`. `BodyNode` does diff --git a/docs/handoff/schema.neo4j.json b/docs/handoff/schema.neo4j.json index 32f8422..7a91e07 100644 --- a/docs/handoff/schema.neo4j.json +++ b/docs/handoff/schema.neo4j.json @@ -1,20 +1,25 @@ { "schema_version": "2.0.0", "generator": "codeanalyzer-python", - "marker_labels": [], + "marker_labels": [ + "PyCanNode" + ], "node_labels": [ { "label": "PyApplication", "merge_label": "PyApplication", - "key": "name", + "key": "id", "properties": { + "id": "string", "name": "string", "schema_version": "string", "analyzer_name": "string", "analyzer_version": "string", "repo_uri": "string", "source_revision": "string", - "repo_dirty": "boolean" + "repo_dirty": "boolean", + "entrypoint_frameworks": "string[]", + "entrypoint_report_json": "string" } }, { @@ -25,10 +30,10 @@ "id": "string", "file_key": "string", "module_name": "string", + "source": "string", "content_hash": "string", "last_modified": "float", - "file_size": "integer", - "_module": "string" + "file_size": "integer" } }, { @@ -41,10 +46,16 @@ "name": "string", "code": "string", "base_classes": "string[]", + "decorators": "string[]", "docstring": "string", "start_line": "integer", "end_line": "integer", - "_module": "string" + "start_column": "integer", + "end_column": "integer", + "start_byte": "integer", + "end_byte": "integer", + "is_entrypoint": "boolean", + "entrypoint_frameworks": "string[]" } }, { @@ -62,11 +73,17 @@ "code_start_line": "integer", "start_line": "integer", "end_line": "integer", + "start_column": "integer", + "end_column": "integer", + "start_byte": "integer", + "end_byte": "integer", "docstring": "string", "decorators": "string[]", + "modifiers": "string[]", "parameters_json": "string", "accessed_symbols_json": "string", - "_module": "string" + "is_entrypoint": "boolean", + "entrypoint_frameworks": "string[]" } }, { @@ -92,28 +109,8 @@ "merge_label": "PyDecorator", "key": "name", "properties": { - "name": "string" - } - }, - { - "label": "PyCallSite", - "merge_label": "PyCallSite", - "key": "id", - "properties": { - "id": "string", - "method_name": "string", - "receiver_expr": "string", - "receiver_type": "string", - "argument_types": "string[]", - "arguments_json": "string", - "return_type": "string", - "callee_signature": "string", - "is_constructor_call": "boolean", - "start_line": "integer", - "start_column": "integer", - "end_line": "integer", - "end_column": "integer", - "_module": "string" + "name": "string", + "qualified_name": "string" } }, { @@ -127,8 +124,7 @@ "initializer": "string", "docstring": "string", "start_line": "integer", - "end_line": "integer", - "_module": "string" + "end_line": "integer" } }, { @@ -140,24 +136,81 @@ "name": "string", "type": "string", "initializer": "string", + "value_json": "string", "scope": "string", "start_line": "integer", "end_line": "integer", - "_module": "string" + "start_column": "integer", + "end_column": "integer", + "start_byte": "integer", + "end_byte": "integer" } }, { - "label": "PyCFGNode", - "merge_label": "PyCFGNode", + "label": "PyBodyNode", + "merge_label": "PyBodyNode", "key": "id", "properties": { "id": "string", "kind": "string", "var": "string", "call_node": "string", + "method_name": "string", + "receiver_expr": "string", + "receiver_type": "string", + "return_type": "string", + "is_constructor_call": "boolean", + "arguments_json": "string", + "callee_signature": "string", + "start_line": "integer", + "end_line": "integer", + "start_column": "integer", + "end_column": "integer", + "start_byte": "integer", + "end_byte": "integer" + } + }, + { + "label": "Artifact", + "merge_label": "Artifact", + "key": "id", + "properties": { + "id": "string", + "path": "string", + "format": "string", + "roles": "string[]", + "size_bytes": "integer", + "sha256": "string", + "source": "string", + "extraction": "string" + } + }, + { + "label": "Package", + "merge_label": "Package", + "key": "id", + "properties": { + "id": "string", + "ecosystem": "string", + "name": "string" + } + }, + { + "label": "ConfigKey", + "merge_label": "ConfigKey", + "key": "id", + "properties": { + "id": "string", + "key": "string", + "namespace": "string", + "value": "string", + "references": "string[]", "start_line": "integer", "end_line": "integer", - "_module": "string" + "start_column": "integer", + "end_column": "integer", + "start_byte": "integer", + "end_byte": "integer" } } ], @@ -216,20 +269,10 @@ ], "properties": {} }, - { - "type": "PY_HAS_CALLSITE", - "from": [ - "PyCallable" - ], - "to": [ - "PyCallSite" - ], - "properties": {} - }, { "type": "PY_RESOLVES_TO", "from": [ - "PyCallSite" + "PyBodyNode" ], "to": [ "PyCallable", @@ -274,36 +317,48 @@ "properties": { "spellings": "string[]", "imported_names": "string[]", - "aliases": "string[]" + "aliases": "string[]", + "positions_json": "string" } }, { "type": "PY_DECORATED_BY", "from": [ - "PyCallable" + "PyCallable", + "PyClass" ], "to": [ "PyDecorator" ], - "properties": {} + "properties": { + "expression": "string", + "positional_arguments": "string[]", + "keyword_arguments_json": "string", + "start_line": "integer", + "end_line": "integer", + "start_column": "integer", + "end_column": "integer", + "start_byte": "integer", + "end_byte": "integer" + } }, { - "type": "PY_HAS_CFG_NODE", + "type": "PY_HAS_BODY_NODE", "from": [ "PyCallable" ], "to": [ - "PyCFGNode" + "PyBodyNode" ], "properties": {} }, { "type": "PY_CFG_NEXT", "from": [ - "PyCFGNode" + "PyBodyNode" ], "to": [ - "PyCFGNode" + "PyBodyNode" ], "properties": { "kind": "string", @@ -313,20 +368,20 @@ { "type": "PY_CDG", "from": [ - "PyCFGNode" + "PyBodyNode" ], "to": [ - "PyCFGNode" + "PyBodyNode" ], "properties": {} }, { "type": "PY_DDG", "from": [ - "PyCFGNode" + "PyBodyNode" ], "to": [ - "PyCFGNode" + "PyBodyNode" ], "properties": { "var": "string", @@ -337,10 +392,10 @@ { "type": "PY_PARAM_IN", "from": [ - "PyCFGNode" + "PyBodyNode" ], "to": [ - "PyCFGNode" + "PyBodyNode" ], "properties": { "var": "string" @@ -349,10 +404,10 @@ { "type": "PY_PARAM_OUT", "from": [ - "PyCFGNode" + "PyBodyNode" ], "to": [ - "PyCFGNode" + "PyBodyNode" ], "properties": { "var": "string" @@ -361,28 +416,129 @@ { "type": "PY_SUMMARY", "from": [ - "PyCFGNode" + "PyBodyNode" + ], + "to": [ + "PyBodyNode" + ], + "properties": {} + }, + { + "type": "HAS_ARTIFACT", + "from": [ + "PyApplication" + ], + "to": [ + "Artifact" + ], + "properties": {} + }, + { + "type": "DEFINES_CONFIG", + "from": [ + "Artifact" + ], + "to": [ + "ConfigKey" + ], + "properties": {} + }, + { + "type": "DECLARES_DEPENDENCY", + "from": [ + "Artifact" + ], + "to": [ + "Package" + ], + "properties": { + "spec": "string", + "kind": "string", + "extras": "string[]", + "prov": "string[]", + "direct": "boolean", + "_k": "string" + } + }, + { + "type": "LOCKS", + "from": [ + "Artifact" + ], + "to": [ + "Package" + ], + "properties": { + "version": "string" + } + }, + { + "type": "PY_PROVIDES", + "from": [ + "Package" ], "to": [ - "PyCFGNode" + "PyExternal" ], "properties": {} + }, + { + "type": "PY_UNRESOLVED_IMPORT", + "from": [ + "PyApplication" + ], + "to": [ + "PyExternal" + ], + "properties": { + "prov": "string[]" + } + }, + { + "type": "PY_USES_CONFIG", + "from": [ + "PyBodyNode" + ], + "to": [ + "ConfigKey" + ], + "properties": { + "prov": "string[]" + } + }, + { + "type": "PY_READS_CONFIG_UNRESOLVED", + "from": [ + "PyApplication" + ], + "to": [ + "PyExternal" + ], + "properties": { + "key": "string", + "reason": "string", + "prov": "string[]", + "_k": "string" + } } ], "constraints": [ - "CREATE CONSTRAINT pyapplication_name IF NOT EXISTS FOR (x:PyApplication) REQUIRE x.name IS UNIQUE", + "CREATE CONSTRAINT pyapplication_id IF NOT EXISTS FOR (x:PyApplication) REQUIRE x.id IS UNIQUE", "CREATE CONSTRAINT pymodule_id IF NOT EXISTS FOR (x:PyModule) REQUIRE x.id IS UNIQUE", "CREATE CONSTRAINT pysymbol_id IF NOT EXISTS FOR (x:PySymbol) REQUIRE x.id IS UNIQUE", "CREATE CONSTRAINT pypackage_name IF NOT EXISTS FOR (x:PyPackage) REQUIRE x.name IS UNIQUE", "CREATE CONSTRAINT pydecorator_name IF NOT EXISTS FOR (x:PyDecorator) REQUIRE x.name IS UNIQUE", - "CREATE CONSTRAINT pycallsite_id IF NOT EXISTS FOR (x:PyCallSite) REQUIRE x.id IS UNIQUE", "CREATE CONSTRAINT pyattribute_id IF NOT EXISTS FOR (x:PyAttribute) REQUIRE x.id IS UNIQUE", "CREATE CONSTRAINT pyvariable_id IF NOT EXISTS FOR (x:PyVariable) REQUIRE x.id IS UNIQUE", - "CREATE CONSTRAINT pycfgnode_id IF NOT EXISTS FOR (x:PyCFGNode) REQUIRE x.id IS UNIQUE" + "CREATE CONSTRAINT pybodynode_id IF NOT EXISTS FOR (x:PyBodyNode) REQUIRE x.id IS UNIQUE", + "CREATE CONSTRAINT artifact_id IF NOT EXISTS FOR (x:Artifact) REQUIRE x.id IS UNIQUE", + "CREATE CONSTRAINT package_id IF NOT EXISTS FOR (x:Package) REQUIRE x.id IS UNIQUE", + "CREATE CONSTRAINT configkey_id IF NOT EXISTS FOR (x:ConfigKey) REQUIRE x.id IS UNIQUE" ], "indexes": [ "CREATE INDEX py_callable_name IF NOT EXISTS FOR (c:PyCallable) ON (c.name)", "CREATE INDEX py_class_name IF NOT EXISTS FOR (c:PyClass) ON (c.name)", - "CREATE FULLTEXT INDEX py_code_fts IF NOT EXISTS FOR (c:PyCallable) ON EACH [c.code, c.docstring]" + "CREATE FULLTEXT INDEX py_code_fts IF NOT EXISTS FOR (c:PyCallable) ON EACH [c.code, c.docstring]", + "CREATE INDEX py_can_node_id IF NOT EXISTS FOR (n:PyCanNode) ON (n.id)" ] } diff --git a/schema.neo4j.json b/schema.neo4j.json index d779e77..7a91e07 100644 --- a/schema.neo4j.json +++ b/schema.neo4j.json @@ -30,6 +30,7 @@ "id": "string", "file_key": "string", "module_name": "string", + "source": "string", "content_hash": "string", "last_modified": "float", "file_size": "integer" @@ -49,6 +50,10 @@ "docstring": "string", "start_line": "integer", "end_line": "integer", + "start_column": "integer", + "end_column": "integer", + "start_byte": "integer", + "end_byte": "integer", "is_entrypoint": "boolean", "entrypoint_frameworks": "string[]" } @@ -68,6 +73,10 @@ "code_start_line": "integer", "start_line": "integer", "end_line": "integer", + "start_column": "integer", + "end_column": "integer", + "start_byte": "integer", + "end_byte": "integer", "docstring": "string", "decorators": "string[]", "modifiers": "string[]", @@ -127,9 +136,14 @@ "name": "string", "type": "string", "initializer": "string", + "value_json": "string", "scope": "string", "start_line": "integer", - "end_line": "integer" + "end_line": "integer", + "start_column": "integer", + "end_column": "integer", + "start_byte": "integer", + "end_byte": "integer" } }, { @@ -147,8 +161,13 @@ "return_type": "string", "is_constructor_call": "boolean", "arguments_json": "string", + "callee_signature": "string", "start_line": "integer", - "end_line": "integer" + "end_line": "integer", + "start_column": "integer", + "end_column": "integer", + "start_byte": "integer", + "end_byte": "integer" } }, { @@ -187,7 +206,11 @@ "value": "string", "references": "string[]", "start_line": "integer", - "end_line": "integer" + "end_line": "integer", + "start_column": "integer", + "end_column": "integer", + "start_byte": "integer", + "end_byte": "integer" } } ], @@ -294,7 +317,8 @@ "properties": { "spellings": "string[]", "imported_names": "string[]", - "aliases": "string[]" + "aliases": "string[]", + "positions_json": "string" } }, { @@ -309,7 +333,13 @@ "properties": { "expression": "string", "positional_arguments": "string[]", - "keyword_arguments_json": "string" + "keyword_arguments_json": "string", + "start_line": "integer", + "end_line": "integer", + "start_column": "integer", + "end_column": "integer", + "start_byte": "integer", + "end_byte": "integer" } }, { diff --git a/test/test_neo4j_config_keys.py b/test/test_neo4j_config_keys.py index c36db30..d4fae63 100644 --- a/test/test_neo4j_config_keys.py +++ b/test/test_neo4j_config_keys.py @@ -13,7 +13,11 @@ def test_catalog_has_config_key_label_and_rel(): assert config_key.properties == { "id": "string", "key": "string", "namespace": "string", "value": "string", "references": "string[]", + # The shared _SPAN, widened to columns + utf-8 byte offsets in #202 so a + # key's position slices out of the artifact text like any other node's. "start_line": "integer", "end_line": "integer", + "start_column": "integer", "end_column": "integer", + "start_byte": "integer", "end_byte": "integer", } rels = {r.type: r for r in REL_TYPES} defines = rels["DEFINES_CONFIG"] diff --git a/test/test_neo4j_schema.py b/test/test_neo4j_schema.py index 6a33a77..15bdd59 100644 --- a/test/test_neo4j_schema.py +++ b/test/test_neo4j_schema.py @@ -172,3 +172,15 @@ def test_param_edges_carry_every_declared_property(): assert not missing, f"{rel} {e.from_ref.value} -> {e.to_ref.value} lacks {missing}" assert e.props["var"], f"{rel} var must be non-empty" + +def test_handoff_schema_snapshot_matches_the_root_one(): + """The handoff bundle ships its own copy of the contract + (``docs/handoff/schema.neo4j.json``, referenced by ``docs/handoff/README.md``) so + it stays self-contained. Nothing guarded it, and it had already drifted two node + labels behind the root snapshot before #202/#203 touched either. Regenerate both + with ``canpy --emit schema``.""" + root = Path(__file__).resolve().parents[1] / "schema.neo4j.json" + handoff = Path(__file__).resolve().parents[1] / "docs" / "handoff" / "schema.neo4j.json" + assert json.loads(handoff.read_text()) == json.loads(root.read_text()), ( + "docs/handoff/schema.neo4j.json is stale — regenerate both copies" + ) diff --git a/test/test_v2_two_projection_agreement.py b/test/test_v2_two_projection_agreement.py index 8fc9bdb..8d41cd8 100644 --- a/test/test_v2_two_projection_agreement.py +++ b/test/test_v2_two_projection_agreement.py @@ -340,3 +340,168 @@ def test_pyapplication_carries_the_entrypoint_report(): rows = project(PyApplication(symbol_table={}), "app", {}) node = next(n for n in rows.nodes if n.labels[0] == "PyApplication") assert "entrypoint_report_json" in node.props + + +# ---------------------------------------------------------------------------------------------- +# JSON == Graph parity (#202, #203); spec docs/design/specs/2026-09-10-neo4j-json-parity.md +# ---------------------------------------------------------------------------------------------- + +_SPAN_KEYS = ("start_line", "end_line", "start_column", "end_column", "start_byte", "end_byte") + + +def _sample_rows(): + from sample_graph_app import make_sample_app + + app, sig_to_id = make_sample_app() + return app, project(app, "sample-app", sig_to_id) + + +def test_module_node_carries_source_and_its_hash_matches_content_hash(): + """#202: the module's whole-file text is on the graph, intact. The hash check is + what proves it survived serialization rather than merely being present.""" + import hashlib + + app, rows = _sample_rows() + mods = [n for n in rows.nodes if "PyModule" in n.labels] + assert mods, "no :PyModule rows projected" + by_id = {m.id: m for m in app.symbol_table.values()} + for row in mods: + mod = by_id.get(row.value) + if mod is None: + continue # a :PyModule ref minted for an unresolved import target + assert "source" in row.props, f"{row.value} carries no source" + assert row.props["source"] == mod.source + assert ( + hashlib.sha256(row.props["source"].encode("utf-8")).hexdigest() + == mod.content_hash + ) + + +def test_source_is_present_even_when_empty(): + """An empty file yields "", never an absent property — otherwise a consumer + cannot tell "not carried" from "empty".""" + mod = PyModule(file_path="e.py", module_name="e", source="") + app = PyApplication(symbol_table={"e.py": mod}) + sig_to_id = assign_ids(app, "myapp") + rows = project(app, "myapp", sig_to_id) + row = next(n for n in rows.nodes if "PyModule" in n.labels) + assert row.props["source"] == "" + + +def test_every_span_bearing_node_carries_all_six_span_properties(): + """#202: `_SPAN` means one thing on every label that spreads it — a consumer + never has to know which labels are sliceable.""" + _, rows = _sample_rows() + spanned = [n for n in rows.nodes if any(k in n.props for k in _SPAN_KEYS)] + assert spanned, "no span-bearing rows projected" + for row in spanned: + # :PyAttribute is the declared exception -- PyClassAttribute carries no + # columns in JSON, so no byte offsets are derivable. Asserted explicitly so + # the exception cannot widen silently. + if "PyAttribute" in row.labels: + assert set(row.props) & set(_SPAN_KEYS) == {"start_line", "end_line"} + continue + missing = [k for k in _SPAN_KEYS if k not in row.props] + assert not missing, f"{row.labels} {row.value} missing {missing}" + # the computed-byte label must actually be exercised by the fixture + labels = {lbl for row in spanned for lbl in row.labels} + assert "PyVariable" in labels and "PyAttribute" in labels + + +def test_byte_offsets_slice_the_module_source_to_the_nodes_own_text(): + """#202: proof the offsets are real rather than declared.""" + app, rows = _sample_rows() + source_by_module = {m.id: m.source for m in app.symbol_table.values()} + checked = 0 + for row in rows.nodes: + code = row.props.get("code") + if code is None or "start_byte" not in row.props: + continue + source = next( + (s for mid, s in source_by_module.items() if row.value.startswith(mid)), None + ) + assert source is not None, f"no owning module source for {row.value}" + sliced = source.encode("utf-8")[ + row.props["start_byte"] : row.props["end_byte"] + ].decode("utf-8") + assert sliced == code, f"{row.value}: byte span does not slice to its code" + checked += 1 + assert checked, "no node carried both code and byte offsets" + + +def test_body_node_callee_signature_equals_the_matching_callsite(): + """#203: carried by a positional join from `call_sites`, asserted per call site + rather than by presence — a positional join can miss silently.""" + from codeanalyzer.semantic_analysis.call_graph import _walk_module_callables + + app, rows = _sample_rows() + by_id = {n.value: n for n in rows.nodes if "PyBodyNode" in n.labels} + expected = 0 + for mod in app.symbol_table.values(): + for c in _walk_module_callables(mod): + for cs in c.call_sites or []: + if not cs.callee_signature: + continue + expected += 1 + row = by_id.get(_global_ordinal(c.id, f"{cs.start_line}:{cs.start_column}")) + assert row is not None, ( + f"no body node at {cs.start_line}:{cs.start_column} in {c.id}" + ) + assert row.props.get("callee_signature") == cs.callee_signature + assert expected, "fixture has no call site with a callee_signature" + + +def test_variable_value_json_round_trips_to_the_json_value(): + """#203: `value` is Optional[Any] and Neo4j takes scalars or arrays of scalars, + so it is always JSON-encoded — one shape for every value, dict included. Built + directly rather than off the shared fixture, whose literal evaluator captures + only `ast.Constant` and so never populates a dict `value`.""" + import json as _json + from codeanalyzer.schema.py_schema import PyVariableDeclaration + + scalar = PyVariableDeclaration(name="PORT", initializer="8080", value=8080, + start_line=1, start_column=0, end_line=1, end_column=11) + nested = PyVariableDeclaration(name="CONF", initializer="{'a': [1, 2]}", + value={"a": [1, 2]}, + start_line=2, start_column=0, end_line=2, end_column=20) + mod = PyModule(file_path="m.py", module_name="m", + source="PORT = 8080\nCONF = {'a': [1, 2]}\n", + variables=[scalar, nested]) + app = PyApplication(symbol_table={"m.py": mod}) + rows = project(app, "myapp", assign_ids(app, "myapp")) + by_name = {n.props["name"]: n for n in rows.nodes if "PyVariable" in n.labels} + assert _json.loads(by_name["PORT"].props["value_json"]) == 8080 + assert _json.loads(by_name["CONF"].props["value_json"]) == {"a": [1, 2]} + # initializer stays the raw source text, value_json the evaluated result + assert by_name["CONF"].props["initializer"] == "{'a': [1, 2]}" + + +def test_decorator_span_rides_the_relationship_not_the_shared_node(): + """#203: :PyDecorator is merged on qualified_name and never pruned, so a + per-application fact on it would accumulate across every project in the DB.""" + _, rows = _sample_rows() + decorated = [e for e in rows.edges if e.type == "PY_DECORATED_BY"] + assert decorated, "fixture projects no PY_DECORATED_BY edge" + for e in decorated: + missing = [k for k in _SPAN_KEYS if k not in (e.props or {})] + assert not missing, f"PY_DECORATED_BY missing {missing}" + for n in rows.nodes: + if "PyDecorator" in n.labels: + assert not any(k in n.props for k in _SPAN_KEYS), \ + "the shared :PyDecorator node must carry no per-application span" + + +def test_py_imports_carries_a_position_for_every_spelling(): + """#203: the edge pre-aggregates per (module, target) and emits `spellings` + sorted, so positions key on the spelling — index alignment is already lost.""" + import json as _json + + _, rows = _sample_rows() + imports = [e for e in rows.edges if e.type == "PY_IMPORTS"] + assert imports, "fixture projects no PY_IMPORTS edge" + for e in imports: + positions = _json.loads(e.props["positions_json"]) + assert set(positions) == set(e.props["spellings"]), \ + "every spelling on the edge needs a position" + for pos in positions.values(): + assert len(pos) == 4 # start_line, start_column, end_line, end_column