Skip to content

Write-gated Neo4j parity modules have been skipped through every leg; all ten tests now fail against current behaviour #360

Open

Description

@rahlk

Is your feature request related to a problem? Please describe.

Two live parity modules are gated on CLDK_TEST_NEO4J_WRITE_*, which no leg has ever set. They have been skipped continuously through legs 1, 1.5, 1.6, 2.5a and 2.5b while the behaviour they pin was deliberately changed underneath them. Pointed at a disposable server for the first time, both fail:

module result
tests/analysis/typescript/test_typescript_neo4j_backend.py 5 failed
tests/analysis/python/test_python_neo4j_backend.py 5 failed, 1 passed

None of the ten is an SDK regression. Every one asserts behaviour that a later leg intentionally replaced, and each replacement is documented elsewhere and asserted by the read-only live suites. The defect is that a whole verification layer went quiet and nothing noticed, because a skip is not a failure.

Describe the solution you'd like

  • Both write-gated modules pass against a disposable server, asserting what the code does today
  • Each tolerance is stated with its cause, in the idiom test_java_neo4j_backend.py already uses — not skipped, not loosened to assert True
  • typescript_analysis.py:154 no longer promises a tags edge attribute that nothing sets
  • A skipped write-gated module is visible: the suite reports at the end that these modules were skipped and what to set, or CI runs them against a throwaway container

Describe alternatives you've considered

Not stated in the original issue.

Additional context

What the ten actually assert

TypeScript (5)

  • test_fields_and_parameters, test_get_method_parameters_module_level_function — expect get_method_parameters to return parameters. It now raises CodeanalyzerExecutionException("...carries no parameters..."), which is deliberate and which test_typescript_e2e_neo4j_live.py asserts with pytest.raises(..., match="no parameters for").
  • test_call_graph_no_dangling_nodes, test_callers_and_callees — assert "tags" in edge_data. The edge carries {provenance, type, weight}. tags is set nowhere in cldk/ — its only surviving mention is a docstring at cldk/analysis/typescript/typescript_analysis.py:154 promising callers-of edges carry "provenance / tags". The docstring is wrong, not the edge.
  • test_call_sites — expects receiver_type == "UserService"; it is None. docs/agent-api-reference.md:128 lists "a call site's method_name, receiver and args" among the projection's documented losses.

Python (5)

All five compare an in-memory module against the Neo4j-rebuilt one with plain ==, where the projection is documented to be lossy. On one module the diff is exactly: file_path absolute vs repo-relative, id present vs '', source present vs ''. The Java live suite handles the same situation correctly — it carries an explicit, reasoned tolerance list rather than asserting equality.

Scope boundary

In scope: make all ten assert current, documented behaviour — with tolerances stated and justified the way test_java_neo4j_backend.py states its own — fix the tags docstring, and make the gate impossible to leave unnoticed again.

Out of scope: changing any SDK behaviour. Every current answer here is the intended one.

Caveats and known risks

  • The write target must be disposable. These modules emit a fixture project over Bolt and delete that application's subgraph afterwards. Pointing them at a reference graph would destroy it; that is why the gate exists and why it has no default (test_python_neo4j_backend.py writes to whatever CLDK_TEST_NEO4J_URI points at, and never cleans up #324 — a default of bolt://localhost:7687 is an ssh tunnel on at least one development machine). Keep the no-default rule.
  • Fixing the assertions is not the same as restoring the coverage. Two of the TypeScript tests will end up asserting a refusal, which is weaker than what they were written to check; say so where it is true rather than leaving a reader to assume parity is proven.

Definition of done

  • Ten green, each pinning behaviour that exists, and a mechanism that makes the next silent skip loud.

Activity

  1. rahlk commented on Sep 7, 2026

    @rahlk
    CollaboratorAuthor

    Fixed. All ten green, no SDK behaviour changed.

    run before after
    TypeScript write-gated module vs a disposable server 5 failed / 14 passed 19 passed
    Python write-gated module vs a disposable server 5 failed / 1 passed 6 passed
    Whole TypeScript suite vs superset, every gate satisfied 5 failed / 539 passed / 5 xfailed 544 passed, 5 xfailed, 0 failed, 0 skipped

    The TypeScript half is in #357, the Python half in #358.

    Two of the five Python failures were not projection lossiness at all, which is worth recording because it shortened the tolerance list rather than lengthening it:

    • The fixture built its AnalysisOptions with the default analysis_level=1 while comparing against a reference built at call_graph (2). At level 1 the graph carries zero PY_RESOLVES_TO edges and no defuse-provenance call edges, so every callee_signature was None and one edge was absent. A level difference was being read as a projection loss. At level 2 the call graph matches exactly and test_call_graph_parity needs no tolerance.
    • The in-memory backend derives the can://python/<app>/ segment from the project directory name while the emitter is passed app_name. Naming the temp directory after the app collapses a whole class of id disagreement, including external-target ids in the call graph.

    Every unnecessary tolerance is somewhere a real regression could hide later, so the shorter list is the point.

    tags was not an SDK gap. Schema v2's TSCallGraphEdge is {src, dst, prov, weight} and PyCallEdge likewise; tags was a schema-1.0.0 call-edge field that v2 dropped, and both TypeScript backends already agree on {type, weight, provenance}. The docstring was five legs stale. The edge assertions moved from membership to an exact key set over every edge — membership is precisely what let a stale key sit unnoticed.

    Three real gaps found and left unfixed under the no-behaviour-change rule, filed as #361: id, PyAttribute.initializer and PyCallsite.arguments are all carried on the graph and never read by reconstruct. Each is asserted in both halves — graph property present, model field empty — so closing any of them breaks loudly.

    Where an assertion ended up weaker than what it was written to prove, the test says so in place. Two of the TypeScript ones now assert a refusal, not parity.

    The silent-skip mechanism is a pytest_terminal_summary hook in tests/conftest.py: it lists the write-gated modules a run skipped and the variables to set. Print-only, no dependency, never touches exit status, and keyed on the variable name rather than a module list — so it already catches a third write-gated module without being told about it. Both modules still skip cleanly when the variable is unset, and the no-default rule from #324 is intact.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions