Repository navigation
Description
Is your feature request related to a problem? Please describe.
Two live parity modules are gated on CLDK_TEST_NEO4J_WRITE_*, which no leg has ever set. They have been skipped continuously through legs 1, 1.5, 1.6, 2.5a and 2.5b while the behaviour they pin was deliberately changed underneath them. Pointed at a disposable server for the first time, both fail:
| module | result |
|---|---|
tests/analysis/typescript/test_typescript_neo4j_backend.py |
5 failed |
tests/analysis/python/test_python_neo4j_backend.py |
5 failed, 1 passed |
None of the ten is an SDK regression. Every one asserts behaviour that a later leg intentionally replaced, and each replacement is documented elsewhere and asserted by the read-only live suites. The defect is that a whole verification layer went quiet and nothing noticed, because a skip is not a failure.
Describe the solution you'd like
- Both write-gated modules pass against a disposable server, asserting what the code does today
- Each tolerance is stated with its cause, in the idiom
test_java_neo4j_backend.pyalready uses — not skipped, not loosened toassert True -
typescript_analysis.py:154no longer promises atagsedge attribute that nothing sets - A skipped write-gated module is visible: the suite reports at the end that these modules were skipped and what to set, or CI runs them against a throwaway container
Describe alternatives you've considered
Not stated in the original issue.
Additional context
What the ten actually assert
TypeScript (5)
test_fields_and_parameters,test_get_method_parameters_module_level_function— expectget_method_parametersto return parameters. It now raisesCodeanalyzerExecutionException("...carries no parameters..."), which is deliberate and whichtest_typescript_e2e_neo4j_live.pyasserts withpytest.raises(..., match="no parameters for").test_call_graph_no_dangling_nodes,test_callers_and_callees— assert"tags" in edge_data. The edge carries{provenance, type, weight}.tagsis set nowhere incldk/— its only surviving mention is a docstring atcldk/analysis/typescript/typescript_analysis.py:154promising callers-of edges carry "provenance/tags". The docstring is wrong, not the edge.test_call_sites— expectsreceiver_type == "UserService"; it isNone.docs/agent-api-reference.md:128lists "a call site'smethod_name, receiver and args" among the projection's documented losses.
Python (5)
All five compare an in-memory module against the Neo4j-rebuilt one with plain ==, where the projection is documented to be lossy. On one module the diff is exactly: file_path absolute vs repo-relative, id present vs '', source present vs ''. The Java live suite handles the same situation correctly — it carries an explicit, reasoned tolerance list rather than asserting equality.
Scope boundary
In scope: make all ten assert current, documented behaviour — with tolerances stated and justified the way test_java_neo4j_backend.py states its own — fix the tags docstring, and make the gate impossible to leave unnoticed again.
Out of scope: changing any SDK behaviour. Every current answer here is the intended one.
Caveats and known risks
- The write target must be disposable. These modules emit a fixture project over Bolt and delete that application's subgraph afterwards. Pointing them at a reference graph would destroy it; that is why the gate exists and why it has no default (test_python_neo4j_backend.py writes to whatever CLDK_TEST_NEO4J_URI points at, and never cleans up #324 — a default of
bolt://localhost:7687is an ssh tunnel on at least one development machine). Keep the no-default rule. - Fixing the assertions is not the same as restoring the coverage. Two of the TypeScript tests will end up asserting a refusal, which is weaker than what they were written to check; say so where it is true rather than leaving a reader to assume parity is proven.
Definition of done
- Ten green, each pinning behaviour that exists, and a mechanism that makes the next silent skip loud.
Activity
rahlk commentedon Sep 7, 2026
Fixed. All ten green, no SDK behaviour changed.
The TypeScript half is in #357, the Python half in #358.
Two of the five Python failures were not projection lossiness at all, which is worth recording because it shortened the tolerance list rather than lengthening it:
AnalysisOptionswith the defaultanalysis_level=1while comparing against a reference built atcall_graph(2). At level 1 the graph carries zeroPY_RESOLVES_TOedges and nodefuse-provenance call edges, so everycallee_signaturewasNoneand one edge was absent. A level difference was being read as a projection loss. At level 2 the call graph matches exactly andtest_call_graph_parityneeds no tolerance.can://python/<app>/segment from the project directory name while the emitter is passedapp_name. Naming the temp directory after the app collapses a whole class of id disagreement, including external-target ids in the call graph.Every unnecessary tolerance is somewhere a real regression could hide later, so the shorter list is the point.
tagswas not an SDK gap. Schema v2'sTSCallGraphEdgeis{src, dst, prov, weight}andPyCallEdgelikewise;tagswas a schema-1.0.0 call-edge field that v2 dropped, and both TypeScript backends already agree on{type, weight, provenance}. The docstring was five legs stale. The edge assertions moved from membership to an exact key set over every edge — membership is precisely what let a stale key sit unnoticed.Three real gaps found and left unfixed under the no-behaviour-change rule, filed as #361:
id,PyAttribute.initializerandPyCallsite.argumentsare all carried on the graph and never read byreconstruct. Each is asserted in both halves — graph property present, model field empty — so closing any of them breaks loudly.Where an assertion ended up weaker than what it was written to prove, the test says so in place. Two of the TypeScript ones now assert a refusal, not parity.
The silent-skip mechanism is a
pytest_terminal_summaryhook intests/conftest.py: it lists the write-gated modules a run skipped and the variables to set. Print-only, no dependency, never touches exit status, and keyed on the variable name rather than a module list — so it already catches a third write-gated module without being told about it. Both modules still skip cleanly when the variable is unset, and the no-default rule from #324 is intact.