Repository navigation
Closed
Labels
enhancementNew feature or requestNew feature or request
Description
Epic: migrate codeanalyzer-python to canonical schema v2 + four analysis levels
Migrate the analyzer from the v1 payload (PyApplication + a separate program_graphs section) to the CLDK canonical schema v2 — one additive Code Property Graph tree — exposed as four gated analysis levels (-a 1|2|3|4), in both the analysis.json and Neo4j projections. This is a breaking major release (the codeanalyzer-backend "existing analyzer → schema v2" path). It supersedes the L3 approach in #67: dataflow moves inline into the tree, and the interprocedural SDG moves from -a 3 to -a 4.
Target shape
application → symbol_table{module} → types/functions → callables{ body{}, cfg[], cdg[], ddg[], summary[] }
call_graph[], param_in[], param_out[] — cross-function edges, at application scope
can://python/<app>/<file>/<type>/<callable-sig>durable ids (≥ callable);<callable-id>@<line>:<col>/@tagordinal ids (< callable).sourcestored once per module + byte-offset spans (every node's text slices off it); the per-callablecodefield is dropped.- Payload envelope:
schema_version: "2.0.0",language,max_level,k_limit.
Levels
| Level | Grows | Adds edges |
|---|---|---|
| 1 | tree to callable + call nodes in body (callee:null) |
— |
| 2 | callee: null→id backfill |
call_graph |
| 3 | rest of body + @entry/@exit |
cfg, cdg, ddg (syntactic, prov:["ssa"]) |
| 4 | synthetic param vertices | param_in, param_out, summary, + ddg (semantic, prov:["points-to"]) |
--graphs sdg requires -a 4 (requesting it at -a 3 is a flag error).
Key decisions
- Reuse the existing dataflow kernels (
cfg/dominance/defuse/sdg/summaries/alias); a new identity-mapping layer translates internal(signature, node_id)keys →can://ids at emit time, which keeps the JSON and Neo4j projections in lockstep. - Scalpel (
python-scalpel) is the primary L4 points-to oracle behind the frozenmay_aliasinterface, withTypeBasedAliasOracleas an automatic fallback. - Analyzer-only scope. The python-sdk Pydantic revision is a separate follow-up (via
cldk-sdk-frontend). Taint stays deferred to the SDK; the backward slicer stays internal as the L3/L4 validation gate (no product "queryable slicer" framing, notaint_flowssection). - Neo4j is a re-identify + one
provproperty + source/span parity + MAJOR bump (schema.neo4j.json→2.0.0); the near-identity projection structure is unchanged. Always full-depth.
Invariants / gates
- Additive superset:
analysis.json(-a 1) ⊆ … ⊆ analysis.json(-a 4), modulo thecallee: null→idrefinement and the L3→L4 DDGssa→points-towidening. - Two-projection agreement: JSON node/edge ids == Neo4j merge-key values.
- Per-level gates (validated per stage): L1 canonical-schema conformance; L2 no dangling call-graph endpoints; L3 intraprocedural backward slice == hand-computed set; L4 no dangling SDG endpoints + PARAM arity + a
summaryfor a known transitive flow.
Stages
Each stage is an independently shippable PR-unit with its own gate. Child issues:
- v2 Stage 0: Scalpel / toolchain verification spike #70 — Stage 0: Scalpel / toolchain verification spike
- v2 Stage 1: schema foundation + L1 #71 — Stage 1: schema foundation + L1
- v2 Stage 2: L2 (callee backfill + call_graph re-identity) #72 — Stage 2: L2 (callee backfill + call_graph re-identity)
- v2 Stage 3: L3 intraprocedural CFG/CDG/DDG (syntactic) #73 — Stage 3: L3 intraprocedural CFG/CDG/DDG (syntactic)
- v2 Stage 4: L4 interprocedural SDG + Scalpel oracle #74 — Stage 4: L4 interprocedural SDG + Scalpel oracle
- v2 Stage 5: docs, versioning, hand-off #75 — Stage 5: docs, versioning, hand-off
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request
Activity
rahlk commentedon Jul 14, 2026
Migration complete — all six stages shipped as stacked PRs
The schema-v2 migration is done. Every stage was built subagent-driven (implement → review → fix) and passed a whole-branch review as merge-ready.
Analysisenvelope,can://ids,source/spans, L1 body, Neo4j re-key, envelope cachecall_graphre-identity, restoredPY_EXTENDS/PY_RESOLVES_TObodystatements +cfg/cdg/ddg(ssa), CPG overlay, cache-level fixScalpelAliasOracle+fallback, param vertices,param_in/out/summary, points-to ddg0.4.0, README/CLAUDE/AGENTS, hand-off bundleMerge order (stacked; each targets its parent): #76 → #77 → #78 → #79 → #81 → #88, all landing on
feat/level3-dataflow-sdg.Result:
codeanalyzer-pythonemits canonical schema v2 across four gated levels (-a 1|2|3|4) in bothanalysis.jsonand Neo4j, with Scalpel as the L4 points-to oracle (optional, type-based fallback).L1 ⊆ L2 ⊆ L3 ⊆ L4is a CI gate; two-projection agreement holds; the slicer stays internal (notaint_flows).Follow-up (out of scope here): the python-sdk Pydantic model revision to v2 is the separate
cldk-sdk-frontendrelease — it consumes the hand-off bundle indocs/handoff/.rahlk commentedon Jul 15, 2026
Epic complete — released as v1.0.0 (
schema_version 2.0.0).All six stages landed on main via the stacked chain, merged at d477207 (PR #88):
-a 4run (6,086 modules, 44,840 call edges, 3.28M DDG edges) passes every keystone gate, zero dangling endpoints, and round-trips the Analysis model.Follow-ons tracked separately: PyCG L2 nondeterminism (#99, known limitation in the release notes), v2-chain audit counterparts (#87, #83), SDK uptake (python-sdk epic codellm-devkit/python-sdk#238 — pin target corrected to 1.0.0 on codellm-devkit/python-sdk#242).