Repository navigation
schema-retirement #5: retire query-path aggregation_id, route directly through sid catalog #272
Description
Tracking issue for the final step of the agg_id retirement migration. Steps #1-#4 already landed (sid-level eviction, sid-level reconcile from streaming-config, sid-level §6.3 ingest barrier, big-bang SchemaRegistry deletion — see PRs #183-#187 + #189). Step #5 is the remaining surface: the query path still resolves precomputed outputs via StreamingConfig.aggregation_id, and ingest still buckets per-(aggregation_id, group_key) even though storage downstream is sid-keyed.
This is the issue that #271 (MVP demo: agent emits full-attr sketches → no sid match) calls out as one of two fix paths for Axis C of the MVP smoke test.
Concrete sites still holding agg_id
data_plane/src/query_engines/asap_query_engine/engine.rs
:2392 "Until schema-retirement #5 ports the per-segment dispatch
to sid-level evaluation directly …"
:945, :990, :969, :1042, :1127, :2066, :2099, :2197, :2305
full per-segment dispatch + error paths still keyed on agg_id
data_plane/src/drivers/ingest/otel.rs
:561 by_group.entry((config.aggregation_id(), group_key))
:640 WorkerMessage::AccumulatorInput { agg_id: config.aggregation_id(), … }
:1203 // DEPRECATED: aggregation_id-keyed write — remove
:520, :723 "Schema retirement #5 — agg_id-keyed `SchemaRegistry` is gone"
(already done; reference comments)
data_plane/src/precompute_engine/output_sink.rs
:206, :232 output sink still tags writes with cfg.aggregation_id()
data_plane/src/storage_engines/sketch_db/backfill/processor.rs
:343, :350, :418, :440, :596, :646 backfill processor maps still keyed on agg_id
data_plane/src/storage_engines/types/precomputed_output.rs
:40, :162 `// with aggregation_id kept for legacy compat; PR 5 retired …`
data_plane/src/main.rs
:491, :562, :739, :784 `// Schema retirement #5 — …` reference comments
What "done" looks like
- Query path's
build_query_execution_context_promql_for_agg_id(engine.rs:2197) folded into a sid-keyed equivalent that takes(metric, agg_kind, attrs_subset)and finds candidate sids whosegroup_by_keys ⊇ attrs_subset, then merges across them. - Ingest bucketing in
otel.rs:561rekeyed from(aggregation_id, group_key)→(sid, …)or(metric, agg_kind, group_key) → sids. output_sink.rsandbackfill/processor.rsrekeyed from agg_id → sid.PolicyFingerprint::as_u64()survives as the streaming-config identity for the OUTSIDE world (controller → backend POST, persistence), but the internalconfig.aggregation_id()accessor disappears from hot paths.- Re-enable the two ignored cross-reconfigure dispatch tests in
tests/schema_timeline_dispatch_tests.rs(gated by sid-level per-segment dispatch — fixture needs reworking too because the two test configs collapse to one sid under content-addressing; seeasapquery-backend-phase5-m2-3-progressnotes). - Delete
pub fn aggregation_id()onAggregationConfig(or downgrade to test-only) once all consumers are gone.
Why this unblocks the MVP demo
With schema-retirement #5 done, the query path resolves directly from the metric name + asked grouping to a set of sids in the catalog — no PolicyFingerprint detour. The full-attr sketches the agent currently emits (because of ASAPCollector#381 — agent doesn't apply OpAMP runtime config) become queryable: their group_by_keys is a superset of [zone], so a sum by (zone) query merges across them at execution time.
(Until #5 lands, the alternate unblock is to have the agent group-reduce upstream of the sketch — needs ASAPCollector#381 first.)
Done already (don't redo)
- agg_id-keyed
SchemaRegistrydeleted (PR feat(sketch_db): retire schema/ module — sid-only lifecycle #189) - sid-level eviction sweep (PRs feat(sketch_db): sid-level timeline_for_metric #183-feat(sketch_db): sid-level §6.3 ingest barrier; drop schemas.is_writable #187)
- sid-level reconcile from streaming-config (PR feat(sketch_db): sid-level reconcile_from_streaming_config #186)
- §6.3 ingest barrier moved to sid (PR feat(sketch_db): sid-level §6.3 ingest barrier; drop schemas.is_writable #187)
- HTTP
/api/v1/db/schemas?:agg_idendpoints rewired to:sid
See asapquery-backend-phase5-m2-3-progress memory + the post-M2.3 sketch_db reorg notes for the broader context.
Tagged mvp-demo, query-engine, schema-retirement-5.
Activity
fix(query): union instances_matching into ASAP-tier sid resolution (s…
zzylol commentedon May 17, 2026
Step 1 (query path: union
instances_matchinginto ASAP-tier sid resolution at the two modern execute() sites) merged in PR #273. End-to-end smoke test now returns a real DDSketch quantile via the modern execute() path. See PR for the before/after.Still TODO (each its own PR):
handle_query/execute_contextpath (engine.rs:945–1127) still emits theNo precomputed outputs found for metric: X, aggregation_id: YWARN for every query. The path is already "superseded" —process_via_simple_enginefalls over to modern execute() when legacy returns None, and post-fix(query): union instances_matching into ASAP-tier sid resolution (schema-retirement #5 step 1) #273 the fallback succeeds. Options: (a) downgrade the WARN to DEBUG since it's no longer fatal, (b) retire the legacy path entirely.WorkerMessage::AccumulatorInput { agg_id, group_key }). Touches the precompute worker's internal hashmap shape.config.aggregation_id()sites inoutput_sink.rs+backfill/processor.rs.tests/schema_timeline_dispatch_tests.rs.pub fn aggregation_id()onAggregationConfigonce all consumers are gone.fix(query): port resolve_sketch_metric_alias to modern execute() (sch…
zzylol commentedon May 17, 2026
Step 2 (port
resolve_sketch_metric_aliasto modern execute() and execute_range_promql_modern) merged in PR #274. Modern can now do the_quantile/_hllrewrite that legacy did.Next sub-steps, in approximate dependency order:
Sequential (each unblocks the next):
3. Spawn capability-miss notify from modern
execute()— prerequisite for the call-order flip4. Reorder
process_via_simple_engineto call modern first; investigate the engine.rs:792 time=0 underflow that surfaces when legacy is reached via the modern-Err fallback5. Migrate
try_handle_query_promql_via_timelineto sid-level per-segment dispatch (the last legacy-only feature)6. Delete the legacy
handle_query/handle_query_promql/execute_contextfamilyIndependent (can be parallelized with 3–6):
otel.rs:561,640from(aggregation_id, group_key)→ sid. Cascades throughWorkerMessage::AccumulatorInput,precompute_engine::{worker,output_sink}. Likely needs 2 PRs (worker shape change is one; output_sink + downstream is another).sketch_db/backfill/processor.rs— 6 sites).End-state cleanup (once everything above lands):
pub fn aggregation_id()onAggregationConfig(andPolicyFingerprint::as_u64()if it has no other consumers)revert+retire(query): delete dead resolve_sketch_metric_alias rewrite (…
fix(query): saturating_sub for time=0 + clean up dangling resolve_ske…
refactor(query): retire legacy handle_query path + delete ast_matchin…
refactor(query): delete legacy handle_query family (B7.5 v2 redo) (#280)
zzylol commentedon May 18, 2026
Status update 2026-05-17 — progress on the 5 remaining sub-PRs:
handle_queryfamily retirementotel.rs+WorkerMessage::AccumulatorInput+ worker bucketing)aggregation_id()sites)pub fn aggregation_id()onAggregationConfigAlso-relevant landed this wave (not in the original step list but they cleared adjacent debt):
resolve_sketch_metric_alias(sketch processors don't rename metric names since the 2026-05 refactor)MetricSuffixfield from sketch processors for the same reasonemit/stage_config.rs(BTreeMap for deterministic yaml)ReportsRemoteConfigcapability, paired with fix(emit): deterministic YAML via BTreeMap (kills agent apply-loop) #281 to fully break the apply-loopNext: B7.6 PR coming; B7.7 follows.
refactor(data_plane): rekey backfill processor + SketchStore from agg…
docs(test): clarify ignore reason for cross-reconfigure timeline test (…
zzylol commentedon May 18, 2026
Status update 2026-05-18 — remaining checklist:
handle_queryfamily retirementpanic!("ignored: …")stubs were placeholders awaiting a port that became unnecessary when the legacy dispatcher was retiredpub fn aggregation_id()onAggregationConfigStep 5 awaits a final sweep of remaining
policy_fp.as_u64()users (these are legitimate policy-registry uses of the same numeric — the accessor name is misleading but the call sites are correct). PR #286 enumerated the surface (~13 sites inStreamingConfigmap-key construction).Also along the way, surfaced + closed:
format_series_key/parse_labels_from_series_keyquote-roundtrip bug (fix(data_plane): make format_series_key + parse_labels_from_series_key roundtrip cleanly #285) — found by refactor(data_plane): rekey ingest bucketing from (agg_id, group_key) to sid (B7.6) #284 agent during B7.6controller_plan_to_query_full_roundtrip_{ddsketch,kll}pre-existing test failures (fix(tests): repair controller_plan_to_query_full_roundtrip post-#280 legacy retirement #289) —start_time_unix_nano: 0building OTLP fixturessum by (...)(feat(query): dispatch sum by (...) PromQL to ExactAgg(Sum) sids (closes Option-B downstream) #291) and rate-over-Sum compose (feat(query): compose rate-over-Sum into ExactAgg(Sum) candidate (multinode topk dispatch) #292) — the data_plane consumers of the post-B2 multi-role worldSister issue #271 (the original MVP demo bug) is now closed — both single-node and multinode validate end-to-end.
Leaving #272 open until step 5 (aggregation_id() accessor retirement) lands.
refactor(types): rename AggregationConfig::aggregation_id() to policy…
zzylol commentedon May 18, 2026
Closing — all steps done.
handle_queryfamily retirementpub fn aggregation_id()onAggregationConfigpolicy_fp_u64(); 55 sites swept across 15 files)Also along the way (out of this issue's original scope but related cleanup):
format_series_key/parse_labels_from_series_keyquote-roundtrip (fix(data_plane): make format_series_key + parse_labels_from_series_key roundtrip cleanly #285)