MVP demo: agent emits full-attr sketches → no sid match for grouped streaming-config

Found during the single-node MVP smoke test (/mydata/mvp-smoke-test/REPORT.md on the dev box). Sister to ASAPCollector#381 (agent OpAMP-push doesn't reload) — this is what makes Axis C (end-to-end PromQL) return empty even though Axis D shows 8800 sids registered.

Correction to an earlier version of this issue: the original write-up framed this as an "agg_id hash mismatch" between two xxh64 schemes. That's wrong — compute_sketch_sid was retired in PR #190/#192 and sid is now a counter (per docs/design-sid-lifecycle.md). The actual root cause is sid identity mismatch: the agent emits sketches with all wire attrs, so the backend mints sids over the full attr set, but the streaming-config (and the still-extant aggregation_id query path) expects sids grouped down to grouping_labels only.

What's stored

SeriesIdResolver mints sid = fetch_add(1) keyed by (metric_name, attrs_fingerprint, agg_kind_canonical). The fake-exporter emits http_requests_total_latency_ms with {zone, rack, node, pod, endpoint, service.name, telemetry.sdk.name, telemetry.sdk.language, telemetry.sdk.version} (9 keys); the agent's DDSketch processor doesn't drop or roll up any of them. So the backend mints 51 sids for http_requests_total_latency_ms — one per unique full-attr tuple — each carrying group_by_keys = [endpoint, node, pod, rack, service.name, telemetry.sdk.*, zone].

What the query path looks for

asap_query_engine/engine.rs:1743 logs:

Query execution failed: No precomputed outputs found for metric: http_requests_total_latency_ms,
                        aggregation_id: 16882953407686652000

16882953407686652000 is PolicyFingerprint::as_u64() of the streaming-config entry — xxh64(metric + agg_type + sub_type + params + grouping_labels=[zone] + aggregated_labels=[]). The store has no rows keyed by that agg_id because nothing on the ingest path computed a PolicyFingerprint with grouping_labels=[zone] (the agent never group-reduced before sketching).

Reproduction

bash /mydata/mvp-smoke-test/run_smoke.sh

After 90 s:

curl http://localhost:19091/api/v1/streaming-config → aggregation_id: 16882953407686652000 registered
curl http://localhost:19091/api/v1/db/schemas → 51 sids for the metric, each with full-attr group_by_keys
curl 'http://localhost:19091/api/v1/db/timeline?metric=http_requests_total_latency_ms&start_ms=...&end_ms=...' → 1 segment under agg_id: 6206425945726099596 — note this is the timeline endpoint's content-derived signature id (xxh64 of metric + agg_kind + group_by_keys per the comment in handle_get_timeline), not the PolicyFingerprint the query path looks up. Two different hash schemes, both surfaced as agg_id in the API output — confusing but real.
curl --data-urlencode 'query=quantile_over_time(0.99, http_requests_total_latency_ms[30s])' http://localhost:19091/api/v1/query → {"error":"No result for query"}

Fix paths

This sits at the intersection of two in-progress migrations:

Schema-retirement feat: vendor DataCollector's modified OTLP proto (PR A, Phase 1) #5 in engine.rs:2392 — finish removing the aggregation_id lookup from the query plan and route queries directly through the sid catalog ((metric, agg_kind, attrs_subset) → sids that subsume the asked grouping). When this lands the agg_id path goes away entirely, and the question becomes "find sids whose group_by_keys are a superset of the query's grouping, then merge by the asked subset."
Agent-side group-reduction — when ASAPCollector#381 lands and the controller's typed-stage-split YAML actually reaches the agent, the per-metric pipeline needs an attributes/keep (or equivalent) processor upstream of each sketch processor that whitelists grouping_labels. Then the agent emits sketches with attrs already reduced to [zone], the resolver mints the desired N-zone sids, and any query that asks for the streaming-config's grouping finds them directly.

Either fix unblocks Axis C. Both might land independently — (1) is purely backend, (2) needs ASAPCollector#381 first.

Tagged bug, mvp-demo, query-engine, schema-retirement-5.

Schema-retirement #5 (one of the two fix paths called out in this issue) is now tracked in #272.

fix(query): union instances_matching into ASAP-tier sid resolution (s…

Fix path 2 (backend-side: union instances_matching into the modern execute() sid resolution) landed in #273. End-to-end smoke test Axis C now returns a real DDSketch quantile:

$ curl --data-urlencode 'query=quantile_over_time(0.99, http_requests_total_latency_ms[5m])' \
       http://localhost:19091/api/v1/query
{"accuracy":{"delta":0.0,"epsilon":0.01,"kind":"relative_quantile"},
 "data":{"result":[{"metric":{"zone":""},
                    "value":[1779043268.182,"90.93548893834691"]}],
         "resultType":"vector"},
 "infos":["accuracy: ε=0.01, δ=0, kind=relative_quantile",
          "precompute_window: [1779042780000, 1779042810000) ms (width 30000 ms)",
          "data_source: asap_query"]}

NB: smoke test must use [5m] (or any range \≥ the agent's 60s window_duration); [30s] returns NoData because the range can land entirely inside an open sketch window. Separately, the sum by (zone) (http_requests_total) query still fails — the agent emits DDSketch but the query needs Sum capability, which is fix path 1 territory (controller pushes appropriate sketch type or agent emits ExactAgg precompute). Closing this issue would be premature until fix path 1 is also resolved or scope-narrowed.

The remaining agg_id retirement work (legacy handle_query path, ingest bucketing, precompute worker, backfill) is tracked in #272.

fix(query): port resolve_sketch_metric_alias to modern execute() (sch…

revert+retire(query): delete dead resolve_sketch_metric_alias rewrite (…

Status update 2026-05-17 — fix path 2 (controller-side) now lands in stages:

B3 (agent emits sketches grouped by [zone])

ASAPQuery-backend feat(emit): inject attributes/keep upstream of each sketch processor (B3) #278 — controller injects attributes/keep upstream of each sketch processor (only when grouping_labels is set)
ASAPQuery-backend fix(emit): B3-pop + B4-window bundle (B2-Sum deferred) #282 — populates metric_to_grouping_labels from WorkloadEntry so the routing-table builder has data to thread
ASAPCollector docs(control_plane): phase-by-phase migration plan for BackendPlan cutover #390 — adds grouping_labels: [zone] to all 8 entries in deploy/mvp-singlenode/configs/mvp-workload.yaml

End-to-end verification post-#390:

$ curl --data-urlencode 'query=quantile_over_time(0.99, http_requests_total_latency_ms[30s])' \
       http://localhost:19091/api/v1/query
# returns real DDSketch quantile, ε=0.01, grouped by zone

Still failing — sum by (zone) (http_requests_total): DDSketch can't answer Sum capability. Tracked as B2 (Sum-as-separate-aggregation). The fix in flight restructures WorkloadStore from HashMap<metric, QueryWorkload> to HashMap<(metric, AggRole), QueryWorkload> so the same metric can carry both a quantile AggregationConfig (DDSketch) and a sum AggregationConfig (ExactAgg) simultaneously.

Closing this issue when B2 lands and Sum-capability returns end-to-end.

Closing as fixed — the wave is end-to-end green on both single-node smoke harness and 4-node multinode deploy.

Path 1 (controller-side, originally tracked here):

B2 (Sum as separate aggregation): feat(planner): (metric, role)-keyed WorkloadStore + PlanStore (B2 full) #283 + fix(emit): cumulative streaming-config across (metric, role) pairs (B2 follow-up) #287 + refactor(control_plane): unify Replanner through typed cumulative emit; retire generate_streaming_config_yaml (Option B) #290 (Option B unified emit)
B3 (group-reduction upstream of sketches): feat(emit): inject attributes/keep upstream of each sketch processor (B3) #278 + fix(emit): B3-pop + B4-window bundle (B2-Sum deferred) #282 + ASAPCollector docs(control_plane): phase-by-phase migration plan for BackendPlan cutover #390
B4 (window=30s from query): fix(emit): B3-pop + B4-window bundle (B2-Sum deferred) #282
B7.5/B7.6/B7.7 (sid-keyed everything): refactor(query): delete legacy handle_query family (B7.5 v2 redo) #280 / refactor(data_plane): rekey ingest bucketing from (agg_id, group_key) to sid (B7.6) #284 / refactor(data_plane): rekey backfill processor + SketchStore from agg_id to sid (B7.7) #286

Path 2 (backend-side, the original "fix path 2"): #273 — landed early in the wave

Last-mile data_plane work:

feat(query): dispatch sum by (...) PromQL to ExactAgg(Sum) sids (closes Option-B downstream) #291 — dispatch sum by (...) to ExactAgg(Sum) sids
feat(query): compose rate-over-Sum into ExactAgg(Sum) candidate (multinode topk dispatch) #292 — compose rate-over-Sum into ExactAgg(Sum) candidate (enables topk(K, sum by (...) (rate(...))))

Multinode harness gap (separate but related): ASAPCollector #394 — added controller container to mvp-multinode runner + fixed replay typo + extended queries-e2e.json with wave queries + cleaned stale bucket: yaml key.

Smoke verification (singlenode):

$ curl --data-urlencode 'query=sum by (zone) (http_requests_total)' \
       http://localhost:19091/api/v1/query
# 4 zones, data_source: asap_query

$ curl --data-urlencode 'query=topk(5, sum by (zone) (rate(http_requests_total[5m])))' \
       http://localhost:19091/api/v1/query
# top-K, data_source: asap_query

Multinode verification (4-node, against node2:9091):

sum by (zone) (http_requests_total) → z0..z3 ~981M each
sum by (zone) (rate(http_requests_total[5m])) → z0..z3 ~4.8M
topk(5, sum by (zone) (rate(...))) → ranked 4 zones
All three: data_source: asap_query
Replay JSONL: 384 success lines (vs 0 pre-wave)

Navigation Menu

Repository navigation

MVP demo: agent emits full-attr sketches → no sid match for grouped streaming-config #271

What's stored

What the query path looks for

Reproduction

Fix paths

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

MVP demo: agent emits full-attr sketches → no sid match for grouped streaming-config #271

Description

What's stored

What the query path looks for

Reproduction

Fix paths

Activity

zzylol commented on May 17, 2026

zzylol commented on May 17, 2026

zzylol commented on May 18, 2026

zzylol commented on May 18, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions