Skip to content

Query engine: roll-ups over keyed aggregations (aggregated_labels) #846

Open

Description

@milindsrivastava1997

Follow-up to #839, which covers roll-ups only for single-subpopulation configs (KLL, DDSketch, HLL, Sum, MinMax: grouping_labels = G, one sketch per store key).

Goal

Let the engine serve a query grouped by Q from a keyed aggregation whose aggregated_labels ⊋ Q. Keyed configs use grouping = [] and aggregated = G, with a DeltaSet key aggregation or a self-keyed value: MultipleSum, MultipleMinMax, MultipleIncrease, CountMinSketch, CountMinSketchWithHeap and HydraKLL.

Why it's a different mechanism

The subpopulations live inside one sketch, so a roll-up can't merge store keys. Instead it has to enumerate the G-keys (from the DeltaSet, or the accumulator's get_keys), project them onto Q, and combine the per-key answers:

  • MultipleSum / MultipleMinMax: an exact sum or min/max of the per-key values. The MILP already gets this through query_time_aggregations as of Query engine: merge sketches across subpopulations (roll-up to a coarser grouping) #839, so the only open question is whether the engine should do it natively.
  • CMS: sum the point estimates. The error grows with the fan-in, so it has to be priced or bounded.
  • MultipleIncrease, CMS+heap: not mergeable across groups. The counter-reset merge only works across time, and the top-k heaps drop items.
  • HydraKLL: the engine wraps a single-key Hydra. sketchlib's full Hydra framework expands each record into its subpopulation subsets, which would answer subsets natively.

Open

Is this worth doing natively in the engine, beyond what query_time_aggregations already covers for the exact types?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions