Skip to content

DeltaSetAggregator must be restricted to tumbling windows #588

Closed

Description

@milindsrivastava1997

DeltaSetAggregator only tracks added/removed keys since the last window, so it's only correct for non-overlapping (tumbling) windows. Nothing currently enforces this: the planner (agg_config.rs) copies whatever window_type the surrounding query config has when emitting a DeltaSetAggregator config, and the precompute engine's DeltaSetAggregatorAccumulator/pane-merge path is window-type-agnostic. A sliding-window DeltaSetAggregator config can be planned and executed today, silently producing incorrect merged add/remove sets across overlapping panes.

Consequence for the query engine: within a single query, key-cardinality accumulators may need different merge strategies per window type — DeltaSetAggregator panes must be read/merged as tumbling windows, while other accumulators (e.g. SetAggregator, CountMinSketch, HydraKLL) tied to the same query may legitimately use sliding windows and need the sliding merge path.

Needs:

  • Planner-side validation/gate rejecting or refusing to select DeltaSetAggregator when window_type is Sliding.
  • Query-engine read path to handle mixed window types within one query (tumbling merge for DeltaSetAggregator, sliding merge for co-located accumulators).

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions