[Feature] Different levels of tests for showing ASAPQuery-backend's benefits #754

User need and MVP

Different levels of tests with targeted queries:

1. the queries can be compiled by ASAPQuery-backend and ASAPPlanner, the outputs are the correct physical plans
2. The queries/physical plans can be executed by the ASAPQuery-backend data plane, and all the mapped physical operators in ASAP are implemented.
3. The execution of such queries and physical plans by ASAPQuery-backend data plane shows lower resource usage and lower query latency compared to exact DB, i.e., Prometheus, VictoriaMetrics, ClickHouse.

Authoritative design

Adding e2e tests for that 3 levels.

Inputs, outputs, and invariants

Evaluation queries: PromQL and ClickHouse SQL

Use a 100 ms scrape interval. Vary label_0 cardinality across 10, 100, 1,000, 10,000, 100,000, and 1,000,000, and vary T across 1m, 10m, 1h, 6h, and 24h. Evaluate instant queries every 1 second and temporal queries every 1 minute. For quantiles, use q = 0.50, 0.75, 0.90, 0.95, and 0.99. Exercise these queries at all three test levels above.

Use a ClickHouse samples table with series_id UInt64, label_0 String, ts_ms Int64 (Unix milliseconds), and value Float64; retain the complete label set in a series dictionary keyed by series_id. Bind {t_ms:Int64} to each evaluation timestamp, {window_ms:Int64} to T in milliseconds, and {q:Float64} to the requested quantile. Each SQL statement starts with the relevant CTEs below. The examples assume unique timestamps per series, finite float samples, no stale markers, and no native histograms. Those additional PromQL semantics need dedicated fixtures and ClickHouse handling before claiming full parity. Keep the benchmark input and evaluation timestamps identical across engines.

Shared CTEs for instant and range-vector input (use the relevant one before each query):

WITH
  {t_ms:Int64} AS t_ms,
  {window_ms:Int64} AS window_ms,
  instant_samples AS (
    SELECT series_id, label_0, argMax(value, ts_ms) AS value
    FROM samples
    WHERE ts_ms <= t_ms AND ts_ms >= t_ms - 300000
    GROUP BY series_id, label_0
  ),
  window_samples AS (
    SELECT series_id, label_0, ts_ms, value
    FROM samples
    WHERE ts_ms > t_ms - window_ms AND ts_ms <= t_ms
  )
SELECT ...;

The 300000 ms instant lookback must match the configured Prometheus lookback. If the fixture includes stale markers, argMax alone is insufficient: inspect the latest sample's stale flag before admitting a series. Apply label predicates within these CTEs. Preserve the series dictionary when comparing output labels.

For queries 6, 7, and 9, append these CTEs after window_samples and use the appropriate final SELECT. They implement Prometheus 3.5.0's float-counter reset correction and extrapolation in ClickHouse syntax; increase_value = rate_value * window_ms / 1000 except for normal floating-point rounding.

, ordered AS (
    SELECT series_id, label_0, ts_ms, value,
           row_number() OVER (PARTITION BY series_id ORDER BY ts_ms) AS sample_index,
           lag(value, 1, 0.) OVER (PARTITION BY series_id ORDER BY ts_ms) AS previous_value
    FROM window_samples
  ), corrected AS (
    SELECT series_id, label_0, count() AS sample_count,
           min(ts_ms) AS first_ms, max(ts_ms) AS last_ms,
           argMin(value, ts_ms) AS first_value,
           argMax(value, ts_ms) AS last_value,
           sum(if(sample_index > 1 AND value < previous_value,
                  previous_value, 0.)) AS reset_correction
    FROM ordered
    GROUP BY series_id, label_0
    HAVING sample_count >= 2
  ), durations AS (
    SELECT *,
           last_value - first_value + reset_correction AS corrected_delta,
           (last_ms - first_ms) / 1000. AS sampled_seconds,
           (first_ms - (t_ms - window_ms)) / 1000. AS gap_start_seconds,
           (t_ms - last_ms) / 1000. AS gap_end_seconds
    FROM corrected
  ), thresholds AS (
    SELECT *, sampled_seconds / (sample_count - 1) AS average_gap_seconds
    FROM durations
  ), adjusted AS (
    SELECT *,
           if(gap_start_seconds >= 1.1 * average_gap_seconds,
              average_gap_seconds / 2, gap_start_seconds) AS start_seconds,
           if(gap_end_seconds >= 1.1 * average_gap_seconds,
              average_gap_seconds / 2, gap_end_seconds) AS end_seconds
    FROM thresholds
  ), extrapolated AS (
    SELECT *,
           if(corrected_delta > 0 AND first_value >= 0,
              least(start_seconds, sampled_seconds * first_value / corrected_delta),
              start_seconds) AS zero_adjusted_start_seconds
    FROM adjusted
  ), per_series_counter AS (
    SELECT series_id, label_0,
           corrected_delta *
             (sampled_seconds + zero_adjusted_start_seconds + end_seconds)
             / sampled_seconds AS increase_value,
           corrected_delta *
             (sampled_seconds + zero_adjusted_start_seconds + end_seconds)
             / sampled_seconds / (window_ms / 1000.) AS rate_value
    FROM extrapolated
  )
SELECT ...;

Replace SELECT ... with the matching final query below. In #2 and #9, ClickHouse ORDER BY label_0, value DESC ... LIMIT 3 BY label_0 ranks rows and keeps up to three original series in each label_0 group. This is the ClickHouse equivalent of PromQL topk by (label_0), which retains the selected series labels. GROUP BY label_0 would collapse the input to one aggregate row per group and would therefore be incorrect here. Ties require a declared deterministic comparator policy.

For supported count_over_time, use SELECT series_id, count() AS value FROM window_samples GROUP BY series_id; for increase, use SELECT series_id, increase_value AS value FROM per_series_counter. Do not use ClickHouse's approximate quantile or approximate topK as the exact SQL baseline. Compare nonfinite values and missing series separately; the finite-only SQL above does not establish those cases.

Supported query forms and compositions

Test each form with and without label filters. Apply equivalent predicates inside the ClickHouse input CTEs. Keep the full series_id grouping for temporal operations and retain original series labels for topk.

Also test multiple spatial aggregations, a temporal aggregation followed by spatial aggregations, and binary operations between aggregation expressions. For ClickHouse SQL, evaluate both operands at the same timestamp, preserve the PromQL grouping and label-matching rules, then join on the correct label set. Include cases for missing series, ties, and nonfinite values before claiming full composition support.

ClickHouse references: exact inclusive quantile, LIMIT BY, window functions, and argMax. The counter formula follows Prometheus 3.5.0 extrapolatedRate.

Before and after behavior

After should have these tests in MVP CI of the ASAPQuery-backend repo.

Unit and end-to-end acceptance tests

Explicit non-goals