[zzylol]

Why

The joint search orders its proposals by the MILP's lower bound, the sum of each selected candidate's local_cost. Query-time scans, windows and builds had local_cost 0, so a plan that rescans 5 minutes of raw rows every evaluation looked free next to an ingestion path that has a positive bound. In example C (#963) the q1 query-time path has true cost 0.1560/s and bound 0, while the ingestion path costs 0.0524/s and has bound 0.0039. The search proposed every zero-bound combination first.

What

costing/lower_bounds.rs now certifies bounds for:

• query-time Scan, TimeRange, TimeShift, Aggregate, SummaryAgg and PaneChain (NotMaterialized or MatAtQuery; never under a chain kept across evaluations, as before);
• ingestion-time PaneChains of every sketch family, not only DDSketch (KLL, CMS, Count Sketch, KMV, HLL, UnivMon).

Cause of the 0: certified_price called price_node with no lookback. A query-time scan's rows are "the arriving rows of the longest window read above it", so with no window price_node fell back to rows_held or failed, and every node above the scan stayed uncertified.

How the bound stays sound:

• Scan: shortest_reader_windows gives each candidate the least window any candidate that may read it reads, counted as the evaluator's windows_read counts windows (a time range's range, a pane chain's panes, through time shifts). The evaluator reads the selected readers' longest window, or every held row when none has one. Both read at least rows(shortest window) capped by rows_held, the quantity priced here.
• Rate: the slowest query rate (query_rate_floor), as for the existing query-reader bounds.
• Nodes above: priced over that scan's rows. CPU and scanned bytes are monotone in rows, so they are lower bounds too.
• Resizable sketches: a summary build is priced at its template's parameters. Stage 3 only resizes a sketch to larger parameters (with_failure_probability takes the maximum with the current size for every family, HLL's min_precision starts at the current precision), and the analytical shape (sketch_shape) never charges a larger sketch less, so the template's price is at most its price in any plan. A measured operation-cost table interpolates between measured sizes and need not grow with size, so a resizable sketch family with any measured entry (any operator, either phase) is not certified (template_price_is_a_floor, measured_family). This also covers the estimates and merges above its builds, which a measured SummaryEstimate or SummaryMerge entry prices by interpolated size: they are certified only over a certified build. A query-time DDSketch build, like a DDSketch pane chain, is priced only from producers that agree on the raw non-negative domain, which decides whether it keeps one bucket store or two.
• Soundness check tolerance: the search's check that an evaluated plan is not below its certified bound now allows a relative rounding step (1e-12). The bounds can be exact (1.8e-8 relative margin measured), and the bound and the evaluator add the same per-node prices in different orders.
• Shifted windows (offset): Stage 3 scans the width of the union of the readers' windows (PhysicalASAPDAG::scan_window) and keeps raw rows up to offset + width at its own charge; the bound prices the shortest reader width and no retention, so it stays below. The bounds are not offset-aware otherwise.
• Cost models: bounds are computed only under UsageOverTime, as before.
• Check: the search already errors when an evaluated plan costs less than its certified bound (search.rs: "actual cost … is below certified local bound"). No suite run below triggered it.

Key code interfaces added / modified

• lower_bounds::shortest_reader_windows(&CandidateConfigurationInventory) -> Vec<Option<u64>> (new, private).
• certified_price takes the readers' shortest windows (private).
• lower_bounds::template_price_is_a_floor(&ASAPOp, &CandidatePlanningContext) -> bool (new, private): whether a summary build's template price bounds every size Stage 3 can give it.
• lower_bounds::measured_family(&SketchParams, &MeasuredOperationCosts) -> bool (new, private): whether the measured table has any entry for the sketch's algorithm.

Major algorithm changes

The query-time whitelist of lower_bounds::apply now includes scans, windows, shifts, aggregates and summary builds; the ingestion whitelist includes pane chains of every sketch family whose template price is a floor.

Before this PR / After this PR

Example C on main (SELECT city, p90 … GROUP BY city with SELECT p90 … WHERE city = 'NYC' OR city = 'SF', 5-minute window, every 10 s), seed round, instrumented:

	Before	After
q1 query-time scan bound	0	0.1160006/s (its evaluated cost)
Proposals of the 0.332/s query-time family after the baseline	4 priced, 8 rejected	0
0.2284651/s first found	evaluation 15	evaluation 11
Proven optimal	after 18 evaluations, 0.88 s	after 11 evaluations, 0.43 s

The remaining 8 proposals before the optimum are rejections (q0 reading a grouped KLL fails its accuracy target), which #964 targets.

Per-city p90 pair (city = 'NYC' and city = 'SF'): on the old main 0.2084510/s against 0.1048378/s separate. This PR alone gives 0.1048378/s. Since #973 merged, main gives it too.

Regression tests, branch against main (both on f06a5556, same machine, run alternately)

	Branch	Main
sharing_issue_regressions (incl. #923 exact_coarse_sum_reuses_fine_sum)	5 of 5 runs pass	5 of 5 runs pass
memo_covers_whole_workload_pass2::topk_reuse	passes (memo 0.7755078/s ≤ whole-workload 0.7756417/s), 2 of 2; un-ignored	ignored (search miss)
memo_covers_whole_workload_pass2::shared_sizes	still fails (memo 3.3738667/s vs 2.4145834/s); stays ignored	ignored
Example C (finer_aggregate_reuse)	0.2284651/s together	0.2284651/s

The earlier version of this PR certified only query-time nodes and made #923 fail 3 of 3: the structural baseline weighed the query-time scan at its certified price while the competing KLL pane chains stayed at 0. Certifying the pane chains at their template size restores the comparison.

Not done

• shared_sizes still misses the whole-workload plan (one query-time KLL k=345 read by both estimates).
• Search: certify lower bounds for query-time scans and builds #963's acceptance item "the seed round on main reaches 0.2284651 within at most 3 evaluations" is not met (11, see above). Hence "Refs", not "Closes".

Verification

Remote, clnode143, on the branch rebased on f06a5556 (#988):

• Failing first: query_time_scans_and_builds_have_certified_bounds fails on main's lower_bounds.rs (scan bound 0). It now also covers sum_over_time(latency[5m] offset 1m): positive bounds, at most the evaluated cost under scan_window pricing.
• New: a_kll_build_bound_holds_for_every_resized_size (the KLL bound stays below the evaluated cost at failure probabilities 0.01, 0.001 and 0.0001, which resize it larger), a_measured_kll_build_is_not_certified, a_measured_kll_estimate_keeps_the_bound_below_the_cost (an independent review's counterexample: measured SummaryEstimate entries Kll{k:8} at 5e9 ns and Kll{k:20000} at 1 ns gave bound 413.51 above the evaluated 407.63 before the fix).
• Soundness: the search's own check (an evaluated plan below its certified bound is an error) did not trip in any run.
• cargo test -p asap-plan-selection --lib: 377 passed, 1 failed: producer_reuse_rules_produce_validated_shared_plans, which fails the same way on main f06a5556.
• Integration suites: independent_plan_seed 3, joint_search_equivalence 14, production_rollup_workloads 4, identical_exact_queries 2, rollup_controlled 2, memo_covers_whole_workload_pass2 7 passed and 1 ignored, finer_aggregate_reuse 10, sharing_issue_regressions 12 (5 runs).
• cargo fmt --all --check; clippy -D warnings on asap-plan-selection and the integration tests: clean.
• The example C seed-round trace comes from an instrumented copy of main before fix(plan-selection): evaluate the queries' separate plans first in the seed round #973 (d4152bce); the instrumentation is not committed.

Refs #963

🤖 Generated with Claude Code