[zzylol]

Why

After #792, the default search still composes every valid whole-workload choice. A 9-query PromQL workload from ASAPQuery's random mixed set (seed-862 workload-01) takes 128 s, because its 7 queries over data and its 2 over data_total multiply each other's alternatives, even though no rewrite can link them.

This PR implements method 5 of docs/design_docs/proposals/joint-planning-search.md, "Split independent components".

Part of #673.

What

• Components. Queries are grouped by the sources they read: two queries are in one component when they read a common source, directly or through other queries. The source is keyed by name, because data facts are keyed by name.

• Search per component. Under cost model 1, each component is searched on its own with Joint planning search: stream Stage 1 to Stage 3 and reject permanently invalid query candidates (methods 2 + incumbent) #792's search. The selections are then combined into one whole-workload candidate, which Stage 3 checks and prices again.

• Exactness preconditions. The combination is used only when all of these hold:

  1. Additive objective, no global coupling. Cost model 1, with no measured-table calibration. Under cost model 2 (peak cores are priced as a ceiling over the whole workload) and cost model 3 (peak cores and cluster limits), the workload is searched as one, and so is a workload with a measured cost table (one calibration factor over the whole workload). Resource frontiers for those cases are follow-up work.
  2. In the joint space. The combination must be one of the joint search's Stage 1 candidates. The Pass 2 rules do not combine every rewrite: rollup and top-k reuse are not combined with summary sharing or panes. The joint search also caps candidates per workload. To check this, the joint Stage 1 candidates of the combined Pass 1 choice are enumerated, and the one with the same flattened roots is the one implemented.
  3. No tolerance-sensitive tie. Each component's runner-up must cost more than its selection by more than 4e-9 of the workload's price, which is above Stage 3's 1e-9 tie tolerance. Candidates within 1e-12 (rounding) of the cheapest count as the same price. Among those, the component's search takes the first in its enumeration order. The joint enumeration order (mixed radix over targets, then Pass 2 order, then materializations) is monotone in each component's order, so that candidate is also the joint search's first.
  4. The sum checks out. The combined candidate's Stage 3 price must equal the sum of the components' prices within 1e-9.

  Otherwise the workload is searched as one, and the report says why.

• Selection report. SearchReport gains:

  • components: queries, candidates checked and selected id for each component;
  • searched_as_one: the reason the workload was searched as one;
  • runner_up_cost_per_second: the selection's margin.

  The combined candidate's id joins the components' ids, e.g. P505-m80+P4-m8.

How

• Union-find. components::source_components runs union-find over the Scan sources each root reaches.
• Per-component search. components::search_components runs run_stage1_to_stage3_from_pass1 on each component's own Pass 1 inventory. It maps each component's selected Pass 1 choice onto the joint inventory's targets (by aggregate node and measure), then finds the matching joint Stage 1 candidate with stage1_candidates_of_choice.
• Rebuilding the materialization. optimize_physical_asap_assigned rebuilds the materialization one summary build at a time. Each build takes the option its component chose for it, in discovery order. Components are disjoint, so their relative discovery order is the same in the joint DAG.
• Margin tracking. PlanSelector tracks the cheapest valid price and the cheapest price more than rounding above it. That runner-up is what the tie check reads.
• Logical candidate in the run. Stage1ToStage3Run gains stage1_selected, the logical candidate the selection implements, so a component run hands its choice and roots to the combination.

Key code interfaces added / modified

crates/plan-selection/src/lib.rs

/// What the search behind a selection checked and discarded. The type stays.
pub struct SearchReport {
    /// The physical candidates Stage 3 checked, valid or not.
    pub candidates_checked: u64,                                     // unchanged
    /// Per query, its own candidates checked and discarded before composition.
    pub queries: Vec<QuerySearchReport>,                             // unchanged
    /// The components searched separately and combined; empty when searched as one.
    pub components: Vec<ComponentReport>,                            // new
    /// Why the workload was searched as one; None when its components were combined.
    pub searched_as_one: Option<SearchedAsOne>,                      // new
    /// Cost models 1 and 2: the cheapest valid price more than rounding (1e-12) above the cheapest; None when all cost the same, under cost model 3, or for a combination.
    pub runner_up_cost_per_second: Option<f64>,                      // new
}

/// One independent component: queries reading no source another component reads.
pub struct ComponentReport {                                         // new
    /// Its queries, by index in the workload.
    pub queries: Vec<usize>,                                         // new
    /// The physical candidates Stage 3 checked for it.
    pub candidates_checked: u64,                                     // new
    /// Its selected candidate's id in its own search.
    pub selected: String,                                            // new
}

/// Why a workload was searched as one; reflects the workload or settings, not an error.
pub enum SearchedAsOne {                                             // new
    /// Its queries all read sources that link them.
    OneComponent,                                                    // new
    /// Cost model 2 or 3: peak cores and cluster limits are not sums over components.
    CostNotAdditive,                                                 // new
    /// The caller asked for the exhaustive search or every candidate.
    EveryCandidateRequested,                                         // new
    /// A measured cost table calibrates the formulas over the whole workload.
    CalibratedOverTheWorkload,                                       // new
    /// A component's two cheapest plans are within tolerance of a tie at the workload's price.
    NearTie,                                                         // new
    /// The combined plan's price is not the sum of its components'.
    NotAdditive,                                                     // new
    /// The combination is not a candidate the joint search enumerates.
    NotInJointSpace,                                                 // new
}

crates/plan-selection/src/pass/stage1_to_stage3.rs

/// The output of Stage 1 to Stage 3 for one workload. The type stays; it gains a field.
pub struct Stage1ToStage3Run<Id> {
    pub stage1_pass1_per_computation: PerComputationCandidates<Id>, // unchanged
    pub combinations: usize,                                         // unchanged
    pub stage1_logical_asap_optimization: Vec<LogicalCandidate>,     // unchanged
    pub stage2_physical_asap_optimization: Vec<PhysicalCandidate>,   // unchanged
    pub stage2_pruned: Vec<PrunedCandidate>,                         // unchanged
    pub stage3_plan_selection: Result<Selection, SelectionError>,    // unchanged
    pub stage3_selected: Option<PhysicalCandidate>,                  // unchanged
    pub stage3_selected_logical: Option<Vec<QueryRoot>>,             // unchanged
    /// Stage 1: the logical candidate the selection implements, with its roots as composed (before Stage 3 sized them).
    pub stage1_selected: Option<LogicalCandidate>,                   // new
}

crates/physical-asap-optimization/src/lib.rs

/// One CandidatePhysicalASAPDAG. The type stays; it gains a field.
pub struct PhysicalCandidate {
    pub id: String,                                                  // unchanged
    pub from_logical: String,                                        // unchanged
    pub roots: Vec<QueryRoot>,                                       // unchanged
    pub dag: PhysicalASAPDAG,                                        // unchanged
    pub materialization: BTreeMap<PhysicalASAPNodeId, Materialization>, // unchanged
    /// The option chosen for each summary build, in discovery order: what optimize_physical_asap_assigned rebuilds it from.
    pub assignment: Vec<Materialization>,                            // new
}

/// Stage 2 for one logical candidate and one materialization assignment: `choose(readers)` picks each build's option in discovery order. Ok(None) when an option is not the build's or the assignment is not down-closed (bad input from the caller).
pub fn optimize_physical_asap_assigned(from_logical: &str, roots: &[QueryRoot], entries: &[QueryWorkloadEntry], data: &DataWorkload, deployment: &DeploymentCapabilities, choose: &mut dyn FnMut(&[usize]) -> Option<Materialization>) -> Result<Option<PhysicalCandidate>, PhysicalASAPOptimizationError>; // new

crates/logical-asap-optimization/src/workload_candidates.rs: unchanged in this PR. stage1_candidates_of_choice from #792 is the membership check.

Usage

The user plans the 9 PromQL queries of seed-862 workload-01 with plan_installable under cost model 1 (Fargate prices):

// The user's workload: q01..q07 read `data` (sum, quantile, topk), q08 and
// q09 read `data_total` (topk over increase, topk over rate); 5% error target.
let workload: PlanningWorkload = /* nine RepeatingEntry, max_error 0.05 */;
// Stage1ToStage3Pass -> run_stage1_to_stage3 -> source_components gives
// [[0..=6], [7, 8]]; each is searched with #792's search; the selections
// P505-m80 (8,208 candidates checked) and P4-m8 (16) are combined, found
// among the joint search's Stage 1 candidates, rebuilt with their
// materializations, and priced by Stage 3 again.
let plan = plan_installable(UserInput::new(&workload, FrontendInput::Promql { histograms: None }, models)).await?;
let report = plan.selection.unwrap();
assert_eq!(report.selected_candidate_id, "P505-m80+P4-m8");
assert_eq!(report.cost_per_second, Some(7.527148973830562e-6)); // the joint search's price
assert_eq!(report.search.searched_as_one, None);
let components: Vec<_> = report.search.components.iter().map(|c| (c.queries.clone(), c.candidates_checked)).collect();
assert_eq!(components, [(vec![0, 1, 2, 3, 4, 5, 6], 8208), (vec![7, 8], 16)]);

(The numbers come from running the local converter against this commit. The converter isn't part of this PR.)

Before this PR

With #792, the 9-query PromQL prefix of seed-862 workload-01 takes 128 s at 217 MB, and checks 131,088 candidates.

After this PR

Release build of the same converter, plan_installable end to end, PromQL queries only:

Workload	Before (#792)	After
seed-862 workload-01, 9 queries	128 s / 217 MB	12.9 s / 83 MB

The 9-query plan is byte-identical, as installable plan JSON apart from the selection report, to the joint search's plan from #792. The 3-, 5- and 6-query prefixes are single-component and unchanged.

All 30 ASAPQuery workloads (seed-862, seed-862-x5, seed-862-long), PromQL only: see the comment below once measured.

The 7-query data component dominates: 8,208 candidates in about 10 s. Later PRs (methods 1, 6, 7, then 3 and 4) target it.

Verification

• cargo test --workspace --locked: passes.
• joint_search_equivalence.rs. Each check compares the default search with Stage1ToStage3Search::exhaustive(). Existing tests keep passing; the random query generator now takes the gauges to draw from. New tests:
  • random_workloads_over_disjoint_sources_combine_their_components: 40 seeded random workloads over two gauges and a counter under cost model 1. They select the exhaustive plan, and at least 10 are actually combined (components non-empty).
  • components_follow_shared_sources: a query over m and n links the m and n queries into one component, and the c_total query is its own.
• Measurements. The table above uses /usr/bin/time around the converter. Plans were compared as JSON with the Joint planning search: stream Stage 1 to Stage 3 and reject permanently invalid query candidates (methods 2 + incumbent) #792 joint search.
• Not run: clippy, fmt check, GitHub CI.

Limitations and follow-up

• No speedup under cost models 2 and 3, or with a measured cost table. Those workloads are searched as one. Combining per-component resource frontiers against cluster limits, and component-aware calibration, are follow-up work.
• Calibration cost, from Joint planning search: stream Stage 1 to Stage 3 and reject permanently invalid query candidates (methods 2 + incumbent) #792. With a measured table, the calibration pass runs Stage 2 on every unpruned combination before the search, which gives up most of Joint planning search: stream Stage 1 to Stage 3 and reject permanently invalid query candidates (methods 2 + incumbent) #792's speedup in that case. A smaller fit set is a possible follow-up, if it can be shown to give the same factor.
• calibration_fit errors. It can fail, with a Stage 3 cost error, on a candidate that a pruned search would never offer. This matches the exhaustive search, which prices that candidate too.
• Monotone order. The tie argument relies on the joint enumeration order being monotone in each component's order. It holds for the mixed-radix Pass 1 order, the Pass 2 binary-counter and block order, and the materialization order. It is exercised by the equivalence tests, not proven in code.