Repository navigation
docs: clean up L3 QueryExpr; add schema flow + L4 SketchExpr + system I/O - #1
Merged
Merged
Conversation
… I/O
Address review feedback on the QueryExpr/AggIntent design:
- Drop redundant L3 nodes (TopK = Sort+Limit; WindowedAgg = Window+Aggregate)
- Move sketch-bound nodes (SketchAgg, JoinSketch) out of L3 into a new L4
IR (SketchExpr) with SketchSubtract/Delete/Estimate/Merge added
- Drop language-shaped nodes from L3 (HistogramQuantile, PromQLSubquery
lower in PromQL L1->L2 instead)
- Rename Dedup -> Distinct{cols} (SQL DISTINCT, generalised to N cols)
- Drop QuantileOverTime/TopK from AggIntent; surrounding Window covers it
- Add Schema/Field/HasSchema with per-node input/output spec table
- Distinguish three schemas: DAG schema vs DB/source schema vs sketch
catalog metadata
- Expand SketchCatalog with mergeability, accuracy, supported intents,
aggregated-key shape, parameter ranges
- New "System I/O contract" subsection: input is QueryWorkload, output
is per-executor sub-DAG assignments under the premise that every
executor has full capability and the controller chooses placement
- Refresh resolved-questions list in Section 12
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- TopK is a first-class AggIntent at L3 (heavy-hitter sketches like SpaceSaving / CMS-with-heap / Misra-Gries serve it as a single primitive). Generic QueryExpr::Sort + QueryExpr::Limit stays for non-heavy-hitter cases (ORDER BY name LIMIT 10). L1→L2→L3 lowering picks the intent form when it recognises a heavy-hitter pattern. - Clarify CTE / let-binding "reuse edges" wording: one producer + N Ref(name) consumer edges, which is what makes the IR a DAG. - Per-executor split: sub-tree -> sub-tree / sub-DAG. - Cut-edge transports: drop Kafka mention, replace with "sketch-merge / compute-from-raw over the precompute engine". - serialise -> serialize. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reframe "DAG, not tree" around the operational distinction: a producer node can have multiple parents that share its precomputed intermediate state. List the three fan-in sources: explicit CTE / recording-rule references, L4 reuse rules introducing shared producers across queries, and L4 stage / shard structure. The cost-model implication (credit build cost once across consumers) is what makes the DAG shape a Pareto win over a tree. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Drop the S / L / R / S_i shorthand; spell out each input and output in plain language so every row reads independently. Also split Sort and Limit into separate rows now that each gets its own constraint description (Sort requires keys' fields be present in input; Limit has no field-level constraint). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a self-contained table covering the seven SketchExpr variants (Logical pass-through, SketchAgg, SketchJoin, SketchSubtract, SketchDelete, SketchEstimate, SketchMerge) with input and output schemas, and introduces DataType::Sketch(SketchKind, SketchParams) as the L4 field type that carries sketch state. Two type-system invariants spelled out: sketch-family mismatch is a plan-time error, and catalog capability flags (subtractable, deletable, mergeable) gate which nodes can fire. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…glossary - Rename scenario→deployment model across all three docs. Crate names get an asap- prefix (deployment-model-asaplifecycle / -asapquery / -asapfusion). "Scenario" understated the bundle; "deployment model" names the choice — rule selection + L5 topology + emitter all reflect deployment constraints. - Distinguish stage from executor as first-class concepts. Stage (StageId) is a categorical lifecycle tier; Executor is a concrete runtime instance occupying a stage with its own id / address / capabilities. StageAllocator stays stage-granularity; PhysicalPlanner fans out per executor via DeploymentConstraints::executors(). - Rename §6 "Three distinct schemas" → "DAG schema, DB schema, sketch catalog — three distinct metadata sources" and clarify the sketch catalog is a registry of available primitives, not a schema. - Add §14 glossary covering architecture roles, layer drivers, IR by layer, metadata sources, and workload identity. Renumber Success criteria to §15. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Naming was inverted: L3 was called "sketch algebra" but contains no sketches (intent-only); L4 is where sketches actually get bound. Split the module accordingly so the names match the contents: - L3: `core::intent_algebra` — QueryExpr + AggIntent (intent only, no sketch type, no params). - L4: `core::sketch_algebra` — SketchExpr (sketch-bound: kind + params committed). Promoted from a #### subsection of the old L3 section to a sibling ### section. Touches the §3 layer table, §6 module sections, the directory tree, the §14 glossary L3/L4 entries, and ~10 cross-references across README.md, design.md, migration-plan.md. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…lanning The previous doc comment described unique_keys as a "sketch-reuse driver" without flagging that it's optional infrastructure for one specific feature (`CostModel::workload_cost` cross-query reuse credit). Tighten the comment to call out: who populates it (per-node spec), who reads it (workload_cost only), and what doesn't read it (Bind* rules, push-down, L5 emitters, single-query plans). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Worked example for `quantile_over_time(0.99, http_request_duration_seconds{service="api"}[5m])`
showing the L1→L5 transformation under each of the three deployment models, so readers can
see how the layered IR + topology-as-parameter design plays out concretely.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
May 27, 2026
…iguate joins (L2 review #1) The issue-#7 qualifier fix reached scalar predicates but not the grouping channels: `Aggregate.keys`, `TopK.by`, and `WindowFunc.partition_by` were plain `Vec<String>`, and `expr_to_group_name` discarded the relation qualifier, so `resolve_named_keys` resolved by first-match `column_id`. On a self-join, `GROUP BY a.k` and `GROUP BY b.k` both bound to column 0 → `b.k` silently grouped by `a.k` (wrong result). Same bug class #7 closed for predicates, still open for keys. Fix: the three group-key channels now carry `ColumnRef` (qualified-capable) like the scalar path, and resolve via `resolve_column_refs` → `column_id_qualified` with the bare-name fallback. `expr_to_group_ref` (SQL) preserves `col.relation`; PromQL emits `ColumnRef::Named` (labels have no qualifier). The Binder seeds the bare names; `resolve_named_keys` is removed (subsumed by `resolve_column_refs`). Confirmed by a self-join test: `GROUP BY b.service` → `Aggregate.by = [5]`, `GROUP BY a.service` → `[1]`. Full suite green (133), clippy + fmt clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
3 tasks done
zzylol
added a commit
that referenced
this pull request
Aug 24, 2026
Restructure the flat 19-section developer guide into the three-part structure the doc owner asked for: Part 1 - Code Architecture, Part 2 - Interfaces and Definitions, Part 3 - How to Add X, Y, Z (each ending in how to verify). Content is moved, not rewritten: Part 1 (Mental model first, per doc-owner follow-up, then a new whole-PR architecture diagram, then "How the current pieces fit together"): - old #1 Mental model -> Part 1 #1 - new: whole-PR architecture diagram (TargetSubDAG's two entry points through ReplacementStrategy, PlanSpace/cost_sorted, explanation.rs, to a downstream consumer) -> Part 1 #2 - old #3 How the current pieces fit together -> Part 1 #3 Part 2: - old Terminology's "Implementation" definition merged into the Glossary as one more entry (### Implementation), next to ReplacementStrategy - old #2 Glossary -> Part 2 #1 (plus the merged Implementation entry and old #10 Matcher, retitled to match glossary-entry style) - old #10 Matcher (implementation.rs) -> ### Matcher inside the Glossary; implementation.rs no longer exists, so the stale title is fixed - old #19's definitional content (ReplacementExplanation/ ExplanationKind shapes, node_hash, why there's no ExplanationRule trait, location-text ownership) -> Part 2 #2 Part 3: - old #4, #5, #6, #7, #13, #14 -> Part 3 #1, Adding a new ReplacementStrategy (ending in Testing a new strategy) - old #8, #9, #15 -> Part 3 #2, Adding or customizing a CostModel (ending in Testing a new cost model) - old #12 -> Part 3 #3, Adding a new sketch algorithm, with its stale implementation.rs/binder references fixed to replacement.rs/ construct_summary vocabulary, plus a new "Verifying a new sketch algorithm" close grounded in the existing coverage-matrix tests - old #11, #16, #17, #18 -> Part 3 #4-#7 (capstone + closing reference material); #18's extension-map table's implementation.rs row fixed to replacement.rs - old #19's "Using it"/"Adding a new kind" content -> Part 3 #8, Using and extending explanation.rs cargo build --workspace --all-targets is clean (docs-only change). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
zzylol
added a commit
that referenced
this pull request
Oct 14, 2026
… directly The identical-expression rule tried all 2^g subsets of the sharing groups and dropped those that repeat a candidate. With nested groups most subsets repeat (14 groups: 16 384 subsets for 130 candidates). Sharing a set of groups also shares the groups it leaves as one node; that closure maps each subset to its candidate, and the distinct candidates are its closed sets. They are now enumerated directly (Next Closure: one closure per step and group), then put in the same binary-counter order, so the output is unchanged. Twenty nested groups give their 20 candidates quickly; a brute-force test checks the same candidates on small cases. Sharing groups are computed once per candidate, and a Pass 2 candidate names its shared groups by kind and number (Scan #1, TimeRange #2), so repeated kinds stay distinguishable. The viewer fixtures are regenerated.
zzylol
added a commit
that referenced
this pull request
Oct 14, 2026
… directly The identical-expression rule tried all 2^g subsets of the sharing groups and dropped those that repeat a candidate. With nested groups most subsets repeat (14 groups: 16 384 subsets for 130 candidates). Sharing a set of groups also shares the groups it leaves as one node; that closure maps each subset to its candidate, and the distinct candidates are its closed sets. They are now enumerated directly (Next Closure: one closure per step and group), then put in the same binary-counter order, so the output is unchanged. Twenty nested groups give their 20 candidates quickly; a brute-force test checks the same candidates on small cases. Sharing groups are computed once per candidate, and a Pass 2 candidate names its shared groups by kind and number (Scan #1, TimeRange #2), so repeated kinds stay distinguishable. The viewer fixtures are regenerated.
zzylol
added a commit
that referenced
this pull request
Oct 14, 2026
…-expression top-k heaps, counter-sample weights, per-second plan-selection cost (#692) fix(logical-asap-optimization): weight Top-K heap sketches by the ranked value Pass 1's Top-K sketch update weighted every heap by the sample value. Over SQL rows that column is the one named `value`, or else the only numeric column: an Int64 key made the heap candidates fail to compose, and a key named `value` would have weighted the heap by the key. The update now reads the column the top-k ranks (the aggregate's measure over grouped rows, the sample value over per-series rows), the same column exact top-k sorts by. `ranked_column` moves from Stage 2 into the shared realization helpers so both stages use it. feat(logical-asap-optimization): Pass 2 identical-expression rule as Stage 3 candidates Stage 1 Pass 2 now applies the identical-expression rule of Algorithm 2. For each composed Pass 1 candidate it finds the sharing groups (two or more nodes with identical input and computation semantics that are legal to share) and adds one candidate per non-empty subset of the groups, keeping the independent candidate. A subset that leaves out a group that the other shared groups already merge gives the same candidate as the larger subset, so it is listed once. Nothing is capped. The variants are ordinary candidates for Stage 2 and Stage 3. PromQL rows that carry the full series identity now declare (identity, timestamp) as a unique key, which makes scans and range selectors legal to share. Plan selection treats totals that differ only by floating-point rounding as a tie and keeps the earlier candidate. A shared NotMaterialized input is charged once per consumer, so it ties with the independent plan. Example 1 gets 9 logical candidates (3 Pass 1 x (1 + 2 Pass 2)) and Example 3a gets 972. stage_pipeline's --max-combinations still counts Pass 1 combinations; its document lists every Stage 1 candidate. The Example 1 and scalar-queries viewer fixtures are regenerated. refactor(logical-asap-optimization): move the keyed-additive heap rule into the realization helpers Pass 1's whole-expression heap reuses the legacy search's keyed-additive rule, so the rule, its result types and schema_column_ref move from replacement.rs into realization.rs, which both use. Pure move. feat(logical-asap-optimization): whole-expression top-k heaps and counter-sample weights Pass 1 gains a whole-expression candidate kind. A top-k over a per-item sum or count that is itself a target (one consumer, no filter or HAVING, realizable by the keyed-additive rule) has its Count-Min and CountSketch heap candidates replaced by whole-expression ones: one heap over the inner aggregate's input, keyed by the ranked item and weighted by the summed value. It absorbs the inner target, which is then not computed. The outer top-k's partition keys are moved onto the input the heap reads. Example 1's Q2 gets exactly three candidates: exact, Count-Min + heap and CountSketch + heap. Enumeration skips choices that differ only in an absorbed target, and PerComputationCandidates::combinations counts the same set. Count-Min needs non-negative weights. Pass 1 does not read the data workload: when a heap's weights are raw samples of metric M, or plain sums of them, it records the open requirement WeightDomain::SamplesOfMetric{M}. Rate, other arithmetic and other functions end it, and no type is inferred from a `_total` name. DataWorkload gains metric_types (declared Prometheus types). Plan selection admits Count-Min over such weights only when M is declared a counter; an unconfirmed requirement counts as unmet. The planner pipeline's selected plan carries the confirmed requirement as the proof NonNegativeWeightProof::CounterSamples, since the executor reads no data workload. Example 1 declares http_requests_total a counter, so its Count-Min heap is now rejected only because a top-k output has no guarantee; without the declaration it is rejected on its weights. Raw-sample precompute now checks the whole-expression heaps. The Example 1 viewer fixture is regenerated. feat(plan-selection): price candidates per second of wall time Plan selection now prices a candidate per second, so queries with different recurrences add up, and takes each query's recurrence: - A NotMaterialized node is charged once per consumer at the sum of their rates (a shared NotMaterialized input still costs what reading it separately does). - A MatAtQuery node is computed once at the combined rate of the queries reaching it; queries at the same interval run together. - A MatAtIngestion node is priced over one second of arriving rows, once per second. - One-off, scheduled and unknown queries are spread over horizon_s. ResourceCalibration gains horizon_s (default 3600 s) and cost_per_retained_byte_second, the price of state kept between computations. Every materialized node's kept state is charged per byte per second: twice its output at ingestion time (the window being built and the completed one), once at query time. Memory held only during a query gets no per-second charge. select_plan, run_stage1_to_stage3 and run_stage1_to_stage3_from_pass1 take the recurrences; the planner pipeline and stage_pipeline read them from the workload. The cost model is described in the stage3_plan_selection module docs. The viewer fixtures are regenerated. fix(plan-selection): charge memory only as kept state per second Plan selection no longer charges a computation for the memory it holds while it runs (#682's per-computation peak-memory term, including the growing exact state of a query-time aggregate). Memory is charged only as state a materialized node keeps between computations: kept bytes times cost_per_retained_byte_second. The kept state is the node's output, or the values an exact aggregate keeps per group when that is more, so a kept exact quantile or distinct count is still sized by what it keeps. Peak capacity belongs to cost model 2 (#683). Every term is now a per-second cost; the per-computation memory coefficient is ignored. The tests that compared exact state against sketches now keep the state at query time, where its size is charged. feat(plan-selection): bind the sign of confirmed counter samples An exact sum or mean of relative-error values keeps the relative bound only over non-negative values. When the folded values are raw samples of one metric, or plain sums of them (an exact sum accumulator, a renaming, or a heap estimate whose weights carry the SamplesOfMetric requirement), Stage 1 now names the unknown sign by that metric, values_non_negative[M], and gates the bound on it. Plan selection binds it to 1 when the data workload declares M a counter; a gauge, an undeclared metric, rate, other arithmetic or a name such as _total leave it unbound, so the target stays unmet. Other folds keep the bare values_non_negative, which nothing binds. feat(devtools): limit Stage 1 candidates, Pass 2's included, while enumerating stage_pipeline's --max-combinations (default 10^4) now counts Stage 1 candidates as they are enumerated, Pass 2's identical-expression candidates included, and stops with an error as soon as the count passes the limit; it never truncates. Exit status 3 (the viewer server's 422) is unchanged. The library stays uncapped: Stage 1 is exposed as a lazy iterator (stage1_logical_asap_optimization) and Stage 2 and Stage 3 run over the candidates a caller enumerated (run_stage2_and_stage3); run_stage1_to_stage3_from_pass1 enumerates them all. planning_stages takes the limit as an Option. perf(logical-asap-optimization): enumerate distinct Pass 2 candidates directly The identical-expression rule tried all 2^g subsets of the sharing groups and dropped those that repeat a candidate. With nested groups most subsets repeat (14 groups: 16 384 subsets for 130 candidates). Sharing a set of groups also shares the groups it leaves as one node; that closure maps each subset to its candidate, and the distinct candidates are its closed sets. They are now enumerated directly (Next Closure: one closure per step and group), then put in the same binary-counter order, so the output is unchanged. Twenty nested groups give their 20 candidates quickly; a brute-force test checks the same candidates on small cases. Sharing groups are computed once per candidate, and a Pass 2 candidate names its shared groups by kind and number (Scan #1, TimeRange #2), so repeated kinds stay distinguishable. The viewer fixtures are regenerated. fix(plan-selection): every consumer gets the selected plan with confirmed weights The counter-sample requirement Stage 3 confirms was made a proof only in the planner pass, so run_stage1_to_stage3's selected_logical and the selected candidate's roots still carried the open requirement, which the executor rejects. Stage 3 now puts the plan on its output: Selection::plan is the selected candidate's roots with confirmed weights, and Stage1ToStage3Run::selected_logical returns the selected logical roots the same way. The pass reads Selection::plan. fix(logical-asap-optimization): an open weight requirement does not certify Count-Min estimate_has_bound treated any weight domain but UnknownOrSigned as proven, so a Count-Min estimate over weights with the open SamplesOfMetric requirement got its bound as if the weights were proven non-negative. Only NonNegative now counts. Such an estimate instead gets its local bound gated on the metric's sign, values_non_negative[M], which plan selection binds to 1 when M is declared a counter and leaves unbound otherwise. test: Example 1's shared-input totals tie with the independent one up to rounding Shows that the tie tolerance is needed: the shared-input candidates' totals differ from the independent candidate's in the last bits, and plan selection reports them as tied and keeps the independent one. Also drops stale wording in the acceptance helpers. test(logical-asap-optimization): Pass 2 candidate labels and roots agree Applying a candidate's label (the groups it shares) gives the same roots the direct enumeration returned, sharing exactly those groups. fix(plan-selection): the sign gate opens only at 1 The sign of one metric's samples gates a relative bound as a factor; a binder supplying 0 would have read as a perfect bound. Plan selection now binds a sign statistic only to 1 (proven) and leaves any other value unbound. docs: update the Example 1, 2 and 3 selections this PR changes The Example 3 acceptance spec and the stage viewer guide now give 3a's 972 candidates and its all-KLL selection, 3b's KLL selection tied with DDSketch, the per-second totals, and the shared-input plans that tie with the independent one.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Address review feedback on the L3 IR (
QueryExpr+AggIntent) and add the missing input/output, schema, and system-I/O specs.L3 deduplicated and made language-orthogonal
TopK(=Sort + Limit),WindowedAgg(=Window+Aggregate),Dedup(renamed toDistinct{cols})SketchAgg,JoinSketchnow live in a new L4 IRSketchExprHistogramQuantile,PromQLSubquerylower in PromQL L1→L2 insteadAggIntentlosesTopKandQuantileOverTime(surroundingWindowcovers the latter)New L4 sketch-bound IR
SketchExprSketchAgg,SketchJoin,SketchSubtract,SketchDelete,SketchEstimate,SketchMerge+Logical(QueryExpr)pass-throughfn apply(&QueryExpr) -> Option<SketchExpr>— the L3/L4 boundary is type-enforcedPer-node input/output specs
Schema/Field/HasSchematypes — every L3 edge carries a typed schemaQueryExprvariantsSketch catalog metadata
SketchEntryexpanded withmergeable,deletable,subtractable,accuracy,aggregated_keys,param_ranges,cost_model,supported_intentsSystem I/O contract
QueryWorkload, output is per-executor sub-DAG assignments§12 Open questions
Test plan
TopK,WindowedAgg,JoinSketch,HistogramQuantile,PromQLSubquery,Dedup,QuantileOverTime) outside the cleanup-rationale tables — verified via grep before push🤖 Generated with Claude Code