Repository navigation
feat: autonomous capability-match allocation — (ε+queries) → {sketches, p, δ}, applied live - #382
Merged
Merged
Conversation
…rse (autonomous-alloc slice 1)
The keystone for autonomous (epsilon + queries) -> {sketch set, p, tau} allocation.
capability_for() already lowers an AggIntent to the Capability a query needs; this
closes the loop with the inverse: sketch_families_for_capability(Capability) names
the allocatable SketchType families that provide it (QuantileApprox->DDSketch|KLL,
CardinalityApprox->HLL, Frequency*->CountSketch|CMS, ExactAgg->archive/none), honoring
concrete SketchKindHandle pins and collapsing heap-bearing hints to their matrix family.
required_sketches_for_capabilities() unions a query set's capabilities into a dedup'd,
order-stable sketch set. 6 unit tests, all green. Pure logic, no infra — entrypoint glue
(query->intents reuse of intent_algebra, epsilon->(p,tau) deriver, /plan/auto endpoint)
follows in later slices.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…plan (autonomous-alloc slice 2)
Composes the existing ASAP-tier analyzer with slice 1's inverse map. Each query is
lowered via analyze_promql_for_asap_tier (which already yields per-candidate
{metric_name, required_capability}); capabilities accumulate per metric (dedup'd),
then required_sketches_for_capabilities unions them into the SketchType families to
allocate. Partially-supported queries split correctly: a warm candidate contributes
its sketch while a residual exact/unsupported fragment is recorded under cold_only.
QuerySetPlan{per_metric, cold_only} is the warm half of (epsilon+queries)->plan.
6 unit tests incl. the real queries-e2e.json suite (nested sum-by/topk/count). green.
Next: slice 3 epsilon+rate->(p,tau) deriver; slice 4 /plan/auto endpoint + emit glue.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…er (autonomous-alloc slice 3)
Completes (epsilon, queries) -> {sketches, p, delta}. derive_sample_p applies the
unified ε-floor p=1/(1+ε²·rate) (mirrors data_plane epsilon_sample_floor; control_plane
has no data_plane dep). derive_delta_threshold sizes the even-split per-site CDM slack
δ=ε·τ/k — with the honest asymmetry that τ is a user/query input (the monitored value
of interest), NOT derivable from ε; metrics with no threshold get no δ. allocate_knobs
joins these onto slice 2's QuerySetPlan (rate + monitor supplied as closures, kept pure
for testing) -> Vec<AllocatedMetric>{metric, sketches, knobs{sample_p, delta_threshold}}.
6 unit tests green.
Next: slice 4 — POST /api/v1/plan/auto {epsilon, queries[]} + emit StreamingConfig.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…c slices 4+5)
The live entrypoint: given {epsilon, queries[], default_rate?, monitors?}, runs the
full pipeline (query_planning -> epsilon_alloc) and returns the per-metric warm plan
{sketches, sample_p, delta_threshold} + cold_only fallthrough. build_auto_plan() holds
the logic in the lib (unit-testable without the HTTP server); handle_plan_auto is a thin
axum wrapper returning AutoPlanResponse as JSON. Slice-5 e2e test posts (epsilon,queries)
incl. an unparseable query and asserts the planned quantile sketch, epsilon-floor p,
CDM delta, cold fallthrough, and clean JSON serialization. 19 autonomous-alloc tests green
(the one remaining lib failure, optimizer::rules invalid_sketch_type_override, pre-exists
on origin/main — unrelated).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…lloc) build_auto_plan now takes a rate resolver instead of a flat default. The handler sources each metric's ε-floor rate from runtime telemetry via metric_rate_from_telemetry, falling back to default_rate. Honest limitation documented: RuntimeSamplesStore is keyed by (source,sketch,impl) with no metric name, so a metric's rate is approximated by the aggregate throughput of its allocated sketch families (throughput_items_per_sec.mean); the surfaced p is a plan-time estimate — the authoritative live p is the coordinator's ε-floor. 2 new tests (telemetry read by sketch family; build_auto_plan prefers telemetry over default). 9 epsilon_alloc tests green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…s-alloc)
POST /api/v1/plan/auto now accepts apply:true. The mechanism: WorkloadRegistry gains a
runtime overlay (Arc<RwLock<Vec<WorkloadEntry>>>) that monitor_intents() reads alongside
the static YAML entries — zero blast radius on the static accessors (entries()/for_role()
unchanged). On apply, the handler synthesizes a runtime entry per allocated metric
(sketch_family_override + MonitorDecl{tau, epsilon, window}) via insert_runtime, then
calls replan_all() so the backend repost emits the monitor(ε,τ); the data-plane
coordinator then derives the live ε-floor p (per the chosen design — push sketch+monitor,
p stays coordinator-driven). sample_p kept 1.0 to avoid double-sampling. Response carries
applied:<count>. Registry overlay unit-tested (runtime_injected_monitor_is_emitted incl.
replace-by-metric); the live POST is validated against a running stack. 831 lib tests pass
(the 1 failure, optimizer::rules invalid_sketch_type_override, pre-exists on origin/main).
Note: new-metric SKETCH-plan registration (workload_store via analyzer) is the documented
remaining wire-up; apply currently lands the monitor→coordinator-p path live.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…close gap) apply now registers BOTH halves per query: analyze the query -> QueryWorkload, pin the chosen sketch, workload_store.set + replan_metric_role (emits the SKETCH plan), AND insert_runtime the monitor entry (emits monitor(ε,τ) -> coordinator derives p). Dedups by (metric, role). Cold-only / unparseable queries are skipped. Closes the documented new-metric sketch-registration gap so apply pushes the full allocation, not just monitors. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ow-derivation
Validated the full coordinator loop end-to-end on the cluster; three fixes:
- apply loop injects the monitor UNCONDITIONALLY (was gated behind analyze() which
rejects quantile_over_time → nothing injected); sketch registration is now best-effort
(analyzer rejection non-fatal) and replan_all() pushes the cumulative config now.
- AutoMonitor gains functional + key so the monitor matches what the edge reports
(whole-sketch sum/key="" per the unified design; cms_point is the retired per-key form).
- AUTO-WINDOW: window_secs is now Option; when omitted it derives from the edge CDM epoch
(CONTROLLER_MONITOR_WINDOW_SECS, default 1s). The coordinator's on_register alignment
guard silently refuses to grant when monitor.window_ms != edge.epoch_window_ms, so a
mismatch (e.g. 30s monitor vs 1s edge) no-ops the whole loop — auto-derivation prevents it.
Result: POST /plan/auto {epsilon,queries,monitors:{m:{tau}},apply:true} → monitor live in
data-plane → coordinator grants p=1/(1+epsilon^2*rate) → edge applies warm-sample-p ~0.08-0.14.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Turn the opaque 'register for unconfigured monitor — ignored' into an actionable diff: on a cfgs miss, log the requested (agg_id, req_key bytes/len) and the configured keys. This is what made the edge↔monitor (agg_id,key) alignment debuggable (revealed the edge registering under a stale 's0' key vs the whole-sketch '' monitor). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
zzylol
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Autonomous capability-match allocation: input
(ε + query strings)→ the control plane automatically derives{which sketches per metric, sampling p, CDM δ}, applies it live, and the data-plane coordinator drives the per-edge ε-floorp. No hand-written workload YAML.Pipeline (pure logic, 21 unit tests)
sketch_selection—Capability → Vec<SketchType>inverse (QuantileApprox→DDSketch|KLL, CardinalityApprox→HLL, Frequency*→CountSketch|CMS, ExactAgg→archive).query_planning— query set → per-metric sketch plan + cold-tier fallthrough (composes the existing ASAP-tier analyzer).epsilon_alloc—ε,rate → (p, δ);build_auto_planflattens to the wire response. Rate sourced from runtime telemetry (by sketch family) with a default fallback.Live apply —
POST /api/v1/plan/auto {epsilon, queries, monitors, apply:true}Per allocated metric: inject a runtime monitor into the
WorkloadRegistryoverlay (zero blast radius on the static path) + best-effort sketch registration via the analyzer, thenreplan_all()pushes the cumulative config. The monitor carriesfunctional/key(whole-sketchsum/key=""per the unified design) and an auto-derived window (CONTROLLER_MONITOR_WINDOW_SECS, default 1s = the edge CDM epoch).Validated end-to-end on the 8-node cluster
POST /plan/auto {ε=0.05, …, apply:true}→ monitor live in data-plane (agg_id, ε, key="", window_ms=1000) → coordinator hot-reloads → coordinated edge learns + registers (key="") → reports rate → coordinator grantsp = 1/(1+ε²·rate)→ edge applieswarm-sample-p 0.08–0.14(differentiated per edge by rate).Three real bugs found + fixed in the process (apply-loop ordering; window-alignment guard silently no-opping grants — fixed by auto-window-derivation; the edge↔monitor
(agg_id,key)alignment — surfaced by theapply_registerdiagnostic, root cause was a stale deployed otel-app binary registering under a legacys0key, not the current source).Note: earlier I suspected a stale otel-app image (it registered under a legacy
s0key); a no-cache rebuild + clean coordinated run disproved that — the image/source were always correct (edge registerskey=""). Thes0was a test artifact: stale per-(agg_id,key)engine state from a cms_point→sum monitor switch on a running edge. Validated end-to-end with the unmodified fresh image:warm-sample-p 0.1379granted.🤖 Generated with Claude Code