Tracking: upgrade to asap_sketchlib v0.4.0 #911

Goal

Move ASAPQuery from the pinned asap_sketchlib rev 5158bf2 (v0.3.0 + 20 commits) to the published asap_sketchlib = "0.4.0". v0.4.0 removes the legacy MessagePack wrappers, MessagePackCodec, protobuf and serde on sketch types; ASAPv1 is the only serialized form. See the v0.4.0 changelog and ProjectASAP/asap_sketchlib#209 for the rename table.

Correctness first: query answers after the upgrade match the old code exactly unless a changelog entry explains the difference.

Steps

• A
• B

Scope of B (one commit each)

1. Set aggregators.
   • DeltaSetAggregatorAccumulator builds SetAggregatorDelta (sketches::set_aggregator) and serializes with to_asapv1.
   • SetAggregatorAccumulator moves from its length-prefixed format to SetAggregator::to_asapv1.
   • Keys stay on to_semicolon_str, as everywhere else in the engine; the encoding fix is query-engine: semicolon-joined group keys collide #912.
2. Removed wrappers → native types, each checked against the old wrapper's source (parameters, defaults, hashing, query semantics), with the mapping written in the commit:
   • CountMinSketch wrapper → native CountMinSketch
   • CountMinSketchWithHeap / CmsHeapItem → native CountMinSketchWithHeap (TopKHeap / HeapEntry; i64 heap counts; heavy_only → count_every_key); drop HeapItemReexport
   • HllSketch / HllVariant → HyperLogLog* (ErtlMLE → Ertl)
   • KllSketch → Kll
   • HydraKllSketch → Hydra with HydraCounter::Kll (query_quantile → query_cdf)
3. Renames.
   • DDSketch → DdSketch; store_counts() → positive_store_counts() in accumulator_factory.rs
   • PromSketch store: EHSketchList → EhBucketSketch, KLL::init_kll → Kll::with_k, EHUnivOptimized → ExponentialHistogramUnivMon (update → update_weighted), calc_* → estimate_*, query_interval_merge → query_interval
4. Deserialize. Every sketch accumulator gets a real deserialize_from_bytes via from_asapv1. CMS-with-heap and HydraKLL return "not implemented" today, and DeltaSet reads a format it never writes.
5. Memory accounting. memory_usage_bytes uses the native memory_bytes() wherever it exists (at least Kll); keep the hand estimates elsewhere.
6. OTLP ingest. otel.rs drops proto::sketchlib::SketchEnvelope; the debug log uses peek_kind_id. The receiver forwarding nothing is tracked in query-engine: OTLP receiver never forwards metrics to the precompute engine #910.
7. Tests that reach into .inner. engines/window_merger.rs, engines/simple_engine/sql.rs, tests/test_utilities/merge_fixtures.rs.
8. Debug script. asap-tools/experiments/post_experiment/debug/read_dumped_precomputes.py decodes ASAPv1.
9. Docs and comments. Stale references to old names (e.g. aggregation_config.rs DDSketch comment, accumulator doc comments).

Tests

A (baselines). Deterministic synthetic streams per sketch accumulator:

• empty sketch (what a query returns)
• single item; duplicates
• negative values and zero (DDSketch)
• a NaN input (KLL)
• large weights
• merges of 2 and 3 parts, in more than one order

B, per sketch:

• A's baselines reproduce exactly. Any deviation must trace to a changelog entry and be checked against exact ground truth. Expected deviations:
  • UnivMon cardinality / L2 fixes (PromSketch EHUniv)
  • seed derivation for sketches with more than 20 rows
  • HydraKLL, if Hydra hashes subpopulation keys differently from the old wrapper
• Merge law: merging parts answers the same as one sketch over the whole stream.
• ASAPv1 round-trip: deserialize_from_bytes(serialize_to_bytes(x)) answers every query the same.
• Error paths: wrong-type merge still Err; poisoned serialize in merge_fixtures.rs still Err.

B, set aggregators:

• Round-trip keeps the same key sets.
• Byte-identical output for the same sets inserted in different orders.
• A key in both added and removed makes serialize_to_bytes return Err.
• peek_kind_id is 08 00 / 08 80.

B, OTLP: a valid ASAPv1 payload logs its kind; garbage and truncated payloads log a decode error without panicking.

B, whole PR:

• Full suite passes; cargo tree -i asap_sketchlib shows only 0.4.0.
• Throughput before/after on a fixed ingest-heavy workload from benchmarks/; a slowdown over 10% is discussed before merge.

Decisions

• Hard cutover. No backward compatibility for the old byte formats. Outside tests, nothing in the repo reads accumulator bytes back except the spill writer, which is write-only, and the debug script.
• Single dependency. Switch the pin to 0.4.0 directly; no side-by-side old/new dependency.
• Bytes. Correctness means query results and merges; the bytes only need to round-trip.
• Toolchain. v0.4.0 needs Rust ≥ 1.85; CI uses stable and the Dockerfiles use rust:1.89. Nothing to change.

Out of scope

• OTLP receiver forwarding nothing to the precompute engine (query-engine: OTLP receiver never forwards metrics to the precompute engine #910)
• Lossless group-key encoding (query-engine: semicolon-joined group keys collide #912); done after this so the baselines isolate the upgrade
• Non-sketch accumulators' byte formats (sum, min/max, increase)