Repository navigation
Benchmark long-lived PET sessions, real concurrent resolves, and inventory scaling #533
Description
Tracking plan: #528
Priority: P2, required before structural performance optimization. Evidence: source inspection and existing benchmark inventory.
Problem
The gated summary starts a fresh PET process for both members of every cold/warm pair. It measures persisted-cache warmth, not repeated operations in a long-lived VS Code session. The function named test_concurrent_resolve_performance actually resolves sequentially, and the workflow selects only the summary test. Inventory checks compare counts, not identities.
The exact-audit-revision release workload contains only 5 environments on Linux, 8 on Windows, and 10 on macOS. This cannot establish bounds on thread growth, cache retention, or large workspace behavior. Existing real-manager benchmarks remain valuable and should not be replaced.
Sources: cold/warm summary, sequential concurrent-resolve test, performance workflow.
Scope
Add a deterministic fixture workload alongside the installed-manager benchmark. Separate process-cold/disk-warm/same-process-warm scenarios, and exercise actual overlapping refresh/resolve/configure activity. Sweep representative inventory/workspace sizes (for example 1, 10, 100, and 1000), with expensive stress/soak variants isolated from the fast PR gate if necessary.
Measure client latency, first-result latency, peak workers/threads, subprocess counts, cache/resource growth, and retained memory where reliable platform APIs exist. Compare normalized environment/manager identities, not just counts. Keep diagnostic output privacy-safe and ensure measurement does not backpressure the timed transport.
Acceptance criteria
- CI runs a repeated-refresh workload against one server process and an actually concurrent resolve workload; test naming and selection match behavior.
- Create/delete/edit/alias-churn scenarios verify fresh, correctly identified environments in a long-lived process.
- Inventory assertions detect one environment disappearing and another replacing it even when counts match.
- Size/concurrency sweeps produce usable latency and resource measurements with explicit sample counts and reproducible fixtures.
- Resource expectations/budgets are calibrated from measurements rather than arbitrary performance claims or unstable host-wide counters.
- Fast deterministic checks remain PR-friendly; longer stress runs have a documented invocation and CI schedule/entry point.
Dependencies
Depends on #531 for accurate timing definitions and #529 for reliable process teardown. Reuse the subprocess fixtures/cleanup contract from #530. Outputs establish the baseline for bounded scheduling, output-lock isolation, and Poetry indexing under #528.
Activity
karthiknadig commentedon Sep 23, 2026
Read-only follow-up after #542 merged: there is no pet-client crate; the timing client is PetClient inside crates/pet/tests/e2e_performance.rs, separate from the timeout/demultiplexing PetJsonRpcClient integration helper. A new client crate/example is not a prerequisite and would unnecessarily expand scope.
A safe, independently shippable next slice is the inventory invariant: assert_stable_inventory and the separate cold diagnostic pass currently compare only (environment count, manager count). They can accept a missing environment replaced by another. Strengthen all measured cold/warm pairs and diagnostics to compare sorted normalized identities while retaining duplicate multiplicity and existing serialized count fields. Normalization should use PET path helpers without filesystem canonicalization; tests must cover reorder-only equivalence, same-count replacement, duplicates, manager changes, and Windows path aliases. Compare privately and emit count/category diagnostics, not raw user paths. This requires no metric schema/budget change and does not depend on #529/#530 teardown or modify #526 probing.
The larger same-process/concurrent workload remains open: requests must be submitted before awaiting responses, use distinct executable fixtures for concurrent resolves, and keep sequential refresh boundaries because notifications carry no request ID. Reliable active-child teardown remains a prerequisite for claiming the full stress/soak acceptance criteria.
test: compare benchmark inventory identities (Fixes #551) (#552)
karthiknadig commentedon Sep 24, 2026
Reusable measurement lesson from #555: separate hosted macOS runs showed a substantial 192/284ms vs 117/183ms RTT gap despite identical runner-image versions and green budgets. Do not repeatedly rerun until green or call that noise without a control. A counterbalanced same-host run of exact base/head binaries with one harness/workspace/environment reduced the summary gap to 99/118ms vs 105/137ms, with overlapping median ranges. Run 36030045611 preserves the eight passes (B-H-H-B-H-B-B-H), exact binary/source/toolchain provenance, and equal SHA-256 fingerprints of sorted normalized inventory identities, including duplicate multiplicity. Instrumentation occurs outside measured windows. Check workflow success as well as artifacts: summary files can be retained before a final assertion fails. Report medians of per-pass percentiles as such, never as pooled percentiles. The diagnostic branch is temporary and not for merge; no budget was relaxed. Full decision: #555 (comment)
test: benchmark long-lived sessions and inventory scale (Fixes #533) (#…