Skip to content

refactor(benchmarks): measure fixed workloads in milliseconds - #383

Draft
tisonkun wants to merge 1 commit into
apache:mainfrom
tisonkun:codex/benchmark-milliseconds
Draft

tisonkun wants to merge 1 commit into
apache:mainfrom
tisonkun:codex/benchmark-milliseconds

Conversation

@tisonkun

Copy link
Copy Markdown
Member

Summary

Adopt Benchmark In Milliseconds across the primitive and ecosystem suites. Replace Divan's per-operation sampling with fixed workloads and integer millisecond elapsed times.

  • Migrate all 160 scenarios (697 parameter and implementation combinations), with shared work sizes for each cross-library comparison.
  • Support filtering, listing, explicit work sizes, and repeated measurements; retain cargo x bench -- --test for CI smoke coverage.
  • Document timing boundaries and comparison guidance in benchmarks/README.md, and remove Divan and its sampling configuration.

Validation: cargo x lint, cargo x test, all 697 smoke cases, the complete measurement matrix, CLI checks, and ten repetitions each of the mutex workload and all four bounded MPSC implementations.

Design Notes

Each measurement times the entire workload, including setup and cleanup, without adaptive calibration, overhead subtraction, or per-operation normalization. Repetitions recreate fixtures and print every result. The threads=N cases divide a fixed total amount of work among the workers. OnceMap initialization/removal use repeatable cycles on an existing working set so rebuilding the map does not dominate each round.

Round counts target hundreds of milliseconds while remaining identical across implementations and revisions. The local sizing run had a median of 275 ms across 697 cases; 642 took 100–999 ms, with a range of 42–1280 ms. These are new workload baselines and cannot be compared directly with the previous Divan operation timings.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant