Repository navigation
Run independent benchmark setup steps in parallel - #11445
Conversation
The SQL, PR and develop benchmark jobs ran every setup step one after another. Group the steps that do not depend on each other into `parallel:` blocks: - runs-on/action alongside checkout, and assert-cpu alongside setup-prebuild. - setup-prebuild, uv, DuckDB, system info, binary download, AWS credentials, the Polar Signals agent and the data cache key all at once before data generation, then chmod, LitData and the cache restore. - debuginfo and binary uploads together after the build. - the S3 data upload alongside the cache save or Polar Signals setup. setup-benchmark.sh still runs after them, since it pins every running process to the housekeeping CPUs. - file size capture alongside the baseline fetch. Also fix the SQL build's debuginfo upload: `polarsignals-cloud-token` and `project-id` were indented into the `paths` block, so the action never got them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAMeLpyH1yASJhrK1N Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
… jobs Extend the parallel setup to the remaining benchmark workflows: - codspeed.yml: runs-on/action alongside checkout (or shard selection), setup-prebuild alongside system-info, the artifact download alongside the toolchain setup, and the cargo-codspeed install alongside unpacking. - pr-bench-gpu-compress.yml: the Rust, DuckDB and uv installs, system info and nvidia-smi together, then the cuDF install alongside the benchmark build. - s3-measurements.yml: only the setup before the timed schedule. The CodSpeed install still waits for setup-prebuild or setup-rust, since it needs the toolchain they install. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAMeLpyH1yASJhrK1N Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
develop (#11443, #11444) made "Capture file sizes" append file-size records to results.ingest.jsonl, which "Fetch baseline" reads, and added the sampled file tree dump and upload. The benchmark job now runs the capture alongside the dump, then the tree upload alongside the baseline fetch, so the fetch still sees the file-size records. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAMeLpyH1yASJhrK1N Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Merging this PR will improve performance by 11.75%2 warnings
Performance Changes
Tip Curious why performance improved? Comment Comparing Footnotes
|
Polar Signals Profiling ResultsLatest Run
Previous Runs (8)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.030x ➖, 0↑ 2↓)
No file size changes detected. |
Once its inputs were fixed, the Polar Signals debuginfo upload took 5m37s in the build job. That took the build job from 4m50s to 8m32s, and every bench job waits for it. The upload now runs in its own job on an arm64-medium runner. It downloads the build artifact and uploads alongside the bench jobs. Polar Signals matches debuginfo to profiles by build ID, so the later upload still symbolizes them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAMeLpyH1yASJhrK1N Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.997x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.018x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.008x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.008x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.990x ➖, 0↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.998x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.992x ➖, 0↑ 0↓)
No file size changes detected. |
As in the SQL benchmarks, pr-bench-runner.yml and develop-bench.yml upload Polar Signals debuginfo in the build job, so the bench job waits for it. Each now uploads from its own job on an arm64-medium runner. That job downloads the build artifact and runs alongside the bench job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAMeLpyH1yASJhrK1N Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.990x ➖, 2↑ 0↓)
datafusion / parquet / ns (0.995x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.992x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.986x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.000x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.002x ➖, 1↑ 1↓)
No file size changes detected. |
|
The red "CodSpeed Performance Analysis" check on
So this looks like walltime noise. Whether to acknowledge it in CodSpeed is the maintainers' call. Generated by Claude Code |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.963x ➖, 3↑ 1↓)
datafusion / parquet / ns (0.966x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.941x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.024x ➖, 0↑ 0↓)
|
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.995x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.998x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.995x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.994x ➖, 0↑ 0↓)
No baseline file sizes found for base commit. |
Benchmarks: String Encoding 📖Commits: PR vortex / vortex-file-compressed / ns (0.999x ➖, 0↑ 0↓)
vortex / vortex-file-compressed / bytes (1.000x ➖, 0↑ 0↓)
|
## Summary A follow-up to #11445. Step timings from recent runs showed CI jobs outside the benchmarks still doing independent setup and checks one step at a time. This groups those steps into `parallel:` blocks, splits the Java job into a matrix, and speeds up the SQL benchmark binary handoff. It also fixes the SQL benchmarks' DuckDB runs, which have been failing since the Rust 1.100 upgrade. ## Changes **SQL benchmarks (`sql-bench-matrix.yml`)** - The build job packs the bench binaries into one tar with multi-threaded zstd and uploads it with `compression-level: 0`, instead of letting upload-artifact zip them with single-threaded zlib. Packing and uploading took 12s, down from 64–90s. Each bench job then spends about 4s unpacking, and every bench job starts 50–75s sooner. The tar also keeps the executable bits, so the `chmod` step is gone. - **Fix:** every DuckDB benchmark failed with `libduckdb.so: cannot open shared object file`, on other branches too. Rust 1.100's cargo moved build-script output from `build/<pkg>-<hash>/out` to `build/<pkg>/<hash>/out`. `vortex-duckdb/build.rs` keeps its DuckDB cache two levels above `OUT_DIR`, so the library moved to `build/vortex-duckdb/vortex-duckdb-cache/`, and the hard-coded upload path matched nothing. upload-artifact skipped the missing file silently. The build now searches `target/release_debug/build` for the library and fails if it is missing, because `duckdb-bench` cannot run without it. It loads the library through an absolute rpath, so the archive keeps its relative path. **Pre-merge CI (`ci.yml`)** - Java: a matrix of four jobs. One leg runs `javadoc check` without the Spark compat suites, and one leg runs each Spark compat suite (4.1, 4.2, 4.3). Gradle `--parallel` only splits work across projects, and all three compat suites live in `:vortex-spark-4.0_2.13`, so they ran one after another. A "Java" gate job keeps the required check name. It runs with `always()` and fails if any leg failed or was cancelled, because a skipped required check counts as passed. - License check: `cargo deny check advisories` and `check bans licenses sources` run at the same time. The advisory warning still runs after them, and now has `!cancelled()` so it isn't skipped when both checks fail. - Python (test): after the main suite has built the extension and synced the environment, the datasets-floor, Polars-floor and docs checks run at the same time. - Lightweight checks: the uv setup runs alongside the apt install. **Other PR-path workflows** - `docs-build.yml`: - Rust, JDK and uv install at the same time. - Javadoc (3m41s) builds alongside the Sphinx build (7m15s). - The Javadoc is copied into `docs/_build/html/_static` after the Sphinx build. The docs link to it with raw HTML, and Sphinx only copies `_static` into the output, so the built site is the same. - `publish-dry-runs.yml`, wheel build: setup-prebuild, Zig and uv install at the same time. The Rust cache still waits for setup-prebuild, because its key includes the rustc version. - `web.yml`, `web-build.yml`: Rust, wasm-pack, Bun and Node install at the same time. **Fuzz, package and release workflows** - `run-fuzzer.yml`, `minimize_fuzz_corpus_workflow.yml`: `runs-on/action` and checkout run together, then building or downloading the fuzzer runs alongside the corpus restore. - `fuzz-coverage.yml`: the corpus download (python3 only) runs alongside setup-prebuild. - `close-fixed-fuzzer-issues.yml`: the LLVM setup runs with the cargo-fuzz install. `setup-fuzz-linker` stays on its own, because both it and the LLVM setup may run `apt-get`. - `wasm-fuzz.yml`: the wasm32 build runs with the wabt install. `setup-fuzz-linker` still comes after, because it sets mold `RUSTFLAGS`. - `build-fuzzers.yml`, `package.yml`, `release-binaries.yml`, `publish.yml`, `compat-gen-upload.yml`: `runs-on/action` and checkout run together, and the JDK, Rust, Zig, AWS-credential and artifact-download steps that don't depend on each other run at the same time. **Not changed** - Windows: splitting the wheel build into its own job was tried and reverted. The runner pool keeps one Windows instance warm and a cold one takes 85–105s to boot, which costs more than the 67s it took off the critical path. - Rust tests (cpu-features): "Check CPU features" and the test step compile into the same cargo target directory, so cargo would make them wait for each other. - The CUDA, sanitizer, musl and Rust/Clippy jobs: their time is almost all compiling and running tests. - No matrix caps `max-parallel`, so all matrix legs already run at once. ## Measured | | Before | After | |---|---|---| | SQL bench: package + upload binaries | 64–90s | 12s | | SQL bench: download + unpack, per bench job | 16–20s | 17–31s | | Java, warm Gradle cache | 100s | 81s longest leg + 2s gate | | Java, cold Gradle cache | 546–588s | the compat suites (about 1.5–2 min each) no longer run in series | | License check | 92–110s | 67–84s | | Docs build | 11m33s | 8m39s–10m36s | The `ci.yml` wall time is still set by Windows (about 8.5–10.5 min), so these changes shorten individual jobs, not the whole run. The SQL benchmark changes shorten the bench run directly. On the latest commit the SQL benchmarks all finished, each with a "No clear signal" verdict, as expected for a CI-only change. ## Checks - `yamllint --strict -c .yamllint.yaml .github/` passes. - For every job in every workflow, I parsed the old and new YAML, flattened the `parallel:` groups and compared the steps. They are identical apart from the rewritten Javadoc copy step, the `!cancelled()` on the advisory warning, the Java matrix, and the SQL bench packaging. - I ran Cargo 1.100 locally to confirm the new `OUT_DIR` layout, and checked that the `libduckdb.so` search matches it. - A separate review pass checked each group for shared files, environments, outputs, `PATH`/`GITHUB_ENV` needs and apt-lock conflicts. - CI on the latest commit is green, including all four Java legs, the Java gate, and the integration smoke tests. An earlier smoke-test failure, where the RustFS bucket creation returned an HTTP error after the health check passed, did not recur. --- 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01UeniBAMeLpyH1yASJhrK1N --------- Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Signed-off-by: Claude <noreply@anthropic.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Summary
The benchmark jobs run every setup step one after another, even though most of the steps don't depend on each other. This change groups the independent steps into
parallel:blocks, the same wayci.ymlalready does, to shorten the time a benchmark machine spends before the timed run. It also moves the Polar Signals debuginfo upload out of the build jobs, so the bench jobs no longer wait for it.