Repository navigation
Run more independent CI steps in parallel - #11450
Conversation
Step timings from recent runs show several jobs still doing independent work one step at a time: - ci.yml, license check: `cargo deny check advisories` (44s) and `check bans licenses sources` (39s) now run at the same time. The advisory warning still runs after them. - ci.yml, Python (test): after the main suite builds the extension and syncs the environment, the datasets-floor, Polars-floor and docs checks run at the same time. - ci.yml, lightweight checks: the uv setup runs alongside the apt install. - publish-dry-runs.yml, wheel build: setup-prebuild, Zig and uv install at the same time. The Rust cache still waits for setup-prebuild, because its key includes the rustc version. - docs-build.yml: Rust, JDK and uv install at the same time. Javadoc (3m41s) now builds alongside the Sphinx build (7m15s) instead of before it, and is copied into docs/_build/html/_static afterwards. The docs link to it with raw HTML and Sphinx only copies _static into the output, so the site is the same. - web.yml and web-build.yml: Rust, wasm-pack, Bun and Node install at the same time. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAMeLpyH1yASJhrK1N Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
- run-fuzzer.yml and minimize_fuzz_corpus_workflow.yml: runs-on/action with checkout, then building or downloading the fuzzer alongside the corpus restore. setup-fuzz-llvm still runs first. - fuzz-coverage.yml: the corpus download (python3 only) runs alongside setup-prebuild. - close-fixed-fuzzer-issues.yml: setup-fuzz-llvm with the cargo-fuzz install. setup-fuzz-linker stays on its own, since both it and the LLVM setup may run apt-get. - wasm-fuzz.yml: the wasm32 build with the wabt install. setup-fuzz-linker still comes after, because it sets RUSTFLAGS for mold. - build-fuzzers.yml, package.yml, release-binaries.yml, publish.yml and compat-gen-upload.yml: runs-on/action with checkout, and the JDK, Rust, Zig, AWS-credential and artifact-download steps that don't depend on each other. Also keep the cargo deny advisory warning when bans/licenses/sources fails in the same group: it now has `!cancelled()`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAMeLpyH1yASJhrK1N Signed-off-by: Claude <noreply@anthropic.com>
…zzer-issues When clang or llvm-symbolizer is missing, setup-fuzz-llvm installs LLVM with aminya/setup-cpp, which can repoint cc and set CC/CXX. That must not happen while cargo-fuzz is being built, so these steps go back to their original order. Only runs-on/action and checkout still run together. I, Joe Isaacs <joe.isaacs@live.co.uk>, hereby add my Signed-off-by to this commit: f9ad7ca Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAMeLpyH1yASJhrK1N Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Merging this PR will not alter performance2 warnings
Comparing Footnotes
|
Pre-merge CI: - Windows: the wheel build (67s) ran after the Rust tests on the same runner, but shares nothing with them. It now runs as "Windows wheel (x64)" beside "Windows tests (x64)". This takes about a minute off the longest PR job. - Java: the Spark 4.1, 4.2 and 4.3 compat suites live in one Gradle project, and Gradle runs one project's tasks one at a time, so they took about 6.5 minutes in series whenever the JNI library changed. Each suite now runs on its own runner, beside the remaining javadoc and check tasks. - "Windows (x64)" and "Java" become small jobs that gather the split jobs and fail if any failed, so required checks with those names still cover everything. Skipped jobs count as passed, as before. Benchmarks: - sql-bench-matrix.yml: upload-artifact zipped the bench binaries with single-threaded zlib (89s on the build job). The build job now packs them into a tar with multi-threaded zstd and uploads it uncompressed, as the CodSpeed jobs already do. The bench and debuginfo jobs unpack it, and the tar keeps the executable bits, so the chmod step is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAMeLpyH1yASJhrK1N Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
- Windows: undo the wheel/test split. The pool keeps only one Windows instance hot and a cold one takes 85-105s to boot, so a second Windows job could cost more than the 67s wheel build it took off the critical path. The job is back exactly as it was on develop. - Java gate: run with `always()`. With `!cancelled()`, cancelling the run skipped the gate, and a skipped required check counts as passed. - sql-bench-matrix.yml: install zstd before unpacking if the runner image lacks it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAMeLpyH1yASJhrK1N Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
The SQL benchmark build failed at "Package binaries": the duckdb-lib-*-prebuilt/libduckdb.so glob matched nothing, so tar got the literal path and stopped. The old upload-artifact paths skipped a missing file silently, and the benchmarks ran without it. Expand the glob with nullglob so a missing libduckdb is left out again, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAMeLpyH1yASJhrK1N Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Polar Signals Profiling ResultsLatest Run
Previous Runs (6)
Powered by Polar Signals Cloud |
Benchmarks: PolarSignals Profiling 📖Commits: PR datafusion / vortex-file-compressed / ns (1.003x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: FineWeb NVMe 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.968x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.002x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.994x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.000x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.984x ➖, 1↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.996x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-H SF=1 on S3 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.132x ➖, 0↑ 4↓)
datafusion / parquet / ns (1.101x ➖, 0↑ 1↓)
duckdb / vortex-file-compressed / ns (0.987x ➖, 1↑ 1↓)
duckdb / parquet / ns (1.046x ➖, 0↑ 0↓)
|
Every SQL benchmark that uses DuckDB failed with "libduckdb.so: cannot open shared object file", on develop's layout as well as this branch. Rust 1.100's cargo moved build script output from build/<pkg>-<hash>/out to build/<pkg>/<hash>/out. vortex-duckdb keeps its DuckDB cache two levels above OUT_DIR, so the library is now under build/vortex-duckdb/vortex-duckdb-cache/, and the hard-coded build/vortex-duckdb-cache/ path matched nothing. upload-artifact skipped the missing file silently. Find the library under target/release_debug/build instead, and fail the build if it is missing, because duckdb-bench cannot run without it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAMeLpyH1yASJhrK1N Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Benchmarks: TPC-H SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.006x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.003x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.007x ➖, 0↑ 0↓)
duckdb / parquet / ns (1.004x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=1 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (1.000x ➖, 0↑ 0↓)
datafusion / parquet / ns (1.001x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.002x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.996x ➖, 1↑ 0↓)
No file size changes detected. |
Benchmarks: Clickbench on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.987x ➖, 0↑ 1↓)
datafusion / parquet / ns (0.993x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (1.004x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.992x ➖, 0↑ 0↓)
No file size changes detected. |
Benchmarks: TPC-DS SF=10 on NVME 📖Commits: PR How to read Verdict and Engines
datafusion / vortex-file-compressed / ns (0.999x ➖, 1↑ 0↓)
datafusion / parquet / ns (0.999x ➖, 0↑ 0↓)
duckdb / vortex-file-compressed / ns (0.993x ➖, 0↑ 0↓)
duckdb / parquet / ns (0.993x ➖, 1↑ 0↓)
No baseline file sizes found for base commit. |
## Summary Every SQL benchmark that runs DuckDB fails with: ``` duckdb-bench: error while loading shared libraries: libduckdb.so: cannot open shared object file: No such file or directory ``` This started with the Rust 1.100 upgrade (#11446). Rust 1.100's cargo moved build script output from `build/<pkg>-<hash>/out` to `build/<pkg>/<hash>/out`. `vortex-duckdb/build.rs` keeps its DuckDB cache two levels above `OUT_DIR`, so the library moved: - before: `target/release_debug/build/vortex-duckdb-cache/duckdb-lib-*/libduckdb.so` - now: `target/release_debug/build/vortex-duckdb/vortex-duckdb-cache/duckdb-lib-*/libduckdb.so` The artifact path still names the old location. upload-artifact leaves out an unmatched path without an error, so the bench jobs get `duckdb-bench` without the library that its absolute rpath points to. ## Changes `sql-bench-matrix.yml`, build job: - The artifact path matches the library at any depth under `target/release_debug/build`, so it works with both layouts. It also matches a DuckDB built from source as well as a prebuilt one. upload-artifact still roots the artifact at `target/release_debug`, so the library unpacks at the path the rpath expects. - A new step fails the build job if `libduckdb.so` is missing. A missing library now stops the build, instead of letting every DuckDB benchmark fail later. A sturdier fix would be for `vortex-duckdb/build.rs` to stop deriving its cache location from `OUT_DIR`, since cargo can change that layout again. I kept this PR to the workflow. Checks: - `yamllint --strict -c .yamllint.yaml` passes. - With cargo 1.100, a local build script prints `OUT_DIR=.../target/debug/build/<pkg>/<hash>/out`. - I ran the same fix, as a tar-based packaging step, on #11450. There FineWeb, TPC-H SF=1 and SF=10, TPC-DS SF=1 and SF=10, ClickBench and TPC-H on S3 all finished with DuckDB results. Before the fix they failed with the error above, on that PR and on other branches. --- --- _Generated by [Claude Code](https://claude.ai/code/session_01UeniBAMeLpyH1yASJhrK1N)_ Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Keep this branch's tar-based packaging of the SQL bench binaries, which already finds and checks libduckdb.so the way #11462 does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01UeniBAMeLpyH1yASJhrK1N Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
Summary
A follow-up to #11445. Step timings from recent runs showed CI jobs outside the benchmarks still doing independent setup and checks one step at a time. This groups those steps into
parallel:blocks, splits the Java job into a matrix, and speeds up the SQL benchmark binary handoff. It also fixes the SQL benchmarks' DuckDB runs, which have been failing since the Rust 1.100 upgrade.Changes
SQL benchmarks (
sql-bench-matrix.yml)compression-level: 0, instead of letting upload-artifact zip them with single-threaded zlib. Packing and uploading took 12s, down from 64–90s. Each bench job then spends about 4s unpacking, and every bench job starts 50–75s sooner. The tar also keeps the executable bits, so thechmodstep is gone.libduckdb.so: cannot open shared object file, on other branches too. Rust 1.100's cargo moved build-script output frombuild/<pkg>-<hash>/outtobuild/<pkg>/<hash>/out.vortex-duckdb/build.rskeeps its DuckDB cache two levels aboveOUT_DIR, so the library moved tobuild/vortex-duckdb/vortex-duckdb-cache/, and the hard-coded upload path matched nothing. upload-artifact skipped the missing file silently. The build now searchestarget/release_debug/buildfor the library and fails if it is missing, becauseduckdb-benchcannot run without it. It loads the library through an absolute rpath, so the archive keeps its relative path.Pre-merge CI (
ci.yml)javadoc checkwithout the Spark compat suites, and one leg runs each Spark compat suite (4.1, 4.2, 4.3). Gradle--parallelonly splits work across projects, and all three compat suites live in:vortex-spark-4.0_2.13, so they ran one after another. A "Java" gate job keeps the required check name. It runs withalways()and fails if any leg failed or was cancelled, because a skipped required check counts as passed.cargo deny check advisoriesandcheck bans licenses sourcesrun at the same time. The advisory warning still runs after them, and now has!cancelled()so it isn't skipped when both checks fail.Other PR-path workflows
docs-build.yml:docs/_build/html/_staticafter the Sphinx build. The docs link to it with raw HTML, and Sphinx only copies_staticinto the output, so the built site is the same.publish-dry-runs.yml, wheel build: setup-prebuild, Zig and uv install at the same time. The Rust cache still waits for setup-prebuild, because its key includes the rustc version.web.yml,web-build.yml: Rust, wasm-pack, Bun and Node install at the same time.Fuzz, package and release workflows
run-fuzzer.yml,minimize_fuzz_corpus_workflow.yml:runs-on/actionand checkout run together, then building or downloading the fuzzer runs alongside the corpus restore.fuzz-coverage.yml: the corpus download (python3 only) runs alongside setup-prebuild.close-fixed-fuzzer-issues.yml: the LLVM setup runs with the cargo-fuzz install.setup-fuzz-linkerstays on its own, because both it and the LLVM setup may runapt-get.wasm-fuzz.yml: the wasm32 build runs with the wabt install.setup-fuzz-linkerstill comes after, because it sets moldRUSTFLAGS.build-fuzzers.yml,package.yml,release-binaries.yml,publish.yml,compat-gen-upload.yml:runs-on/actionand checkout run together, and the JDK, Rust, Zig, AWS-credential and artifact-download steps that don't depend on each other run at the same time.Not changed
max-parallel, so all matrix legs already run at once.Measured
The
ci.ymlwall time is still set by Windows (about 8.5–10.5 min), so these changes shorten individual jobs, not the whole run. The SQL benchmark changes shorten the bench run directly. On the latest commit the SQL benchmarks all finished, each with a "No clear signal" verdict, as expected for a CI-only change.Checks
yamllint --strict -c .yamllint.yaml .github/passes.parallel:groups and compared the steps. They are identical apart from the rewritten Javadoc copy step, the!cancelled()on the advisory warning, the Java matrix, and the SQL bench packaging.OUT_DIRlayout, and checked that thelibduckdb.sosearch matches it.PATH/GITHUB_ENVneeds and apt-lock conflicts.🤖 Generated with Claude Code
https://claude.ai/code/session_01UeniBAMeLpyH1yASJhrK1N