Skip to content

Add a TSBS cpu-only time-series benchmark suite - #10218

Closed
joseph-isaacs wants to merge 3 commits into
ji/tsm-benchfrom
ji/tsbs-bench
Closed

joseph-isaacs wants to merge 3 commits into
ji/tsm-benchfrom
ji/tsbs-bench

Conversation

@joseph-isaacs

Copy link
Copy Markdown
Contributor

Summary

Adds tsbs, which runs the cpu-only use case of the Time Series Benchmark Suite (TSBS). TSBS is the benchmark most time-series databases publish results on. The suite runs only under the action/bench-sql-extended label. Stacked on #10217.

The data uses the configuration from the TSBS README: 4,000 hosts over three days with one reading every 10 seconds, which is 103,680,000 rows. Each row has ten text tags and ten CPU metrics stored as doubles. The data is synthetic, as it is in every published time-series benchmark.

Changes

  • vortex-bench/src/tsbs: runs tsbs_generate_data at a pinned commit through go run, parses its output and writes zstd level 3 Parquet. It checks the row count and the generator's header lines. Columns are typed the way TSBS's TimescaleDB loader creates them.
  • vortex-bench/sql/tsbs.sql: one query of each of the 15 DevOps query types. They were produced by tsbs_generate_queries at the same commit, seed and scale, in its TimescaleDB format without a tags table. They were then changed so the same text runs on DataFusion and DuckDB:
    • time_bucket became date_trunc, which gives the same 1-minute and 1-hour buckets.
    • Time literals with a +0000 offset became TIMESTAMP literals.
    • The time column is quoted.
  • vortex-bench/sql/tsbs.md: explains the suite and lists those changes.
  • The suite is registered in the bench crate, the v3 result dimensions and the orchestrator. The suite lists in bench-orchestrator/README.md and the bench-performance skill now name both time-series suites.
  • bench-orchestrator/.../catalog.py: adds tsbs-nvme to the pr-full preset only, running DataFusion and DuckDB on Parquet and Vortex.
  • .github/workflows/sql-bench-matrix.yml: installs Go through actions/setup-go v7.0.0, pinned by commit hash, only for the TSBS matrix entry.

Checks run:

  • cargo clippy -p vortex-bench --all-targets --all-features -- -D warnings passes.
  • cargo test -p vortex-bench passes, including new tests for the generator-output parser, header validation and the query file.
  • cargo +nightly-2026-09-10 fmt -p vortex-bench -- --check is clean.
  • The orchestrator tests pass, 33 in total, and ruff and yamllint --strict are clean. vx-bench matrix lists tsbs-nvme only for pr-full.
  • data-gen tsbs --formats parquet,vortex built the dataset in about 7 minutes: 984 MB as Parquet and 1.7 GB as Vortex.
  • datafusion-bench tsbs --formats parquet,vortex ran all 15 queries.
  • The DuckDB bench binary was not run. The same SQL ran on the DuckDB 1.5.6 Python package, over a coarser sample with all 4,000 hosts, and returned the same row counts as DataFusion.

Two things for reviewers:

  • Vortex compresses this data worse than Parquet. The Vortex files are 1.7 GB against 984 MB for Parquet. That is relevant to the epic's encoding work.
  • lastpoint is slow. It takes about a minute per run per target, because DataFusion sorts every row for DISTINCT ON. Its iterations could be reduced if the extended run takes too long.

The Go setup step has not run on the bench runners yet. The first extended run will exercise it.

🤖 Generated with Claude Code

https://claude.ai/code/session_0168W9zatdzbCo2A9kw7sJLJ


Generated by Claude Code

claude added 3 commits October 2, 2026 13:43
Add `tsbs`, which runs the cpu-only use case of the Time Series
Benchmark Suite at its README configuration: 4,000 hosts over three
days, 103,680,000 rows of 10-second CPU metrics.

The data is generated with `tsbs_generate_data` at a pinned commit via
`go run`, then parsed and written as zstd level 3 Parquet. The 15
DevOps queries were generated with `tsbs_generate_queries` at the same
commit and seed, and rewritten from TimescaleDB SQL so the same text
runs on DataFusion and DuckDB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168W9zatdzbCo2A9kw7sJLJ
Signed-off-by: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168W9zatdzbCo2A9kw7sJLJ
Signed-off-by: Claude <noreply@anthropic.com>
Add TSBS to the `pr-full` preset alone, and install Go for its matrix
entry, since its data is generated with `go run`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0168W9zatdzbCo2A9kw7sJLJ
Signed-off-by: Claude <noreply@anthropic.com>
@github-actions
github-actions Bot temporarily deployed to docs-preview/pr-10218 October 2, 2026 14:09 Inactive
@github-actions

Copy link
Copy Markdown
Contributor

This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days

@github-actions github-actions Bot added the stale This PR is stale and will be auto-closed soon label Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

This PR is still active. It is one of the four time-series benchmark PRs needed for #10171: #10214 (Westermo), #10216 (LO2), #10217 (TSM-Bench) and #10218 (TSBS). Removing the stale label.


Generated by Claude Code

@joseph-isaacs joseph-isaacs removed the stale This PR is stale and will be auto-closed soon label Oct 20, 2026 — with Claude
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

This PR has been marked as stale because it has been open for 14 days with no activity. Please comment or remove the stale label if you wish to keep it active, otherwise it will be closed in 7 days

@github-actions github-actions Bot added the stale This PR is stale and will be auto-closed soon label Nov 7, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This PR was closed because it has been inactive for 7 days since being marked as stale.

@github-actions github-actions Bot closed this Nov 17, 2026

This branch was previously deployed

1 inactive deployment
docs-preview/pr-10218 — f98a455a Deployed Oct 2, 2026 by github-actions[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale This PR is stale and will be auto-closed soon

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants