diff --git a/CLAUDE.md b/CLAUDE.md index 10dcf6d..cd07246 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -83,7 +83,7 @@ The previous site generations (v2 and v3) are documented in [`docs/bench/legacy. - **Don't refetch on every scope change.** Once a chart's payload is in memory, pan/zoom/slider/range-strip all rebuild in place via the in-memory LTTB pass on the cached payload ([`src/bench/components/chart/controller.ts`](src/bench/components/chart/controller.ts)). The single exception is the latest-100 to full-history zoom-out path: when the user first zooms past the materialized window, the chart lazy-fetches `?n=all` once and replaces the payload in place. - **Group docs links mirror the monorepo's `Benchmark::doc_path`.** [`src/bench/lib/benchmark-docs.ts`](src/bench/lib/benchmark-docs.ts) maps each SQL dataset name to the benchmark's explainer doc in `vortex-data/vortex`, which is the same path that the monorepo's PR benchmark comment uses. The producer does not put `doc_path` on the wire, and upstream does not enforce a path convention, so update the explicit mapping when a suite is added or its doc moves. Groups without a matching explainer render no docs link. Do not guess a path or link to an unrelated general guide. - **Every group kind has a summary, by default.** [`src/bench/lib/summary.ts`](src/bench/lib/summary.ts) has no allowlist gate: a suite that lands in one of the six fact tables gets a rollup card from its first ingest. The three timing families (query, random access, vector search) rank through one `rankSeries` model, so a new suite in any of them needs no summary code at all. `collectGroupSummary`'s exhaustive switch makes a missing arm for a seventh fact table a compile error rather than a silently blank card. Do NOT reintroduce a per-dataset allowlist; the v2-era one is what left `spatialbench`, `fineweb`, `gharchive`, `appian`, `public-bi`, `clickbench-sorted`, and every vector-search group with no card. -- **A summary ranks the whole group, never one chart.** Random access is the cautionary case: the producer emits `dataset` as `{dataset}/{pattern}`, so the group holds ~nine charts, and the old summary published the alphabetically first chart's raw times under the group-wide title "Random Access Performance" — reporting `lance` at 352us when it is over 1ms on most of the other charts. Sum chart medians by dataset, then rank the dataset totals. Include the legacy taxi chart in the taxi total. Apply a penalty when a series lacks one dataset total. Report dataset coverage through `measured`/`total`. The secondary runtime is the arithmetic mean of complete dataset totals. +- **A summary ranks the whole group, never one chart.** Random access is the cautionary case: the producer emits `dataset` as `{dataset}/{pattern}`, so the group holds ~nine charts, and the old summary published the alphabetically first chart's raw times under the group-wide title "Random Access Performance" — reporting `lance` at 352us when it is over 1ms on most of the other charts. Sum chart medians by dataset, then rank the dataset totals. Include the legacy taxi chart in the taxi total. Apply a penalty when a series lacks one dataset total. Report dataset coverage through `measured`/`total`. The secondary runtime is the arithmetic mean of complete dataset totals. In a query summary, the penalty is not enough: a series that misses a query ranks below every series that ran them all, so a series cannot rank first by failing or skipping the hardest query (Lance skips TPC-DS q72). - **Don't write a server-side classifier for live ingest.** The emitter produces structured records directly. Classifying loose name strings at read time was the v2-era weakness every later generation existed to escape; it belongs nowhere in the live pipeline. - **Every root-relative bench URL goes through `benchPath()`.** A link, a fetch, an icon or a selector that names one of bench's paths must work on bench.vortex.dev (from the root) and everywhere else (under `/bench`); see "Repository layout". A path read from `location.pathname` already carries the base. diff --git a/docs/bench/CONTRACT.md b/docs/bench/CONTRACT.md index e27c06e..1d818eb 100644 --- a/docs/bench/CONTRACT.md +++ b/docs/bench/CONTRACT.md @@ -64,6 +64,10 @@ column-by-column; an unknown `kind` or a missing dimension field fails the inges `compression_size` records include `uncompressed_bytes`, the Arrow memory size after the source Parquet file is decoded. The column is nullable so historical rows remain valid. +`compress-bench` also writes two size-only formats, with no `compression_time` records: +`parquet-snappy` (Parquet as DuckDB writes it, with Snappy; `parquet` is zstd level 3) and +`vortex-compact` (Vortex with the compact strategy). Each is one write per dataset that is not +timed. `string-bench` shares the two compression tables with `compress-bench`. Its records use the dataset `strings/`, the string encoder (`onpair-12`, `fsst`, ...) as `dataset_variant`, @@ -73,11 +77,26 @@ rows in the `String Encoding` groups and compares each encoder with `fsst`, so t `fsst` label are part of the contract. `file_size` records are not written by `vortex-bench`: the monorepo's -`scripts/capture-file-sizes.py` writes one per Vortex file a SQL benchmark leaves on disk, with +`scripts/capture-file-sizes.py` writes one per data file a SQL benchmark leaves on disk, with `benchmark` (the `vx-bench` subcommand), `scale_factor`, `format`, `file` (the path below the format -directory) and `value_bytes`. The read service charts each benchmark and scale factor in the -`SQL File Size` group, one line per format summed over its files. CI reads them back as the develop -baseline for the PR comment's file-size section. +directory) and `value_bytes`. A file's `format` is the name of its format directory, not its +codec. `parquet` is the suite's base Parquet: zstd level 3 where the benchmarks write it, and the +dataset's published files where the suite prescribes them, as ClickBench's are Snappy. +`parquet-snappy` is Parquet as the DuckDB CLI writes it, with Snappy, from the base Parquet. CI +reads the records back as the develop baseline for the PR comment's file-size section. + +The read service charts both tables in the `File Sizes` group, one line per format. It charts every +current `compress-bench` dataset in `compression_sizes`, and three `file_sizes` scopes, each summed +over its files: ClickBench, TPC-H at SF10, and TPC-DS at SF1 (`src/bench/lib/file-sizes.ts`). Its +summary compares each format with zstd Parquet on the `compress-bench` datasets and on TPC-DS. +On develop, TPC-DS SF1 has every format `compress-bench` measures. It queries DuckDB's Snappy +Parquet (`parquet-snappy`) on DataFusion and DuckDB and Lance on DataFusion, and writes Arrow IPC after its +queries, for its size only. Lance skips q72, which it cannot run; the query summary ranks a series +that misses a query below every series that ran them all. + +The `SQL File Sizes` group charts every `file_sizes` scope, at every scale factor, and leaves out +`compression_sizes`. Its summary compares each format with `vortex-file-compressed`, per scope at +that scope's newest commit with a Vortex size. `random_access_time` records include `open_mode`. The value is `cached` for a reused accessor or `reopen` when each timed take includes a new accessor. Historical rows use `cached`. diff --git a/docs/bench/architecture/read-path.md b/docs/bench/architecture/read-path.md index c7c1afa..6589550 100644 --- a/docs/bench/architecture/read-path.md +++ b/docs/bench/architecture/read-path.md @@ -120,7 +120,7 @@ the v3 Axum server (so the frontend is generation-agnostic). The landing-page order is the curated `GROUP_ORDER` in `queries.ts`: ``` -Compression, Compression Size, Random Access, Clickbench, +Compression, File Sizes, SQL File Sizes, Random Access, Clickbench, TPC-H (NVMe/S3) SF=1, SF=10, SF=100 (alternating), TPC-DS (NVMe) SF=1, Statistical and Population Genetics, Polar Signals Profiling, diff --git a/lighthouserc.cjs b/lighthouserc.cjs index 1476e21..ea7e0ff 100644 --- a/lighthouserc.cjs +++ b/lighthouserc.cjs @@ -145,7 +145,13 @@ module.exports = { settings: { preset: "desktop", throttlingMethod: "simulate", - skipAudits: ["uses-http2"] + skipAudits: ["uses-http2"], + // Vercel adds its Toolbar to preview deployments, even with the + // proxy's `x-vercel-skip-toolbar` header. It loads from vercel.live + // (and Pusher, for live comments), and its third-party cookies and + // issues cut Best Practices to 0.78 on every page from 2026-11-20. + // The site never loads either host, so block them. + blockedUrlPatterns: ["*vercel.live*", "*pusher.com*"] } }, assert: { diff --git a/src/bench/components/SummaryCard.test.tsx b/src/bench/components/SummaryCard.test.tsx index da04c1d..08c851c 100644 --- a/src/bench/components/SummaryCard.test.tsx +++ b/src/bench/components/SummaryCard.test.tsx @@ -152,6 +152,8 @@ describe('SummaryCard', () => { maxRatio: 1, compressionRatio: 4, totalBytes: null, + measured: 1, + total: 1, }, ], explanation: 'e', @@ -160,6 +162,30 @@ describe('SummaryCard', () => { expect(html).toContain('class="compression-size-heading compression-size-total-unit">'); }); + it('says how many datasets a partly measured size ranking covers', () => { + const ranking = (name: string, measured: number) => ({ + name, + minRatio: 1, + ratio: 1, + maxRatio: 1, + compressionRatio: null, + totalBytes: measured === 2 ? 10 : null, + measured, + total: 2, + }); + const html = render({ + type: 'compressionSize', + title: 'File Sizes Summary', + rankings: [ranking('parquet', 2), ranking('vortex-compact', 1)], + explanation: 'e', + }); + expect(html).toContain('title="vortex-compact · measured on 1 of 2 datasets"'); + expect(html).toContain(''); + expect(html).toContain(', measured on 1 of 2 datasets'); + // A complete ranking carries no coverage. + expect(html.match(/measured on/g)).toHaveLength(2); + }); + it('renders nothing for a compression card with no rankings', () => { expect(render({ type: 'compression', title: 't', rankings: [], explanation: 'e' })).toBe(''); }); @@ -176,6 +202,8 @@ describe('SummaryCard', () => { maxRatio: 1.55, compressionRatio: 8.25, totalBytes: 1_603_980_000, + measured: 1, + total: 1, }, { name: 'parquet', @@ -184,6 +212,8 @@ describe('SummaryCard', () => { maxRatio: 1, compressionRatio: 4.5, totalBytes: 1_428_150_000, + measured: 1, + total: 1, }, { name: 'lance', @@ -192,6 +222,8 @@ describe('SummaryCard', () => { maxRatio: 1.3, compressionRatio: null, totalBytes: 2_500_000, + measured: 1, + total: 1, }, ], explanation: 'higher is better', @@ -311,6 +343,8 @@ describe('SummaryCard', () => { maxRatio: 1, compressionRatio: 8, totalBytes: 100, + measured: 1, + total: 1, }, { name: 'lance', @@ -319,6 +353,8 @@ describe('SummaryCard', () => { maxRatio: 1, compressionRatio: 2, totalBytes: 200, + measured: 1, + total: 1, }, ], explanation: 'e', diff --git a/src/bench/components/SummaryCard.tsx b/src/bench/components/SummaryCard.tsx index 0ad86a9..5679791 100644 --- a/src/bench/components/SummaryCard.tsx +++ b/src/bench/components/SummaryCard.tsx @@ -144,14 +144,27 @@ export function SummaryCard({ summary }: { summary?: Summary | undefined }) {
{summary.rankings.map((item, idx) => { const label = displayFormat(item.name); + // A format measured on only some datasets has its means over those alone, so + // the row says how many, and it ranks after the complete ones. + const partial = item.measured < item.total; + const coverage = `measured on ${item.measured} of ${item.total} datasets`; return (
#{idx + 1} - + {label} + {partial ? ( + <> + + {`, ${coverage}`} + + ) : null} {item.totalBytes === null || totalUnit === null ? ( diff --git a/src/bench/lib/chart-format.test.ts b/src/bench/lib/chart-format.test.ts index 18c4d7d..44343c6 100644 --- a/src/bench/lib/chart-format.test.ts +++ b/src/bench/lib/chart-format.test.ts @@ -139,6 +139,30 @@ describe('seriesStyle', () => { } }); + it("draws a Parquet codec variant dotted, in Parquet's color, thinner, behind Parquet", () => { + for (const theme of ['dark', 'light'] as const) { + const zstd = seriesStyle('parquet', { format: 'parquet' }, theme); + const snappy = seriesStyle('parquet-snappy', { format: 'parquet-snappy' }, theme); + expect(snappy.color).toBe(zstd.color); + expect(snappy.dash).toEqual([2, 3]); + expect(snappy.width).toBeLessThan(zstd.width); + } + const order = (format: string) => seriesOrder(format, { format }); + expect(order('parquet')).toBeLessThan(order('parquet-snappy')); + expect(order('parquet-snappy')).toBeLessThan(order('vortex-compact')); + }); + + it("keeps a query chart's four Parquet lines apart", () => { + const styles = ['parquet', 'parquet-snappy'].flatMap((format) => + ['datafusion', 'duckdb'].map((engine) => + seriesStyle(`${engine}:${format}`, { engine, format }, 'dark'), + ), + ); + const looks = new Set(styles.map((style) => `${style.width}/${style.dash.join(',')}`)); + expect(looks.size).toBe(4); + expect(seriesStyle('duckdb:parquet-snappy', null, 'dark').dash).toEqual([6, 3, 2, 3]); + }); + it('draws a cold random-access run dashed and a hot one solid', () => { const cold = seriesStyle('parquet:cold', { format: 'parquet' }, 'dark'); const hot = seriesStyle('parquet:hot', { format: 'parquet' }, 'dark'); diff --git a/src/bench/lib/chart-format.ts b/src/bench/lib/chart-format.ts index 0702063..3d42b51 100644 --- a/src/bench/lib/chart-format.ts +++ b/src/bench/lib/chart-format.ts @@ -85,7 +85,8 @@ export const MAX_VISIBLE_POINTS = 500; // purple, DuckDB green, and Arrow orange. The color means the format and // nothing else, so a format looks the same on every engine: the line style // shows the engine instead. A duckdb run is dashed, as is a cold random-access -// run (the two never share a chart). Widths carry the importance tiers. +// run (the two never share a chart). A Parquet codec variant is dotted, or +// dash-dotted on duckdb. Widths carry the importance tiers. // // The palette has one set of colors per card theme, because one set cannot // stay legible on both cards: a color with 3:1 contrast against both the @@ -113,6 +114,10 @@ export interface SeriesStyle { /** The dash of a duckdb run and of a cold random-access run. */ const DASH: readonly number[] = [6, 4]; +/** The dash of a Parquet codec variant, which a query chart can hold on both + * engines beside zstd Parquet: dotted, and dash-dotted on duckdb. */ +const DOT: readonly number[] = [2, 3]; +const DASH_DOT: readonly number[] = [6, 3, 2, 3]; /** Line widths per importance tier, reinforcing the color hierarchy. */ const HERO_WIDTH = 2.6; @@ -160,6 +165,17 @@ const FORMAT: Record = { 'arrow-ipc': ARROW, }; +/** + * The format a codec variant is drawn as: Parquet with a codec other than zstd + * (`parquet-snappy`, `parquet-uncompressed`, as the size tables and TPC-DS's + * queries name them) is Parquet's teal, thinner and dotted, so it reads as a + * kind of Parquet without a color of its own. Its dots differ from duckdb's + * dash, so a query chart's four Parquet lines stay apart. + */ +function codecVariantOf(format: string): string | undefined { + return format.startsWith('parquet-') ? 'parquet' : undefined; +} + /** The catch-all style for an otherwise-unknown series: a neutral gray with * 3:1 contrast on both cards. */ const FALLBACK_COLOR = '#8a8a87'; @@ -190,12 +206,24 @@ export function seriesStyle( theme: ChartTheme, ): SeriesStyle { const { engine, format } = seriesDims(name, meta); - const dash = engine === 'duckdb' || name.endsWith(':cold') ? [...DASH] : []; - const style = format ? FORMAT[format] : undefined; + const variantOf = format === undefined ? undefined : codecVariantOf(format); + const duckdbOrCold = engine === 'duckdb' || name.endsWith(':cold'); + const dash = + variantOf !== undefined + ? [...(engine === 'duckdb' ? DASH_DOT : DOT)] + : duckdbOrCold + ? [...DASH] + : []; + const base = variantOf ?? format; + const style = base ? FORMAT[base] : undefined; if (!style) { return { color: FALLBACK_COLOR, width: MUTED_WIDTH, dash }; } - return { color: style.color[theme], width: style.width, dash }; + return { + color: style.color[theme], + width: variantOf === undefined ? style.width : MUTED_WIDTH, + dash, + }; } /** Format draw priority, front (low) to back (high) -- Chart.js renders a lower @@ -231,7 +259,12 @@ export function seriesOrder( meta: { engine?: string; format?: string } | null | undefined, ): number { const { engine, format } = seriesDims(name, meta); - const formatRank = rankIn(FORMAT_ORDER, format, FORMAT_ORDER_FALLBACK); + // A codec variant sits just behind its format. + const variantOf = format === undefined ? undefined : codecVariantOf(format); + const formatRank = + variantOf === undefined + ? rankIn(FORMAT_ORDER, format, FORMAT_ORDER_FALLBACK) + : rankIn(FORMAT_ORDER, variantOf, FORMAT_ORDER_FALLBACK) + 0.5; const engineRank = rankIn(ENGINE_ORDER, engine, 0); return formatRank * 10 + engineRank; } diff --git a/src/bench/lib/compression-suite.ts b/src/bench/lib/compression-suite.ts index e9919e4..66a6082 100644 --- a/src/bench/lib/compression-suite.ts +++ b/src/bench/lib/compression-suite.ts @@ -15,10 +15,11 @@ export const STRING_DATASET_PREFIX = 'strings/'; export const STRING_DATASET_PATTERN = `${STRING_DATASET_PREFIX}%`; /** - * `LIKE` patterns for the `compress-bench` datasets the compression summaries leave out: the + * `LIKE` patterns for the `compress-bench` datasets the compression time summary leaves out: the * `wide table cols=… chunks=… rows=…` and `sparse-wide-*` suites. They are pathological cases * with many variants each, so in a geomean over all datasets they would be heavily - * over-represented. They still chart in the group. Both patterns are anchored at the start of the + * over-represented. They still chart in the group. The File Sizes summary counts each suite as + * one dataset instead (`SIZE_SUMMARY_FAMILIES`). Both patterns are anchored at the start of the * name, as `compress-bench` writes it, so a dataset that only mentions them is not dropped. */ export const SUMMARY_EXCLUDED_DATASET_PATTERNS: readonly string[] = [ @@ -27,25 +28,38 @@ export const SUMMARY_EXCLUDED_DATASET_PATTERNS: readonly string[] = [ ]; /** - * `LIKE` patterns for the datasets the compression size summary also leaves out, on top of - * `SUMMARY_EXCLUDED_DATASET_PATTERNS`. They still chart in the group, and the compression time - * summary keeps them. + * `LIKE` patterns for the `compress-bench` datasets the File Sizes summary leaves out. They still + * chart in the group, and the compression time summary keeps them (minus + * `SUMMARY_EXCLUDED_DATASET_PATTERNS`). * * - `tpc-h l_comment chunked` writes the same column as `tpc-h l_comment canonical`, so their * files are the same size. Only the input to the encoder differs, which matters for time but * not for size. The size summary counts the column once, as `canonical`. + * - The wide tables with 50 chunks write the same files as the ones with 1 chunk, for the same + * reason. The size summary counts each column count once, as `chunks=1`. * - `statpopgen`, which joined `compress-bench` on 2026-11-06, is left out of the size summary. * The pattern also matches the older `statpopgen/100000` rows, which have no Parquet size and * so were never in the summary. * - * Both names are exact. `_` is a `LIKE` wildcard, but no other dataset name matches. + * `_` is a `LIKE` wildcard, but no other dataset name matches these patterns. */ export const SIZE_SUMMARY_EXCLUDED_DATASET_PATTERNS: readonly string[] = [ - ...SUMMARY_EXCLUDED_DATASET_PATTERNS, 'tpc-h l_comment chunked', + 'wide table cols=% chunks=50 rows=%', 'statpopgen', ]; +/** + * Synthetic `compress-bench` families that the File Sizes summary counts as one dataset each: + * the geometric mean of the family's ratios is one entry in the summary's geometric mean. A + * family has several datasets of one kind, so counted one by one they would outweigh the real + * datasets. Each family is the datasets whose names start with its prefix. + */ +export const SIZE_SUMMARY_FAMILIES: readonly { name: string; prefix: string }[] = [ + { name: 'wide table', prefix: 'wide table ' }, + { name: 'sparse-wide', prefix: 'sparse-wide-' }, +]; + export function compressionSuite(dataset: string): CompressionSuite { return dataset.startsWith(STRING_DATASET_PREFIX) ? 'strings' : 'compress'; } diff --git a/src/bench/lib/descriptions.test.ts b/src/bench/lib/descriptions.test.ts index 7cba09b..06d66a5 100644 --- a/src/bench/lib/descriptions.test.ts +++ b/src/bench/lib/descriptions.test.ts @@ -20,8 +20,14 @@ describe('groupDescription', () => { expect(groupDescription('Compression')).toBe( 'Measures write and scan throughput for Vortex, Parquet, and Lance files', ); - expect(groupDescription('Compression Size')).toBe( - 'Compares compressed file sizes and compression ratios across different encoding strategies', + expect(groupDescription('File Sizes')).toBe( + 'Compares file sizes and compression ratios across formats against zstd Parquet, on the ' + + 'compression datasets and on one scale factor each of ClickBench, TPC-H, and TPC-DS', + ); + expect(groupDescription('SQL File Sizes')).toBe( + 'Tracks the size of the files each SQL benchmark writes, at every scale factor, against ' + + 'Vortex. Unlike File Sizes, it leaves out the compression datasets and does not rank ' + + 'formats against Parquet', ); expect(groupDescription('Clickbench')).toBe( "ClickHouse's analytical benchmark suite testing real-world query patterns on web " + diff --git a/src/bench/lib/descriptions.ts b/src/bench/lib/descriptions.ts index 2aaa6de..5fa0958 100644 --- a/src/bench/lib/descriptions.ts +++ b/src/bench/lib/descriptions.ts @@ -46,20 +46,21 @@ function staticDescription(name: string): string | null { return 'Tests performance of selecting arbitrary row indices from a file on S3 storage'; case 'Compression': return 'Measures write and scan throughput for Vortex, Parquet, and Lance files'; - case 'Compression Size': + case 'File Sizes': return ( - 'Compares compressed file sizes and compression ratios across different encoding ' + - 'strategies' + 'Compares file sizes and compression ratios across formats against zstd Parquet, on the ' + + 'compression datasets and on one scale factor each of ClickBench, TPC-H, and TPC-DS' + ); + case 'SQL File Sizes': + return ( + 'Tracks the size of the files each SQL benchmark writes, at every scale factor, against ' + + 'Vortex. Unlike File Sizes, it leaves out the compression datasets and does not rank ' + + 'formats against Parquet' ); case 'String Encoding': return 'Measures Vortex file write and scan throughput for each string encoder'; case 'String Encoding Size': return 'Compares Vortex file sizes for each string encoder'; - case 'SQL File Size': - return ( - 'Tracks the total size of the Vortex files each SQL benchmark writes, per benchmark and ' + - 'scale factor' - ); case 'Clickbench': return ( "ClickHouse's analytical benchmark suite testing real-world query patterns on web " + diff --git a/src/bench/lib/file-sizes.ts b/src/bench/lib/file-sizes.ts new file mode 100644 index 0000000..302f346 --- /dev/null +++ b/src/bench/lib/file-sizes.ts @@ -0,0 +1,92 @@ +// SPDX-License-Identifier: Apache-2.0 +// SPDX-FileCopyrightText: Copyright the Vortex contributors + +/** + * Which charts the File Sizes group holds. It joins two tables: `compression_sizes`, the + * size of each `compress-bench` dataset in each format, and `file_sizes`, the files that the + * SQL suites write. Both are one dataset's size per format per commit. + */ + +import { compressionSuite } from './compression-suite'; + +/** The File Sizes group's display name. */ +export const FILE_SIZES_GROUP_NAME = 'File Sizes'; + +/** + * The SQL File Sizes group's display name. That group charts every SQL benchmark and scale factor + * in `file_sizes`, and its summary compares the formats with Vortex. + */ +export const SQL_FILE_SIZES_GROUP_NAME = 'SQL File Sizes'; + +/** One SQL suite's data at one scale factor, as `file_sizes` spells them. */ +export interface FileSizeScope { + benchmark: string; + scale_factor: string; +} + +/** + * The SQL suites' data the group charts, in this order. The other scopes in `file_sizes` stay + * in the table, for the PR benchmark comments, but the group leaves them out. TPC-DS is at + * SF1 because develop writes it on every commit, with Vortex Compact and DuckDB; it writes SF10 + * only nightly, with Parquet and Vortex alone. + */ +export const FILE_SIZE_SCOPES: readonly FileSizeScope[] = [ + { benchmark: 'clickbench', scale_factor: '1.0' }, + { benchmark: 'tpch', scale_factor: '10.0' }, + { benchmark: 'tpcds', scale_factor: '1.0' }, +]; + +/** + * The SQL scopes the summary ranks: TPC-DS alone. + * + * - ClickBench measures the Parquet files it downloads, which are not zstd, so it has no + * baseline to compare with. + * - TPC-H comes from the same generator as the `compress-bench` TPC-H tables (lineitem, orders, + * and partsupp), which the summary already ranks. Ranking both would count TPC-H twice. + */ +export const FILE_SIZE_SUMMARY_SCOPES: readonly FileSizeScope[] = [ + { benchmark: 'tpcds', scale_factor: '1.0' }, +]; + +/** Whether the group charts this SQL scope. */ +export function isFileSizeScope(benchmark: string, scaleFactor: string): boolean { + return FILE_SIZE_SCOPES.some( + (scope) => scope.benchmark === benchmark && scope.scale_factor === scaleFactor, + ); +} + +/** + * `compression_sizes` charts the group leaves out. Each entry is a `dataset` and + * `dataset_variant`, exactly. + * + * - The rows that the SQL suites wrote before `file_sizes` existed: Vortex sizes only, the last + * on 2026-05-28. `file_sizes` holds those suites' sizes now. + * - The 10-column wide tables, which `compress-bench` stopped writing on 2025-10-07. + * + * None of them has a Parquet size, so the summary, which pairs every size with Parquet's, never + * counts them either. + */ +const HIDDEN_SIZE_DATASETS: readonly (readonly [string, string | null])[] = [ + ['appian', null], + ['clickbench', null], + ['fineweb', null], + ['polarsignals', '1000000'], + ['statpopgen', '100000'], + ['tpcds', null], + ['tpch', null], + ['tpch', '10'], + ['tpch', '100'], + ['wide table cols=10 chunks=1 rows=1000', null], + ['wide table cols=10 chunks=50 rows=1000', null], +]; + +/** Whether the group charts this `compression_sizes` dataset: a current `compress-bench` one. */ +export function isFileSizesDataset(dataset: string, datasetVariant: string | null): boolean { + return ( + compressionSuite(dataset) === 'compress' && + !HIDDEN_SIZE_DATASETS.some( + ([hiddenDataset, hiddenVariant]) => + hiddenDataset === dataset && hiddenVariant === datasetVariant, + ) + ); +} diff --git a/src/bench/lib/group-charts.test.ts b/src/bench/lib/group-charts.test.ts index 519c21e..c8a6ead 100644 --- a/src/bench/lib/group-charts.test.ts +++ b/src/bench/lib/group-charts.test.ts @@ -473,27 +473,27 @@ describe.skipIf(!databaseAvailable)('groupChartPayloads at the window edges', () [measurementId, sha, dataset, variant, valueBytes], ); } - // File sizes: TPC-H SF=1 has two files per format on C0, E and F (compact only from E); - // SF=10 has one file on F. - for (const [sha, scaleFactor, format, file, valueBytes] of [ - [C0, '1.0', 'vortex-file-compressed', 'a.vortex', 100], - [C0, '1.0', 'vortex-file-compressed', 'b.vortex', 10], - [E, '1.0', 'vortex-file-compressed', 'a.vortex', 101], - [E, '1.0', 'vortex-file-compressed', 'b.vortex', 11], - [E, '1.0', 'vortex-compact', 'a.vortex', 80], - [E, '1.0', 'vortex-compact', 'b.vortex', 8], - [F, '1.0', 'vortex-file-compressed', 'a.vortex', 102], - [F, '1.0', 'vortex-file-compressed', 'b.vortex', 12], - [F, '1.0', 'vortex-compact', 'a.vortex', 79], - [F, '1.0', 'vortex-compact', 'b.vortex', 7], - [F, '10.0', 'vortex-file-compressed', 'a.vortex', 1000], + // File sizes: TPC-DS SF=1 has two files per format on C0, E and F (compact only from E); + // TPC-H SF=10 has one file on F. + for (const [sha, benchmark, scaleFactor, format, file, valueBytes] of [ + [C0, 'tpcds', '1.0', 'vortex-file-compressed', 'a.vortex', 100], + [C0, 'tpcds', '1.0', 'vortex-file-compressed', 'b.vortex', 10], + [E, 'tpcds', '1.0', 'vortex-file-compressed', 'a.vortex', 101], + [E, 'tpcds', '1.0', 'vortex-file-compressed', 'b.vortex', 11], + [E, 'tpcds', '1.0', 'vortex-compact', 'a.vortex', 80], + [E, 'tpcds', '1.0', 'vortex-compact', 'b.vortex', 8], + [F, 'tpcds', '1.0', 'vortex-file-compressed', 'a.vortex', 102], + [F, 'tpcds', '1.0', 'vortex-file-compressed', 'b.vortex', 12], + [F, 'tpcds', '1.0', 'vortex-compact', 'a.vortex', 79], + [F, 'tpcds', '1.0', 'vortex-compact', 'b.vortex', 7], + [F, 'tpch', '10.0', 'vortex-file-compressed', 'a.vortex', 1000], ] as const) { measurementId += 1; await getPool().query( `INSERT INTO file_sizes (measurement_id, commit_sha, benchmark, scale_factor, format, file, value_bytes) - VALUES ($1, $2, 'tpch', $3, $4, $5, $6)`, - [measurementId, sha, scaleFactor, format, file, valueBytes], + VALUES ($1, $2, $3, $4, $5, $6, $7)`, + [measurementId, sha, benchmark, scaleFactor, format, file, valueBytes], ); } }); @@ -584,24 +584,29 @@ describe.skipIf(!databaseAvailable)('groupChartPayloads at the window edges', () })), ], [ + // The File Sizes group mixes the two tables: `compression_sizes` charts, a NULL variant + // between two non-NULL ones, and `file_sizes` charts. { k: 'CompressionSizeGroup' }, [ + { k: 'FileSize', benchmark: 'tpcds', scale_factor: '1.0' }, { k: 'CompressionSize', dataset: 'sx', dataset_variant: 'v1' }, { k: 'CompressionSize', dataset: 'sx', dataset_variant: null }, + { k: 'FileSize', benchmark: 'tpch', scale_factor: '10.0' }, { k: 'CompressionSize', dataset: 'sy', dataset_variant: 'v2' }, ], ], [ + // The SQL File Sizes group holds `file_sizes` charts alone, at every scale factor. { k: 'FileSizeGroup' }, [ - { k: 'FileSize', benchmark: 'tpch', scale_factor: '1.0' }, { k: 'FileSize', benchmark: 'tpch', scale_factor: '10.0' }, + { k: 'FileSize', benchmark: 'tpcds', scale_factor: '1.0' }, ], ], ]; it.each(['1', '2', '3', '4', 'all'])( - 'matches chartPayload for random-access, vector-search, compression-size and file-size groups at ?n=%s', + 'matches chartPayload for random-access, vector-search, and file-size groups at ?n=%s', async (n) => { const window = commitWindow(n); for (const [group, keys] of OTHER_FAMILIES) { @@ -644,11 +649,11 @@ describe.skipIf(!databaseAvailable)('groupChartPayloads at the window edges', () }); it('sums each file-size chart over its files, one line per format', async () => { - const [sf1, sf10] = await groupChartPayloads( - ...defined(OTHER_FAMILIES[3]), + const [sf1, , , sf10] = await groupChartPayloads( + ...defined(OTHER_FAMILIES[2]), parseCommitWindow('all'), ); - expect(sf1?.display_name).toBe('tpch sf=1.0'); + expect(sf1?.display_name).toBe('tpcds sf=1.0'); expect(sf1?.unit_kind).toBe('bytes'); expect(sf1?.series).toEqual({ 'vortex-compact': [null, null, null, null, 88, 86], diff --git a/src/bench/lib/groups.test.ts b/src/bench/lib/groups.test.ts index 7d3784c..ce996dd 100644 --- a/src/bench/lib/groups.test.ts +++ b/src/bench/lib/groups.test.ts @@ -74,7 +74,7 @@ describe.skipIf(!databaseAvailable)( // Random-access sections precede Clickbench in GROUP_ORDER. expect(groups.map((g) => g.name)).toEqual([ 'Compression', - 'Compression Size', + 'File Sizes', 'Random Access (NVMe)', 'TPC-H (NVMe) (SF=1)', 'cohere-large-10m / partitioned', @@ -175,7 +175,7 @@ describe.skipIf(!databaseAvailable)( it('computes compression-size rankings for Vortex, Parquet, Lance, and Arrow IPC', async () => { const groups = await collectGroups(); const summary = expectDefined( - groups.find((g) => g.name === 'Compression Size')?.summary, + groups.find((g) => g.name === 'File Sizes')?.summary, 'compression size summary', ); if (summary.type !== 'compressionSize') { @@ -251,7 +251,7 @@ describe.skipIf(!databaseAvailable)( const body = (await res.json()) as GroupsResponse; expect(body.groups.map((g) => g.name)).toEqual([ 'Compression', - 'Compression Size', + 'File Sizes', 'Random Access (NVMe)', 'TPC-H (NVMe) (SF=1)', 'cohere-large-10m / partitioned', @@ -566,7 +566,7 @@ describe.skipIf(!databaseAvailable)('summary math fidelity (testcontainers Postg const groups = await collectGroups(); expect(groups.map((group) => group.name)).toEqual([ 'Compression', - 'Compression Size', + 'File Sizes', 'String Encoding', 'String Encoding Size', ]); @@ -626,23 +626,29 @@ describe.skipIf(!databaseAvailable)('summary math fidelity (testcontainers Postg ]); }); - it('discovers one file-size chart per benchmark and scale factor, with a summary', async () => { + it('charts the listed SQL scopes, and ranks TPC-DS against zstd Parquet', async () => { const develop = 'a'.repeat(40); const nightly = 'b'.repeat(40); await insertCommit(develop, '2026-04-23T12:00:00Z'); await insertCommit(nightly, '2026-04-24T03:00:00Z'); - // TPC-H at SF 1 and 10 from develop, and SF 100 from nightly, the newest commit. Clickbench - // has no scale-factor directory, so it is recorded as 1.0. + // The group charts ClickBench, TPC-H SF10, and TPC-DS SF1. It leaves out TPC-H SF1 and + // SF100 and Appian. The newest commit, a nightly run, writes Parquet and Snappy Parquet + // for TPC-DS but no Vortex. ClickBench has no scale-factor directory, so it is recorded as + // 1.0. for (const [sha, benchmark, scaleFactor, format, file, bytes] of [ [develop, 'tpch', '1.0', 'vortex-file-compressed', 'lineitem.vortex', 300], - [develop, 'tpch', '1.0', 'vortex-file-compressed', 'orders.vortex', 100], - [develop, 'tpch', '1.0', 'vortex-compact', 'lineitem.vortex', 150], - [develop, 'tpch', '1.0', 'vortex-compact', 'orders.vortex', 50], [develop, 'tpch', '10.0', 'vortex-file-compressed', 'lineitem.vortex', 4000], - [develop, 'tpch', '10.0', 'vortex-compact', 'lineitem.vortex', 1000], + [develop, 'tpch', '10.0', 'parquet', 'lineitem.parquet', 2000], [nightly, 'tpch', '100', 'vortex-file-compressed', 'lineitem.vortex', 40000], [develop, 'clickbench', '1.0', 'vortex-file-compressed', 'hits_0.vortex', 1000], - [develop, 'tpcds', '1.0', 'vortex-file-compressed', 'store_sales.vortex', 500], + [develop, 'tpcds', '1.0', 'vortex-file-compressed', 'store_sales.vortex', 300], + [develop, 'tpcds', '1.0', 'vortex-file-compressed', 'item.vortex', 100], + [develop, 'tpcds', '1.0', 'vortex-compact', 'store_sales.vortex', 150], + [develop, 'tpcds', '1.0', 'vortex-compact', 'item.vortex', 50], + [develop, 'tpcds', '1.0', 'parquet', 'store_sales.parquet', 350], + [develop, 'tpcds', '1.0', 'parquet', 'item.parquet', 50], + [nightly, 'tpcds', '1.0', 'parquet', 'store_sales.parquet', 400], + [nightly, 'tpcds', '1.0', 'parquet-snappy', 'store_sales.parquet', 600], [develop, 'appian', '1.0', 'vortex-file-compressed', 'data.vortex', 70], ] as const) { await getPool().query( @@ -654,40 +660,63 @@ describe.skipIf(!databaseAvailable)('summary math fidelity (testcontainers Postg } const groups = await collectGroups(); - expect(groups.map((group) => group.name)).toEqual(['SQL File Size']); - const [group] = groups; - expect(groupKeyFromSlug(defined(group).slug)).toEqual({ k: 'FileSizeGroup' }); - // Clickbench, TPC-H and TPC-DS lead; the rest follow by name. TPC-H always names its scale - // factor, while Clickbench and Appian have none. + expect(groups.map((group) => group.name)).toEqual(['File Sizes', 'SQL File Sizes']); + const [group, sql] = groups; + expect(groupKeyFromSlug(defined(group).slug)).toEqual({ k: 'CompressionSizeGroup' }); expect(defined(group).charts.map((chart) => chart.name)).toEqual([ 'clickbench', - 'tpch sf=1.0', 'tpch sf=10.0', - 'tpch sf=100', 'tpcds sf=1.0', - 'appian', ]); - expect(chartKeyFromSlug(defined(defined(group).charts[3]).slug)).toEqual({ + expect(chartKeyFromSlug(defined(defined(group).charts[2]).slug)).toEqual({ k: 'FileSize', - benchmark: 'tpch', - scale_factor: '100', + benchmark: 'tpcds', + scale_factor: '1.0', }); - // Compact is half the size at SF 1 and a quarter at SF 10. Totals cover the scopes both - // formats have: 200 + 1000 against 400 + 4000. - expect(defined(group).summary).toMatchObject({ + // Only TPC-DS is ranked. Each format uses its own newest commit that has zstd Parquet: + // Snappy the nightly one, and the Vortex formats the develop one. + const summary = defined(group).summary; + expect(summary).toMatchObject({ type: 'compressionSize', - baseline: 'vortex', + title: 'File Sizes Summary', rankings: [ - { - name: 'vortex-compact', - minRatio: 0.25, - ratio: expect.closeTo(0.5 ** 1.5), - maxRatio: 0.5, - totalBytes: 1200, - }, - { name: 'vortex-file-compressed', ratio: 1, compressionRatio: null, totalBytes: 4400 }, + { name: 'vortex-compact', ratio: 0.5, totalBytes: 200, measured: 1, total: 1 }, + { name: 'parquet', ratio: 1, totalBytes: 400, measured: 1, total: 1 }, + { name: 'vortex-file-compressed', ratio: 1, compressionRatio: null, totalBytes: 400 }, + { name: 'parquet-snappy', ratio: 1.5, totalBytes: 600, measured: 1, total: 1 }, ], }); + expect(summary).not.toHaveProperty('baseline'); + + // The SQL File Sizes group charts every scope, and ranks each format against Vortex at each + // scope's newest commit with a Vortex size. TPC-DS's newest commit with Vortex is the + // develop one, so the nightly Snappy file is not in its totals. + expect(groupKeyFromSlug(defined(sql).slug)).toEqual({ k: 'FileSizeGroup' }); + expect(defined(sql).charts.map((chart) => chart.name)).toEqual([ + 'clickbench', + 'tpch sf=1.0', + 'tpch sf=10.0', + 'tpch sf=100', + 'tpcds sf=1.0', + 'appian', + ]); + expect(defined(sql).summary).toMatchObject({ + type: 'compressionSize', + title: 'SQL File Sizes Summary', + baseline: 'vortex', + }); + const sqlSummary = defined(defined(sql).summary); + if (sqlSummary.type !== 'compressionSize') { + throw new Error(`expected a compressionSize summary, got ${sqlSummary.type}`); + } + expect( + Object.fromEntries(sqlSummary.rankings.map((ranking) => [ranking.name, ranking.ratio])), + ).toEqual({ + 'vortex-file-compressed': 1, + 'vortex-compact': 0.5, + // TPC-H SF10's Parquet is half its Vortex size, and TPC-DS's is the same size. + parquet: expect.closeTo(Math.sqrt(0.5 * 1), 12), + }); }); it('ranks random access over every chart, not the alphabetically first one', async () => { @@ -869,14 +898,20 @@ describe.skipIf(!databaseAvailable)('summary math fidelity (testcontainers Postg expect(defined(size.rankings[1]).ratio).toBeCloseTo(1.0, 6); }); - it('leaves the wide-table and sparse-wide datasets out of the compression summaries', async () => { + it('leaves the synthetic datasets out of the time summary, and counts each family once in the size summary', async () => { const sha = 'e'.repeat(40); await insertCommit(sha, '2026-04-23T12:00:00Z'); await insertCompTimePair(sha, 'encode', 1_000, 3_000); await insertCompSizePair(sha, 1_000, 4_000); - for (const dataset of ['wide table cols=1000 chunks=50 rows=1000', 'sparse-wide-nested-runs']) { + for (const [dataset, vortexBytes, parquetBytes] of [ + ['wide table cols=1000 chunks=1 rows=1000', 9_000, 1_000], + // The same file as `chunks=1`, so the size summary leaves it out. + ['wide table cols=1000 chunks=50 rows=1000', 1, 1_000], + ['sparse-wide-nested-runs', 1_000, 4_000], + ['sparse-wide-flat-runs', 4_000, 1_000], + ] as const) { await insertCompTimePair(sha, 'encode', 9_000, 1_000, dataset); - await insertCompSizePair(sha, 9_000, 1_000, dataset); + await insertCompSizePair(sha, vortexBytes, parquetBytes, dataset); } // The patterns match only at the start of the name, so this dataset stays in. await insertCompTimePair(sha, 'encode', 1_000, 3_000, 'taxi-sparse-wide'); @@ -899,18 +934,22 @@ describe.skipIf(!databaseAvailable)('summary math fidelity (testcontainers Postg const size = expectDefined( await collectGroupSummary({ k: 'CompressionSizeGroup' }), - 'compression-size summary', + 'file size summary', ); if (size.type !== 'compressionSize') { throw new Error(`expected compressionSize summary, got ${size.type}`); } const vortexSize = expectDefined( size.rankings.find((ranking) => ranking.name === 'vortex-file-compressed'), - 'Vortex compression-size ranking', + 'Vortex file size ranking', ); + // Four entries: tpch-lineitem 0.25, taxi-sparse-wide 1, the wide tables 9, and the + // sparse-wide family sqrt(0.25 * 4) = 1. + expect(vortexSize.ratio).toBeCloseTo(2.25 ** 0.25, 6); + expect([vortexSize.measured, vortexSize.total]).toEqual([4, 4]); expect(vortexSize.minRatio).toBeCloseTo(0.25, 6); - expect(vortexSize.ratio).toBeCloseTo(0.5, 6); - expect(vortexSize.maxRatio).toBeCloseTo(1.0, 6); + expect(vortexSize.maxRatio).toBeCloseTo(9, 6); + expect(vortexSize.totalBytes).toBe(1_000 + 1_000 + 9_000 + 1_000 + 4_000); }); it('selects one commit when the latest timestamps collide', async () => { diff --git a/src/bench/lib/headline.test.ts b/src/bench/lib/headline.test.ts index e989764..1f44615 100644 --- a/src/bench/lib/headline.test.ts +++ b/src/bench/lib/headline.test.ts @@ -186,7 +186,16 @@ describe('groupHeadline', () => { title: 's', explanation: 'e', rankings: [ - { name: VX, minRatio: ratio, ratio, maxRatio: ratio, compressionRatio: 5, totalBytes: 1 }, + { + name: VX, + minRatio: ratio, + ratio, + maxRatio: ratio, + compressionRatio: 5, + totalBytes: 1, + measured: 1, + total: 1, + }, ], }); expect(h && headlineText(h)).toBe(expected); @@ -202,7 +211,16 @@ describe('groupHeadline', () => { explanation: 'e', baseline: 'vortex', rankings: [ - { name: VX, minRatio: 1, ratio: 1, maxRatio: 1, compressionRatio: null, totalBytes: 1 }, + { + name: VX, + minRatio: 1, + ratio: 1, + maxRatio: 1, + compressionRatio: null, + totalBytes: 1, + measured: 1, + total: 1, + }, ], }), ).toBeNull(); diff --git a/src/bench/lib/methodology.ts b/src/bench/lib/methodology.ts index 8649757..5598d92 100644 --- a/src/bench/lib/methodology.ts +++ b/src/bench/lib/methodology.ts @@ -11,8 +11,11 @@ * `bench-orchestrator/bench_orchestrator/ci_matrix/catalog.py`, the runners in * `bench-orchestrator/bench_orchestrator/cli.py`, `scripts/random-access-split.py` * and `scripts/compress-split.py`, `scripts/setup-benchmark.sh`, and the versions - * in `Cargo.toml`, `Cargo.lock` and `vortex-duckdb/build.rs`. Update this file - * when those change. + * in `Cargo.toml`, `Cargo.lock` and `vortex-duckdb/build.rs`. The Parquet + * variants come from vortex-data/vortex#11720: `parquet_writer_properties()` in + * `vortex-bench/src/utils/parquet.rs`, `vortex-bench/src/parquet_copies.rs`, and + * the DuckDB version in `.github/actions/setup-duckdb/action.yml`. Update this + * file when those change. * The scoring is this site's own (`summary.ts`). */ @@ -68,9 +71,17 @@ export const METHODOLOGY: readonly MethodologySection[] = [ text: [ 'Queries run on DataFusion 55 and DuckDB 2.0. The formats are Vortex (its default ' + 'and compact write strategies), Apache Parquet, Lance 13, Arrow IPC, DuckDB’s own ' + - 'format, and, for random access, LitData. Not every suite runs every format. The ' + - 'compression and file size benchmarks write Parquet with parquet-rs 59 and zstd at ' + - 'level 3. ClickBench reads its published Parquet files.', + 'format, and, for random access, LitData. Not every suite runs every format.', + 'Parquet has two variants, one for size and one for speed. Parquet is zstd at level ' + + '3, written with parquet-rs 59. It is the baseline of the throughput and file size ' + + 'scores. Parquet (Snappy) is Parquet as DuckDB writes it by default: Snappy, with ' + + 'DuckDB’s row groups, dictionaries, statistics, and Bloom filters. The DuckDB 1.5 ' + + 'command line tool writes it from the same rows. The compression benchmark records ' + + 'its size, and TPC-DS at scale factor 1 queries it.', + 'Every SQL suite’s Parquet is zstd Parquet, except in ClickBench and FineWeb, which ' + + 'query their published Parquet files as Parquet. ClickBench prescribes its files, ' + + 'which are Snappy, so its Parquet results are not zstd Parquet and are not DuckDB’s ' + + 'Parquet (Snappy) either.', ], }, { @@ -96,6 +107,9 @@ export const METHODOLOGY: readonly MethodologySection[] = [ 'In both, a query or dataset a series did not run counts as twice that series’ own ' + 'slowest time, and at least twice the fastest series’ time, so a series cannot ' + 'rank well by skipping work. Each summary says how many a series ran.', + 'A query suite also ranks a series that missed a query below every series that ran ' + + 'them all. Lance cannot run TPC-DS query 72, so it ranks below the complete series ' + + 'on TPC-DS at scale factor 1.', `Write and scan throughput scores are ${SCORES.throughput}`, `File size scores are ${SCORES.size}`, 'Total is the sum of a format’s file sizes over the datasets that every format has.', diff --git a/src/bench/lib/queries.test.ts b/src/bench/lib/queries.test.ts index 777d7da..74e7ac1 100644 --- a/src/bench/lib/queries.test.ts +++ b/src/bench/lib/queries.test.ts @@ -371,21 +371,19 @@ describe('compareGroupSortKey canonical group ordering (no DB)', () => { ]); }); - it('puts compression and SQL file size before random access and string encoding last', () => { + it('puts compression and file sizes before random access and string encoding last', () => { const sorted = [ 'String Encoding Size', 'Clickbench', 'cohere-large-10m / partitioned', 'String Encoding', 'Random Access (NVMe)', - 'SQL File Size', - 'Compression Size', + 'File Sizes', 'Compression', ].sort(compareGroupSortKey); expect(sorted).toEqual([ 'Compression', - 'Compression Size', - 'SQL File Size', + 'File Sizes', 'Random Access (NVMe)', 'Clickbench', 'cohere-large-10m / partitioned', diff --git a/src/bench/lib/queries.ts b/src/bench/lib/queries.ts index de6d0a3..d00c53c 100644 --- a/src/bench/lib/queries.ts +++ b/src/bench/lib/queries.ts @@ -42,6 +42,13 @@ import { STRING_DATASET_PREFIX, } from './compression-suite'; import { groupDescription } from './descriptions'; +import { + FILE_SIZE_SCOPES, + FILE_SIZES_GROUP_NAME, + isFileSizeScope, + isFileSizesDataset, + SQL_FILE_SIZES_GROUP_NAME, +} from './file-sizes'; import type { FilterUniverse } from './chart-format'; import { type ChartKind, compareCodeUnits, FAMILIES, type GroupKind } from './families'; import { chartKeyFromSlug, chartKeyToSlug, groupKeyFromSlug, groupKeyToSlug } from './slug'; @@ -782,8 +789,12 @@ function chartInGroup(group: GroupKey, key: ChartKey): boolean { key.k === 'CompressionTime' && compressionSuite(key.dataset) === (group.suite ?? 'compress') ); case 'CompressionSizeGroup': + if (group.suite === 'strings') { + return key.k === 'CompressionSize' && compressionSuite(key.dataset) === 'strings'; + } return ( - key.k === 'CompressionSize' && compressionSuite(key.dataset) === (group.suite ?? 'compress') + (key.k === 'FileSize' && isFileSizeScope(key.benchmark, key.scale_factor)) || + (key.k === 'CompressionSize' && isFileSizesDataset(key.dataset, key.dataset_variant)) ); case 'RandomAccessGroup': return key.k === 'RandomAccess' && randomAccessStorage(key.dataset) === group.storage; @@ -877,6 +888,10 @@ function queryChartEarliest(pins: (alias: string) => string): string { /** One per-chart column of a [`groupRowsQuery`] block: its name in `ch`, element type, values. */ type ChartColumn = readonly [name: string, type: string, values: readonly unknown[]]; +/** The key of a chart from either compression table. */ +type CompressionChartKey = Extract; +type FileSizeChartKey = Extract; + /** * One row per (chart, fact row in the window), plus one row with NULL fact * columns for a chart with no rows there. `ord` is the chart's 1-based position @@ -934,6 +949,74 @@ function groupRowsQuery( ${rows} ) d ON true`); }; + /** + * The blocks for compression charts, each at its 1-based position in `charts`. + * `dataset_variant` varies per chart. `IS NOT DISTINCT FROM ch.dataset_variant` would plan as + * if it matched one row and rescan the window per fact row, so the charts split by whether the + * variant is NULL and each block pins it sargably, as `sargableDimEq` does. + */ + const compressionBlocks = ( + kind: CompressionChartKey['k'], + charts: readonly (readonly [CompressionChartKey, number])[], + ): void => { + const time = kind === 'CompressionTime'; + const table = time ? 'compression_times' : 'compression_sizes'; + const value = time ? 'value_ns' : 'value_bytes'; + const fields = time ? ['format', 'op'] : ['format']; + for (const nullVariant of [true, false]) { + const picked = charts.filter(([key]) => (key.dataset_variant === null) === nullVariant); + if (picked.length === 0) { + continue; + } + const columns: ChartColumn[] = [['dataset', 'text', picked.map(([key]) => key.dataset)]]; + if (!nullVariant) { + columns.push(['dataset_variant', 'text', picked.map(([key]) => key.dataset_variant)]); + } + const pins = (alias: string): string => + `${alias}.dataset = ch.dataset + AND ${alias}.dataset_variant ${nullVariant ? 'IS NULL' : '= ch.dataset_variant'}`; + block( + fields, + picked.map(([, position]) => position), + columns, + `SELECT MIN(c2.timestamp) + FROM ${table} f2 + JOIN ${BRANCH_COMMITS} c2 ON c2.commit_sha = f2.commit_sha + WHERE ${pins('f2')}`, + `SELECT f.commit_sha, + ${fields.map((field) => `f.${field}`).join(', ')}, f.${value}::float8 AS value + FROM ${table} f + JOIN ${BRANCH_COMMITS} c USING (commit_sha) + WHERE ${pins('f')}${factWindowFilter(params, window)}`, + ); + } + }; + /** One block for `file_sizes` charts: each chart's size per format, summed over its files. */ + const fileSizeBlocks = (files: readonly [FileSizeChartKey, number][]): void => { + if (files.length === 0) { + return; + } + const pins = (alias: string): string => + `${alias}.benchmark = ch.benchmark AND ${alias}.scale_factor = ch.scale_factor`; + block( + ['format'], + files.map(([, position]) => position), + [ + ['benchmark', 'text', files.map(([k]) => k.benchmark)], + ['scale_factor', 'text', files.map(([k]) => k.scale_factor)], + ], + `SELECT MIN(c2.timestamp) + FROM file_sizes f2 + JOIN ${BRANCH_COMMITS} c2 ON c2.commit_sha = f2.commit_sha + WHERE ${pins('f2')}`, + `SELECT f.commit_sha, + f.format, SUM(f.value_bytes)::float8 AS value + FROM file_sizes f + JOIN ${BRANCH_COMMITS} c USING (commit_sha) + WHERE ${pins('f')}${factWindowFilter(params, window)} + GROUP BY f.commit_sha, f.format`, + ); + }; switch (group.k) { case 'QueryGroup': { const keys = chartsOf(group, charts, 'QueryMeasurement'); @@ -957,50 +1040,31 @@ function groupRowsQuery( ); break; } - case 'CompressionTimeGroup': + case 'CompressionTimeGroup': { + const keys = chartsOf(group, charts, 'CompressionTime'); + compressionBlocks( + 'CompressionTime', + keys.map((key, i) => [key, i + 1]), + ); + break; + } case 'CompressionSizeGroup': { - const time = group.k === 'CompressionTimeGroup'; - const table = time ? 'compression_times' : 'compression_sizes'; - const value = time ? 'value_ns' : 'value_bytes'; - const fields = time ? ['format', 'op'] : ['format']; - const keys = chartsOf(group, charts, time ? 'CompressionTime' : 'CompressionSize'); - // `dataset_variant` varies per chart. `IS NOT DISTINCT FROM ch.dataset_variant` would plan - // as if it matched one row and rescan the window per fact row, so split the charts by - // whether the variant is NULL and pin each block sargably, as `sargableDimEq` does. - for (const nullVariant of [true, false]) { - const positions: number[] = []; - const picked: typeof keys = []; - keys.forEach((key, i) => { - if ((key.dataset_variant === null) === nullVariant) { - positions.push(i + 1); - picked.push(key); - } - }); - if (picked.length === 0) { - continue; + // The File Sizes group holds two families. Both give one size per format, so their blocks + // share columns. The String Encoding Size group holds `compression_sizes` charts alone. + const sizes: [CompressionChartKey, number][] = []; + const files: [FileSizeChartKey, number][] = []; + charts.forEach((key, i) => { + if (!chartInGroup(group, key)) { + throw new Error(`chart ${chartKeyToSlug(key)} is not in group ${groupKeyToSlug(group)}`); } - const columns: ChartColumn[] = [['dataset', 'text', picked.map((k) => k.dataset)]]; - if (!nullVariant) { - columns.push(['dataset_variant', 'text', picked.map((k) => k.dataset_variant)]); + if (key.k === 'CompressionSize') { + sizes.push([key, i + 1]); + } else if (key.k === 'FileSize') { + files.push([key, i + 1]); } - const pins = (alias: string): string => - `${alias}.dataset = ch.dataset - AND ${alias}.dataset_variant ${nullVariant ? 'IS NULL' : '= ch.dataset_variant'}`; - block( - fields, - positions, - columns, - `SELECT MIN(c2.timestamp) - FROM ${table} f2 - JOIN ${BRANCH_COMMITS} c2 ON c2.commit_sha = f2.commit_sha - WHERE ${pins('f2')}`, - `SELECT f.commit_sha, - ${fields.map((field) => `f.${field}`).join(', ')}, f.${value}::float8 AS value - FROM ${table} f - JOIN ${BRANCH_COMMITS} c USING (commit_sha) - WHERE ${pins('f')}${factWindowFilter(params, window)}`, - ); - } + }); + compressionBlocks('CompressionSize', sizes); + fileSizeBlocks(files); break; } case 'RandomAccessGroup': { @@ -1048,27 +1112,7 @@ function groupRowsQuery( break; } case 'FileSizeGroup': { - const keys = chartsOf(group, charts, 'FileSize'); - const pins = (alias: string): string => - `${alias}.benchmark = ch.benchmark AND ${alias}.scale_factor = ch.scale_factor`; - block( - ['format'], - keys.map((_, i) => i + 1), - [ - ['benchmark', 'text', keys.map((k) => k.benchmark)], - ['scale_factor', 'text', keys.map((k) => k.scale_factor)], - ], - `SELECT MIN(c2.timestamp) - FROM file_sizes f2 - JOIN ${BRANCH_COMMITS} c2 ON c2.commit_sha = f2.commit_sha - WHERE ${pins('f2')}`, - `SELECT f.commit_sha, - f.format, SUM(f.value_bytes)::float8 AS value - FROM file_sizes f - JOIN ${BRANCH_COMMITS} c USING (commit_sha) - WHERE ${pins('f')}${factWindowFilter(params, window)} - GROUP BY f.commit_sha, f.format`, - ); + fileSizeBlocks(chartsOf(group, charts, 'FileSize').map((key, i) => [key, i + 1])); break; } } @@ -1248,8 +1292,8 @@ export interface GroupChartsResponse { */ const GROUP_ORDER: readonly string[] = [ 'Compression', - 'Compression Size', - 'SQL File Size', + 'File Sizes', + 'SQL File Sizes', 'Random Access (NVMe)', 'Random Access (S3)', 'Clickbench', @@ -1531,46 +1575,61 @@ async function collectQueryGroups(): Promise { type DatasetVariantRow = { dataset: string; dataset_variant: string | null }; -/** Group display names per compression suite and family. */ -const COMPRESSION_GROUP_NAMES: Record< - 'CompressionTimeGroup' | 'CompressionSizeGroup', - Record -> = { +/** + * Group display names per compression suite and family. The size family's group without a suite + * is the File Sizes group, which [`collectFileSizesGroup`] builds, so the size family names only + * `string-bench`'s group here. + */ +const COMPRESSION_GROUP_NAMES = { CompressionTimeGroup: { compress: 'Compression', strings: 'String Encoding' }, - CompressionSizeGroup: { compress: 'Compression Size', strings: 'String Encoding Size' }, -}; + CompressionSizeGroup: { strings: 'String Encoding Size' }, +} as const; + +/** A compression `(dataset, dataset_variant)` row's chart link, named as its chart is. */ +function compressionChartLink( + row: DatasetVariantRow, + chartKind: CompressionChartKey['k'], +): ChartLink { + const dataset = + compressionSuite(row.dataset) === 'strings' + ? row.dataset.slice(STRING_DATASET_PREFIX.length) + : row.dataset; + return { + name: row.dataset_variant !== null ? `${dataset}/${row.dataset_variant}` : dataset, + slug: chartKeyToSlug({ + k: chartKind, + dataset: row.dataset, + dataset_variant: row.dataset_variant, + }), + }; +} /** - * Split one compression table's `(dataset, dataset_variant)` rows into the `compress-bench` - * group and the `string-bench` group, dropping a group with no rows. String charts drop the - * shared `strings/` prefix from their names. + * Split one compression table's `(dataset, dataset_variant)` rows into the family's suite + * groups, dropping a group with no rows. String charts drop the shared `strings/` prefix from + * their names. */ function compressionGroups( rows: readonly DatasetVariantRow[], groupKind: 'CompressionTimeGroup' | 'CompressionSizeGroup', ): Group[] { const chartKind = groupKind === 'CompressionTimeGroup' ? 'CompressionTime' : 'CompressionSize'; + const names: Partial> = COMPRESSION_GROUP_NAMES[groupKind]; const suites: CompressionSuite[] = ['compress', 'strings']; return suites.flatMap((suite) => { + const name = names[suite]; + if (name === undefined) { + return []; + } const charts: ChartLink[] = rows .filter((row) => compressionSuite(row.dataset) === suite) - .map((row) => { - const dataset = - suite === 'strings' ? row.dataset.slice(STRING_DATASET_PREFIX.length) : row.dataset; - return { - name: row.dataset_variant !== null ? `${dataset}/${row.dataset_variant}` : dataset, - slug: chartKeyToSlug({ - k: chartKind, - dataset: row.dataset, - dataset_variant: row.dataset_variant, - }), - }; - }); + .map((row) => compressionChartLink(row, chartKind)); if (charts.length === 0) { return []; } - const key: GroupKey = suite === 'strings' ? { k: groupKind, suite } : { k: groupKind }; - return [{ name: COMPRESSION_GROUP_NAMES[groupKind][suite], slug: groupKeyToSlug(key), charts }]; + const key: GroupKey = + suite === 'strings' ? { k: groupKind, suite } : { k: 'CompressionTimeGroup' }; + return [{ name, slug: groupKeyToSlug(key), charts }]; }); } @@ -1586,16 +1645,27 @@ async function collectCompressionTimeGroup(): Promise { return compressionGroups(rows, 'CompressionTimeGroup'); } -/** The `Compression Size` and `String Encoding Size` groups, or `[]` if the fact table is empty. */ -async function collectCompressionSizeGroup(): Promise { +/** Every `(dataset, dataset_variant)` in `compression_sizes`. */ +async function compressionSizeDatasets(): Promise { const text = ` SELECT dataset, dataset_variant FROM compression_sizes GROUP BY dataset, dataset_variant ORDER BY dataset, dataset_variant NULLS FIRST `; - const rows = (await getPool().query(text)).rows; - return compressionGroups(rows, 'CompressionSizeGroup'); + return (await getPool().query(text)).rows; +} + +/** + * The File Sizes group and the `String Encoding Size` group, each left out when it has no + * charts. + */ +async function collectCompressionSizeGroup(): Promise { + const datasets = await compressionSizeDatasets(); + return [ + ...(await collectFileSizesGroup(datasets)), + ...compressionGroups(datasets, 'CompressionSizeGroup'), + ]; } /** One random-access group per storage backend, omitting empty sections. */ @@ -1673,6 +1743,48 @@ async function collectVectorSearchGroups(): Promise { type FileSizeScopeRow = { benchmark: string; scale_factor: string }; +/** + * The File Sizes group, or `[]` if neither table has a chart for it: the SQL suites' scopes in + * [`FILE_SIZE_SCOPES`] that have rows, in that order, then the `compress-bench` datasets of + * `datasets`, the rows of `compression_sizes`, by name. + */ +async function collectFileSizesGroup(datasets: readonly DatasetVariantRow[]): Promise { + // One index probe of `idx_file_sizes_baseline` per listed scope. + const scopesText = ` + SELECT s.benchmark, s.scale_factor + FROM unnest($1::text[], $2::text[]) WITH ORDINALITY AS s(benchmark, scale_factor, ord) + WHERE EXISTS ( + SELECT 1 + FROM file_sizes f + WHERE f.benchmark = s.benchmark + AND f.scale_factor = s.scale_factor) + ORDER BY s.ord + `; + const scopes = await getPool().query(scopesText, [ + FILE_SIZE_SCOPES.map((scope) => scope.benchmark), + FILE_SIZE_SCOPES.map((scope) => scope.scale_factor), + ]); + const charts: ChartLink[] = [ + ...scopes.rows.map((row) => ({ + name: fileSizeChartName(row.benchmark, row.scale_factor), + slug: chartKeyToSlug({ + k: 'FileSize', + benchmark: row.benchmark, + scale_factor: row.scale_factor, + }), + })), + ...datasets + .filter((row) => isFileSizesDataset(row.dataset, row.dataset_variant)) + .map((row) => compressionChartLink(row, 'CompressionSize')), + ]; + if (charts.length === 0) { + return []; + } + return [ + { name: FILE_SIZES_GROUP_NAME, slug: groupKeyToSlug({ k: 'CompressionSizeGroup' }), charts }, + ]; +} + /** Benchmarks whose file-size charts lead the group, in this order; the rest follow by name. */ const FILE_SIZE_CHART_ORDER: readonly string[] = ['clickbench', 'tpch', 'tpcds']; @@ -1683,8 +1795,8 @@ function fileSizeChartRank(benchmark: string): number { } /** - * The `SQL File Size` group: one chart per benchmark and scale factor, or `[]` if the table is - * empty. Charts order by [`FILE_SIZE_CHART_ORDER`], then by benchmark name, then by numeric + * The SQL File Sizes group: one chart per benchmark and scale factor in `file_sizes`, or `[]` if + * the table is empty. Charts order by [`FILE_SIZE_CHART_ORDER`], then by benchmark name, then by numeric * scale factor, so `10.0` follows `1.0`. */ async function collectFileSizeGroup(): Promise { @@ -1722,7 +1834,7 @@ async function collectFileSizeGroup(): Promise { ); return [ { - name: 'SQL File Size', + name: SQL_FILE_SIZES_GROUP_NAME, slug: groupKeyToSlug({ k: 'FileSizeGroup' }), charts: rows.map((row) => ({ name: fileSizeChartName(row.benchmark, row.scale_factor), diff --git a/src/bench/lib/site-files.test.ts b/src/bench/lib/site-files.test.ts index cfb6e13..147edf6 100644 --- a/src/bench/lib/site-files.test.ts +++ b/src/bench/lib/site-files.test.ts @@ -136,6 +136,8 @@ describe('summaryLines', () => { maxRatio: 1.549, compressionRatio: 5.744, totalBytes: 1_603_980_000, + measured: 1, + total: 1, }, ], }).at(-1), diff --git a/src/bench/lib/site-files.ts b/src/bench/lib/site-files.ts index 8a5dd95..685edcc 100644 --- a/src/bench/lib/site-files.ts +++ b/src/bench/lib/site-files.ts @@ -191,6 +191,7 @@ const FORMATS: ReadonlyMap = new Map([ ['vortex-file-compressed', 'Vortex'], ['vortex-compact', 'Vortex (compact)'], ['parquet', 'Parquet'], + ['parquet-snappy', 'Parquet (Snappy)'], ['lance', 'Lance'], ['arrow-ipc', 'Arrow IPC'], ['arrow', 'Arrow'], @@ -323,7 +324,10 @@ export function summaryLines(summary: Summary): string[] { (item.compressionRatio === null ? '' : `, ${formatCompressionSizeRatio(item.compressionRatio)} smaller than Arrow in memory`) + - (item.totalBytes === null ? '' : `, ${formatBytes(item.totalBytes)} in total`), + (item.totalBytes === null ? '' : `, ${formatBytes(item.totalBytes)} in total`) + + (item.measured < item.total + ? `, measured on ${item.measured} of ${item.total} datasets` + : ''), ), ]; } diff --git a/src/bench/lib/slug.ts b/src/bench/lib/slug.ts index 89d6a97..7643668 100644 --- a/src/bench/lib/slug.ts +++ b/src/bench/lib/slug.ts @@ -33,7 +33,7 @@ export type ChartKey = | { k: 'CompressionSize'; dataset: string; dataset_variant: string | null } | { k: 'RandomAccess'; dataset: string } | { k: 'VectorSearch'; dataset: string; layout: string; threshold: number } - // One SQL benchmark's data at one scale factor; its series are the Vortex formats. + // One SQL benchmark's data at one scale factor; its series are the formats it writes. | { k: 'FileSize'; benchmark: string; scale_factor: string }; /** Strongly-typed group key parsed from a slug. Discriminated on `k`. */ @@ -47,9 +47,11 @@ export type GroupKey = } // `suite` is absent for the `compress-bench` group, so its slug predates the split. | { k: 'CompressionTimeGroup'; suite?: 'strings' } + // Without `suite`, the File Sizes group: `compress-bench`'s sizes and three SQL scopes. | { k: 'CompressionSizeGroup'; suite?: 'strings' } | { k: 'RandomAccessGroup'; storage: RandomAccessStorage } | { k: 'VectorSearchGroup'; dataset: string; layout: string } + // The SQL File Sizes group: every SQL benchmark and scale factor in `file_sizes`. | { k: 'FileSizeGroup' }; const CHART_KINDS: readonly ChartKind[] = [ diff --git a/src/bench/lib/summary.test.ts b/src/bench/lib/summary.test.ts index d053b1f..f9cdab4 100644 --- a/src/bench/lib/summary.test.ts +++ b/src/bench/lib/summary.test.ts @@ -20,6 +20,9 @@ const GROUP_KINDS = FAMILIES.map((family) => family.groupKind); describe('compression summaries', () => { beforeEach(() => { query.mockReset(); + // The File Sizes summary also reads the SQL suites' sizes; a test that sets no rows for that + // second query gets none. + query.mockResolvedValue({ rows: [] }); }); it('uses geometric mean size ratios', async () => { @@ -82,12 +85,17 @@ describe('compression summaries', () => { // Both formats have both datasets, so each total sums both. expect(byFormat.get('vortex-file-compressed')?.totalBytes).toBe(901); expect(byFormat.get('parquet')?.totalBytes).toBe(104); + expect(summary.title).toBe('File Sizes Summary'); expect(summary.explanation).toBe( - 'Total compressed size, and geometric means of compressed sizes versus Arrow (higher is better) and versus Parquet-zstd (lower is better)', + 'Total file size, and geometric means of file sizes versus Arrow (higher is better) and versus Parquet-zstd (lower is better)', ); + expect(summary.rankings.map((ranking) => [ranking.measured, ranking.total])).toEqual([ + [2, 2], + [2, 2], + ]); }); - it('sums totals over the datasets every format has', async () => { + it('totals every dataset, for the formats measured on all of them', async () => { const row = (format: string, dataset: string, valueBytes: number, datasetVariant = null) => ({ format, dataset, @@ -102,8 +110,9 @@ describe('compression summaries', () => { row('vortex-file-compressed', 'b', 1_000), row('parquet', 'a', 20), row('parquet', 'b', 2_000), - // Lance has no size for dataset b, so no total counts b. - row('lance', 'a', 30), + // Lance has no size for dataset b, so it has no total, and it ranks after the formats + // measured on both datasets although its ratio is the lowest. + row('lance', 'a', 3), ], }); @@ -111,14 +120,94 @@ describe('compression summaries', () => { if (summary === null || summary.type !== 'compressionSize') { throw new Error('expected a compressionSize summary'); } - const totals = new Map(summary.rankings.map((ranking) => [ranking.name, ranking.totalBytes])); - expect(totals).toEqual( - new Map([ - ['vortex-file-compressed', 10], - ['parquet', 20], - ['lance', 30], + expect( + summary.rankings.map((ranking) => [ + ranking.name, + ranking.totalBytes, + ranking.measured, + ranking.total, ]), - ); + ).toEqual([ + ['vortex-file-compressed', 1_010, 2, 2], + ['parquet', 2_020, 2, 2], + ['lance', null, 1, 2], + ]); + }); + + it('counts each synthetic family as one dataset', async () => { + const row = (format: string, dataset: string, valueBytes: number) => ({ + format, + dataset, + datasetVariant: null, + valueBytes, + baselineBytes: 100, + uncompressedBytes: null, + }); + query.mockResolvedValueOnce({ + rows: [ + row('vortex-file-compressed', 'taxi', 400), + // Three wide tables and two sparse-wide shapes: one entry each, their geometric means + // 1 and 0.25. + row('vortex-file-compressed', 'wide table cols=100 chunks=1 rows=1000', 50), + row('vortex-file-compressed', 'wide table cols=1000 chunks=1 rows=1000', 100), + row('vortex-file-compressed', 'wide table cols=10000 chunks=1 rows=1000', 200), + row('vortex-file-compressed', 'sparse-wide-flat-runs', 5), + row('vortex-file-compressed', 'sparse-wide-lists-runs', 125), + ], + }); + + const summary = await collectGroupSummary({ k: 'CompressionSizeGroup' }); + if (summary === null || summary.type !== 'compressionSize') { + throw new Error('expected a compressionSize summary'); + } + const vortex = defined(summary.rankings[0]); + // The cube root of 4 * 1 * 0.25 is 1. Counted one by one, the six datasets give 2^(1/6). + expect(vortex.ratio).toBeCloseTo(1, 6); + expect([vortex.measured, vortex.total]).toEqual([3, 3]); + // The range is over single datasets, and the total over every one of them. + expect(vortex.minRatio).toBeCloseTo(0.05, 6); + expect(vortex.maxRatio).toBeCloseTo(4, 6); + expect(vortex.totalBytes).toBe(880); + }); + + it("ranks the SQL suites' totals with the compress-bench sizes", async () => { + const row = (format: string, dataset: string, valueBytes: number, baselineBytes: number) => ({ + format, + dataset, + datasetVariant: dataset === 'tpcds' ? 'sf=1.0' : null, + valueBytes, + baselineBytes, + uncompressedBytes: null, + }); + query.mockResolvedValueOnce({ + rows: [row('vortex-file-compressed', 'taxi', 50, 100), row('parquet', 'taxi', 100, 100)], + }); + query.mockResolvedValueOnce({ + rows: [ + row('vortex-file-compressed', 'tpcds', 200, 100), + row('vortex-compact', 'tpcds', 50, 100), + row('parquet', 'tpcds', 100, 100), + ], + }); + + const summary = await collectGroupSummary({ k: 'CompressionSizeGroup' }); + if (summary === null || summary.type !== 'compressionSize') { + throw new Error('expected a compressionSize summary'); + } + expect(query.mock.calls[1]?.[1]).toEqual([['tpcds'], ['1.0'], 'parquet']); + expect( + summary.rankings.map((ranking) => [ + ranking.name, + ranking.ratio, + ranking.measured, + ranking.totalBytes, + ]), + ).toEqual([ + ['parquet', 1, 2, 200], + ['vortex-file-compressed', 1, 2, 250], + // Vortex Compact has a size for TPC-DS alone, so it ranks after the complete formats. + ['vortex-compact', 0.5, 1, null], + ]); }); it('gives no total when the formats share no dataset', async () => { @@ -192,18 +281,24 @@ describe('compression summaries', () => { rows: [ { format: 'vortex-file-compressed', + dataset: 'a', + datasetVariant: null, valueBytes: 200, baselineBytes: 100, uncompressedBytes: 2_000, }, { format: 'parquet', + dataset: 'a', + datasetVariant: null, valueBytes: 100, baselineBytes: 100, uncompressedBytes: 500, }, { format: 'lance', + dataset: 'a', + datasetVariant: null, valueBytes: 300, baselineBytes: 100, uncompressedBytes: null, @@ -290,7 +385,8 @@ describe('compression summaries', () => { await collectGroupSummary({ k: 'CompressionTimeGroup' }); await collectGroupSummary({ k: 'CompressionSizeGroup' }); - expect(query).toHaveBeenCalledTimes(2); + // The File Sizes summary reads the SQL suites' sizes in a third query. + expect(query).toHaveBeenCalledTimes(3); const timingParams = [ ['vortex-file-compressed', 'parquet', 'lance'], ['vortex-file-compressed', 'parquet'], @@ -299,14 +395,21 @@ describe('compression summaries', () => { ['wide table %', 'sparse-wide-%'], ]; const sizeParams = [ - ['vortex-file-compressed', 'parquet', 'lance', 'arrow-ipc'], - ['vortex-file-compressed', 'parquet'], + [ + 'vortex-file-compressed', + 'parquet', + 'parquet-snappy', + 'vortex-compact', + 'lance', + 'arrow-ipc', + ], + ['vortex-file-compressed', 'parquet', 'parquet-snappy', 'vortex-compact'], 'vortex-file-compressed', 'parquet', 'arrow-ipc', - ['wide table %', 'sparse-wide-%', 'tpc-h l_comment chunked', 'statpopgen'], + ['tpc-h l_comment chunked', 'wide table cols=% chunks=50 rows=%', 'statpopgen'], ]; - const calls = query.mock.calls as Array<[string, unknown[]]>; + const calls = (query.mock.calls as Array<[string, unknown[]]>).slice(0, 2); expect(defined(calls[0])[1]).toEqual(timingParams); expect(defined(calls[1])[1]).toEqual(sizeParams); for (const [text] of calls) { @@ -717,6 +820,39 @@ describe('timing summaries (shared ranking model)', () => { }, ); + it('ranks a series that misses a query below every series that ran them all', async () => { + // `partial` is twice as fast on the queries it runs and skips the slowest one. The penalty + // scores the skipped query at 2x the best, so its score, 2^(1/3), beats `complete`'s, 2^(2/3). + query.mockResolvedValueOnce({ + rows: [ + { query_idx: 1, series: 'lance:lance', value_ns: 1_000_000 }, + { query_idx: 1, series: 'datafusion:parquet', value_ns: 2_000_000 }, + { query_idx: 2, series: 'lance:lance', value_ns: 1_000_000 }, + { query_idx: 2, series: 'datafusion:parquet', value_ns: 2_000_000 }, + { query_idx: 3, series: 'datafusion:parquet', value_ns: 10_000_000 }, + ], + }); + + const summary = await collectGroupSummary({ + k: 'QueryGroup', + dataset: 'tpcds', + dataset_variant: null, + scale_factor: '1', + storage: 'nvme', + }); + if (summary === null || summary.type !== 'queryBenchmark') { + throw new Error('expected a queryBenchmark summary'); + } + + expect(summary.rankings.map((ranking) => ranking.name)).toEqual([ + 'datafusion:parquet', + 'lance:lance', + ]); + const [complete, partial] = summary.rankings; + expect(defined(partial).score).toBeLessThan(defined(complete).score); + expect(defined(partial)).toMatchObject({ measured: 2, total: 3 }); + }); + it('summarizes every query group, with no dataset allowlist', async () => { // `spatialbench` (and every other suite outside the retired v2 five) used to // fall through to `null` and render no card at all. @@ -750,7 +886,7 @@ describe('timing summaries (shared ranking model)', () => { storage: 'nvme', }, { k: 'CompressionTimeGroup' }, - { k: 'CompressionSizeGroup' }, + { k: 'CompressionSizeGroup', suite: 'strings' }, { k: 'RandomAccessGroup', storage: 'nvme' }, { k: 'VectorSearchGroup', dataset: 'd', layout: 'l' }, { k: 'FileSizeGroup' }, diff --git a/src/bench/lib/summary.ts b/src/bench/lib/summary.ts index 43edd96..3804a52 100644 --- a/src/bench/lib/summary.ts +++ b/src/bench/lib/summary.ts @@ -41,8 +41,10 @@ import { getPool } from './db'; import { randomAccessStorage, type RandomAccessStorage } from './random-access'; import { compareCodeUnits } from './families'; import type { GroupKey } from './slug'; +import { FILE_SIZE_SUMMARY_SCOPES } from './file-sizes'; import { SIZE_SUMMARY_EXCLUDED_DATASET_PATTERNS, + SIZE_SUMMARY_FAMILIES, STRING_DATASET_PATTERN, SUMMARY_EXCLUDED_DATASET_PATTERNS, } from './compression-suite'; @@ -58,6 +60,13 @@ const COMPRESSION_SHARED_SNAPSHOT_FORMATS = ['vortex-file-compressed', 'parquet' const COMPRESSION_INDEPENDENT_SNAPSHOT_FORMATS = ['lance']; const ARROW_IPC_FORMAT = 'arrow-ipc'; +/** + * Size-only formats that `compress-bench` writes beside the timed ones on every run, so they + * share the Vortex snapshot too. `parquet-snappy` is Parquet with Snappy; `parquet` is zstd. + * `vortex-compact` is Vortex with the compact strategy, as the SQL suites write it. + */ +const SIZE_ONLY_SHARED_SNAPSHOT_FORMATS = ['parquet-snappy', 'vortex-compact']; + const COMPRESSION_SNAPSHOT_ANCHOR = 'vortex-file-compressed'; const COMPRESSION_BASELINE = 'parquet'; @@ -82,13 +91,10 @@ function compressionSizeSummaryQueryParams(): [ string, string[], ] { + const shared = [...COMPRESSION_SHARED_SNAPSHOT_FORMATS, ...SIZE_ONLY_SHARED_SNAPSHOT_FORMATS]; return [ - [ - ...COMPRESSION_SHARED_SNAPSHOT_FORMATS, - ...COMPRESSION_INDEPENDENT_SNAPSHOT_FORMATS, - ARROW_IPC_FORMAT, - ], - COMPRESSION_SHARED_SNAPSHOT_FORMATS, + [...shared, ...COMPRESSION_INDEPENDENT_SNAPSHOT_FORMATS, ARROW_IPC_FORMAT], + shared, COMPRESSION_SNAPSHOT_ANCHOR, COMPRESSION_BASELINE, ARROW_IPC_FORMAT, @@ -132,7 +138,7 @@ interface CompressionRanking { throughputGbS?: number; } -/** One format in the compression size summary. */ +/** One format in a size summary. */ interface CompressionSizeRanking { /** On-disk format. */ name: string; @@ -145,10 +151,15 @@ interface CompressionSizeRanking { /** Geomean ratio of Arrow memory size to the format size. */ compressionRatio: number | null; /** - * Sum of this format's sizes, in bytes, over the datasets that every format in the summary - * has, so the totals compare directly. `null` when the formats share no dataset. + * Sum of this format's sizes, in bytes, over a set of datasets that is the same for every + * format, so the totals compare directly. See [`SizeTotals`]. `null` when the format has no + * total. */ totalBytes: number | null; + /** Datasets this format has a size for, counting a family once ([`sizeBucket`]). */ + measured: number; + /** Datasets in the summary, counted as `measured` is: the denominator it is out of. */ + total: number; } /** @@ -203,7 +214,7 @@ export function collectGroupSummary(key: GroupKey): Promise { case 'CompressionSizeGroup': return key.suite === 'strings' ? collectStringEncodingSizeSummary() - : collectCompressionSizeSummary(); + : collectFileSizesSummary(); case 'RandomAccessGroup': return collectRandomAccessSummary(key.storage); case 'VectorSearchGroup': @@ -711,20 +722,126 @@ async function compressionSamples(): Promise { return (await getPool().query(text, compressionSummaryQueryParams())).rows; } -async function collectCompressionSizeSummary(): Promise { - const rankings = compressionSizeRankings(await compressionSizeSamples()); +/** The format every other SQL benchmark file format is compared with. */ +const FILE_SIZE_BASELINE = 'vortex-file-compressed'; + +/** + * The SQL File Sizes rollup: each format's total file size against `vortex-file-compressed`, per + * benchmark scope (benchmark and scale factor) at that scope's newest commit with a baseline. + * Unlike the File Sizes summary, it covers every scope in `file_sizes` and leaves out + * `compress-bench`. Scopes come from nightly as well as develop runs, so each one uses its own + * newest commit. File sizes record no Arrow size, so the Arrow ratio is empty. + */ +async function collectFileSizeSummary(): Promise { + // `scopes` is a skip scan of `idx_file_sizes_baseline`. `latest` walks the read branch's + // commits newest first and probes that index, so it stops at the newest commit with a baseline. + const text = ` + WITH RECURSIVE scopes AS ( + (SELECT f.benchmark, f.scale_factor + FROM file_sizes f + ORDER BY f.benchmark, f.scale_factor + LIMIT 1) + UNION ALL + SELECT nxt.benchmark, nxt.scale_factor + FROM scopes s + CROSS JOIN LATERAL ( + SELECT f.benchmark, f.scale_factor + FROM file_sizes f + WHERE (f.benchmark, f.scale_factor) > (s.benchmark, s.scale_factor) + ORDER BY f.benchmark, f.scale_factor + LIMIT 1 + ) nxt + ), + latest AS ( + SELECT s.benchmark, s.scale_factor, l.commit_sha + FROM scopes s + CROSS JOIN LATERAL ( + SELECT c.commit_sha + FROM ${BRANCH_COMMITS} c + WHERE EXISTS ( + SELECT 1 + FROM file_sizes f + WHERE f.benchmark = s.benchmark + AND f.scale_factor = s.scale_factor + AND f.format = $1 + AND f.commit_sha = c.commit_sha) + ORDER BY c.timestamp DESC, c.commit_sha DESC + LIMIT 1 + ) l + ), + totals AS ( + SELECT f.benchmark, f.scale_factor, f.format, SUM(f.value_bytes)::float8 AS bytes + FROM file_sizes f + JOIN latest l + ON l.benchmark = f.benchmark + AND l.scale_factor = f.scale_factor + AND l.commit_sha = f.commit_sha + GROUP BY f.benchmark, f.scale_factor, f.format + ) + SELECT t.format AS format, + t.benchmark AS dataset, + t.scale_factor AS "datasetVariant", + t.bytes AS "valueBytes", + b.bytes AS "baselineBytes", + NULL::float8 AS "uncompressedBytes" + FROM totals t + JOIN totals b + ON b.benchmark = t.benchmark + AND b.scale_factor = t.scale_factor + AND b.format = $1 + WHERE t.bytes > 0 + AND b.bytes > 0 + ORDER BY t.format, t.benchmark, t.scale_factor + `; + const rows = (await getPool().query(text, [FILE_SIZE_BASELINE])).rows; + const rankings = compressionSizeRankings(rows); + if (rankings.length === 0) { + return null; + } + return { + type: 'compressionSize', + title: 'SQL File Sizes Summary', + rankings, + explanation: + 'Total size of the files each SQL benchmark writes at every scale factor, and geometric means of sizes versus vortex (lower is better)', + // The card names formats by their display names, which call `vortex-file-compressed` `vortex`. + baseline: 'vortex', + }; +} + +/** + * The `File Sizes` rollup: each format's sizes against Parquet's (zstd) over the + * `compress-bench` datasets ([`compressionSizeSamples`]) and the SQL scopes in + * [`FILE_SIZE_SUMMARY_SCOPES`] ([`fileSizeSamples`]). Each synthetic family in + * [`SIZE_SUMMARY_FAMILIES`] counts as one dataset. Only `compress-bench` records Arrow memory + * sizes, so a compression ratio covers its datasets alone. The suites write different formats, so + * each ranking says how many datasets it covers. + */ +async function collectFileSizesSummary(): Promise { + const [sizes, files] = await Promise.all([compressionSizeSamples(), fileSizeSamples()]); + const rankings = compressionSizeRankings([...sizes, ...files], 'complete'); if (rankings.length === 0) { return null; } return { type: 'compressionSize', - title: 'Compression Size Summary', + title: 'File Sizes Summary', rankings, explanation: - 'Total compressed size, and geometric means of compressed sizes versus Arrow (higher is better) and versus Parquet-zstd (lower is better)', + 'Total file size, and geometric means of file sizes versus Arrow (higher is better) and versus Parquet-zstd (lower is better)', }; } +/** + * Which datasets a size summary's totals cover: + * - `shared`: the datasets that every ranked format has. Every format gets a total. + * - `complete`: every dataset in the summary. Only a format measured on all of them gets a + * total. The File Sizes summary uses it: its SQL scopes record DuckDB and Vortex Compact, + * which `compress-bench` does not write, and `compress-bench` records Arrow IPC, which the SQL + * suites do not, so no dataset is shared by every format. + */ +type SizeTotals = 'shared' | 'complete'; + /** One size paired with the baseline's size for the same dataset and commit. */ interface CompressionSizeSample { format: string; @@ -736,28 +853,62 @@ interface CompressionSizeSample { } /** - * Rank each format by its size ratios to the baseline and its compression ratio to Arrow. The - * order is by compression ratio once every format has one, and by baseline ratio until then. + * The summary entry a dataset counts toward: its family's name for a dataset in a + * [`SIZE_SUMMARY_FAMILIES`] family, and the dataset itself otherwise. + */ +function sizeBucket(dataset: string, datasetVariant: string | null): string { + const family = SIZE_SUMMARY_FAMILIES.find((candidate) => dataset.startsWith(candidate.prefix)); + return family === undefined + ? JSON.stringify([dataset, datasetVariant]) + : JSON.stringify([family.name]); +} + +/** Geometric mean of each bucket's geometric mean: one entry per bucket. */ +function bucketGeoMean(byBucket: ReadonlyMap): number | null { + return geoMean([...byBucket.values()].flatMap((values) => geoMean(values) ?? [])); +} + +/** + * Rank each format by its size ratios to the baseline and its compression ratio to Arrow. + * + * Each dataset is one entry in the geometric means, except that a family counts once + * ([`sizeBucket`]). `minRatio` and `maxRatio` are over single datasets. `measured` and `total` + * count buckets. A format measured on every bucket ranks above one that is not, because their + * means are over different datasets. Within each tier, the order is by compression ratio once + * every format in the summary has one, and by baseline ratio until then. */ -function compressionSizeRankings(rows: readonly CompressionSizeSample[]): CompressionSizeRanking[] { +function compressionSizeRankings( + rows: readonly CompressionSizeSample[], + totals: SizeTotals = 'shared', +): CompressionSizeRanking[] { const grouped = new Map< string, - { sizeRatios: number[]; compressionRatios: number[]; bytesByDataset: Map } + { + sizeRatios: Map; + compressionRatios: Map; + bytesByDataset: Map; + } >(); + const buckets = new Set(); for (const row of rows) { const ratios = grouped.get(row.format) ?? { - sizeRatios: [], - compressionRatios: [], + sizeRatios: new Map(), + compressionRatios: new Map(), bytesByDataset: new Map(), }; + const bucket = sizeBucket(row.dataset, row.datasetVariant); if ( row.valueBytes > 0 && Number.isFinite(row.valueBytes) && row.baselineBytes > 0 && Number.isFinite(row.baselineBytes) ) { - ratios.sizeRatios.push(row.valueBytes / row.baselineBytes); + ratios.sizeRatios.set(bucket, [ + ...(ratios.sizeRatios.get(bucket) ?? []), + row.valueBytes / row.baselineBytes, + ]); ratios.bytesByDataset.set(JSON.stringify([row.dataset, row.datasetVariant]), row.valueBytes); + buckets.add(bucket); } if ( row.uncompressedBytes !== null && @@ -766,23 +917,29 @@ function compressionSizeRankings(rows: readonly CompressionSizeSample[]): Compre row.valueBytes > 0 && Number.isFinite(row.valueBytes) ) { - ratios.compressionRatios.push(row.uncompressedBytes / row.valueBytes); + ratios.compressionRatios.set(bucket, [ + ...(ratios.compressionRatios.get(bucket) ?? []), + row.uncompressedBytes / row.valueBytes, + ]); } grouped.set(row.format, ratios); } // Lance and Arrow IPC come from their own snapshots, so a format can lack a dataset that the - // others have. Totals use only the datasets every ranked format has. - const ranked = [...grouped.values()].filter((ratios) => ratios.sizeRatios.length > 0); - const commonDatasets = [...(ranked[0]?.bytesByDataset.keys() ?? [])].filter((dataset) => - ranked.every((ratios) => ratios.bytesByDataset.has(dataset)), - ); + // others have. Totals cover one set of datasets for every format (`SizeTotals`). + const ranked = [...grouped.values()].filter((ratios) => ratios.sizeRatios.size > 0); + const commonDatasets = + totals === 'shared' + ? [...(ranked[0]?.bytesByDataset.keys() ?? [])].filter((dataset) => + ranked.every((ratios) => ratios.bytesByDataset.has(dataset)), + ) + : [...new Set(ranked.flatMap((ratios) => [...ratios.bytesByDataset.keys()]))]; const rankings: CompressionSizeRanking[] = []; for (const [name, ratios] of grouped) { - const ratio = geoMean(ratios.sizeRatios); + const ratio = bucketGeoMean(ratios.sizeRatios); if (ratio !== null) { let minRatio = Infinity; let maxRatio = -Infinity; - for (const sizeRatio of ratios.sizeRatios) { + for (const sizeRatio of [...ratios.sizeRatios.values()].flat()) { minRatio = Math.min(minRatio, sizeRatio); maxRatio = Math.max(maxRatio, sizeRatio); } @@ -791,19 +948,27 @@ function compressionSizeRankings(rows: readonly CompressionSizeSample[]): Compre minRatio, ratio, maxRatio, - compressionRatio: geoMean(ratios.compressionRatios), + compressionRatio: bucketGeoMean(ratios.compressionRatios), totalBytes: - commonDatasets.length === 0 + commonDatasets.length === 0 || + !commonDatasets.every((dataset) => ratios.bytesByDataset.has(dataset)) ? null : commonDatasets.reduce( (sum, dataset) => sum + (ratios.bytesByDataset.get(dataset) ?? 0), 0, ), + measured: ratios.sizeRatios.size, + total: buckets.size, }); } } const hasCompleteArrowRatios = rankings.every((ranking) => ranking.compressionRatio !== null); + const partial = (ranking: CompressionSizeRanking) => (ranking.measured < ranking.total ? 1 : 0); rankings.sort((a, b) => { + const tier = partial(a) - partial(b); + if (tier !== 0) { + return tier; + } if (!hasCompleteArrowRatios) { return a.ratio - b.ratio || compareCodeUnits(a.name, b.name); } @@ -1023,64 +1188,73 @@ async function collectStringEncodingSizeSummary(): Promise { }; } -/** The format every other SQL benchmark file format is compared with. */ -const FILE_SIZE_BASELINE = 'vortex-file-compressed'; - /** - * The `SQL File Size` rollup: each Vortex format's total file size against - * `vortex-file-compressed`, per benchmark scope (benchmark and scale factor) at that scope's - * newest commit with a baseline. Scopes come from nightly as well as develop runs, so each one - * uses its own newest commit. File sizes record no Arrow size, so the Arrow ratio is empty. + * Size samples for the SQL scopes in [`FILE_SIZE_SUMMARY_SCOPES`]: each format's total over a + * scope's files, with Parquet's total from the same commit. Each format uses its own newest + * commit that also has Parquet, as Lance does in the `compress-bench` samples, so a format that a + * suite records less often than every run still counts. Scopes and formats come from a skip scan + * of `idx_file_sizes_baseline`. `latest` walks the read branch's commits newest first and probes + * that index, so it stops at the newest commit with both formats. */ -async function collectFileSizeSummary(): Promise { - // `scopes` is a skip scan of `idx_file_sizes_baseline`. `latest` walks the read branch's - // commits newest first and probes that index, so it stops at the newest commit with a baseline. +async function fileSizeSamples(): Promise { const text = ` - WITH RECURSIVE scopes AS ( - (SELECT f.benchmark, f.scale_factor + WITH RECURSIVE scope_formats AS ( + (SELECT f.benchmark, f.scale_factor, f.format FROM file_sizes f - ORDER BY f.benchmark, f.scale_factor + ORDER BY f.benchmark, f.scale_factor, f.format LIMIT 1) UNION ALL - SELECT nxt.benchmark, nxt.scale_factor - FROM scopes s + SELECT nxt.benchmark, nxt.scale_factor, nxt.format + FROM scope_formats p CROSS JOIN LATERAL ( - SELECT f.benchmark, f.scale_factor + SELECT f.benchmark, f.scale_factor, f.format FROM file_sizes f - WHERE (f.benchmark, f.scale_factor) > (s.benchmark, s.scale_factor) - ORDER BY f.benchmark, f.scale_factor + WHERE (f.benchmark, f.scale_factor, f.format) > (p.benchmark, p.scale_factor, p.format) + ORDER BY f.benchmark, f.scale_factor, f.format LIMIT 1 ) nxt ), latest AS ( - SELECT s.benchmark, s.scale_factor, l.commit_sha - FROM scopes s + SELECT t.benchmark, t.scale_factor, t.format, l.commit_sha + FROM scope_formats t + JOIN unnest($1::text[], $2::text[]) AS s(benchmark, scale_factor) + ON s.benchmark = t.benchmark + AND s.scale_factor = t.scale_factor CROSS JOIN LATERAL ( SELECT c.commit_sha FROM ${BRANCH_COMMITS} c WHERE EXISTS ( SELECT 1 FROM file_sizes f - WHERE f.benchmark = s.benchmark - AND f.scale_factor = s.scale_factor - AND f.format = $1 + WHERE f.benchmark = t.benchmark + AND f.scale_factor = t.scale_factor + AND f.format = t.format + AND f.commit_sha = c.commit_sha) + AND EXISTS ( + SELECT 1 + FROM file_sizes f + WHERE f.benchmark = t.benchmark + AND f.scale_factor = t.scale_factor + AND f.format = $3 AND f.commit_sha = c.commit_sha) ORDER BY c.timestamp DESC, c.commit_sha DESC LIMIT 1 ) l ), totals AS ( - SELECT f.benchmark, f.scale_factor, f.format, SUM(f.value_bytes)::float8 AS bytes - FROM file_sizes f - JOIN latest l - ON l.benchmark = f.benchmark - AND l.scale_factor = f.scale_factor - AND l.commit_sha = f.commit_sha - GROUP BY f.benchmark, f.scale_factor, f.format + SELECT l.benchmark, l.scale_factor, l.format, f.format AS measured, + SUM(f.value_bytes)::float8 AS bytes + FROM latest l + JOIN file_sizes f + ON f.benchmark = l.benchmark + AND f.scale_factor = l.scale_factor + AND f.commit_sha = l.commit_sha + AND f.format IN (l.format, $3) + GROUP BY l.benchmark, l.scale_factor, l.format, f.format ) SELECT t.format AS format, t.benchmark AS dataset, - t.scale_factor AS "datasetVariant", + 'sf=' || t.scale_factor AS "datasetVariant", t.bytes AS "valueBytes", b.bytes AS "baselineBytes", NULL::float8 AS "uncompressedBytes" @@ -1088,25 +1262,20 @@ async function collectFileSizeSummary(): Promise { JOIN totals b ON b.benchmark = t.benchmark AND b.scale_factor = t.scale_factor - AND b.format = $1 - WHERE t.bytes > 0 + AND b.format = t.format + AND b.measured = $3 + WHERE t.measured = t.format + AND t.bytes > 0 AND b.bytes > 0 ORDER BY t.format, t.benchmark, t.scale_factor `; - const rows = (await getPool().query(text, [FILE_SIZE_BASELINE])).rows; - const rankings = compressionSizeRankings(rows); - if (rankings.length === 0) { - return null; - } - return { - type: 'compressionSize', - title: 'SQL File Size Summary', - rankings, - explanation: - 'Total size of the Vortex files each SQL benchmark writes, and geometric means of sizes versus vortex (lower is better)', - // The card names formats by their display names, which call `vortex-file-compressed` `vortex`. - baseline: 'vortex', - }; + return ( + await getPool().query(text, [ + FILE_SIZE_SUMMARY_SCOPES.map((scope) => scope.benchmark), + FILE_SIZE_SUMMARY_SCOPES.map((scope) => scope.scale_factor), + COMPRESSION_BASELINE, + ]) + ).rows; } async function collectQuerySummary( @@ -1293,6 +1462,12 @@ async function collectQuerySummary( if (rankings.length === 0) { return null; } + // A series that has no time for a query, because it fails it or skips it (Lance skips TPC-DS + // q72), ranks below every series that ran all of them. The penalty alone makes a missing query + // cost only 2x the best time on it, so a series could rank first by skipping the slowest query. + // Its score and its (measured/total) coverage still show. The sort is stable, so each part keeps + // its score order. + rankings.sort((a, b) => Number(a.measured < a.total) - Number(b.measured < b.total)); return { type: 'queryBenchmark', title: 'Performance Summary', diff --git a/src/lib/bench-summary.ts b/src/lib/bench-summary.ts index 8889aac..480842a 100644 --- a/src/lib/bench-summary.ts +++ b/src/lib/bench-summary.ts @@ -99,6 +99,11 @@ type Source = { kind: Panel["kind"]; better: Panel["better"]; group: string; + /** + * The group's earlier names. The page reads production bench while it + * builds, so a deploy that renames a group still reads the old name. + */ + formerGroups?: readonly string[]; /** The rankings that list the systems, from the group's summary. */ rankings: (summary: Ranking) => unknown; /** The ranking's name for a system's format. */ @@ -107,6 +112,14 @@ type Source = { where?: Ranking; /** The ranking's field that holds the score. */ score: "score" | "ratio"; + /** + * Show a system's score even when bench measured it on only part of the + * workload. The file size summary covers the SQL suites' files, which + * have no Lance or Arrow IPC copies, so those two are partial there. A + * size ratio is to Parquet's size on the same dataset, so it stays + * comparable over the datasets the system has. + */ + partialScores?: true; /** The systems the panel shows. */ systems: readonly string[]; /** @@ -203,14 +216,22 @@ const SOURCES: (Source | Panel)[] = [ unit: "×", kind: "size", better: "lower", - group: "Compression Size", + group: "File Sizes", + formerGroups: ["Compression Size"], rankings: (s) => s.rankings, name: (f) => f, score: "ratio", + partialScores: true, systems: WITH_ARROW } ]; +// A source's group in bench's `/api/groups`, by its name or an earlier one. +const findGroup = (groups: unknown[], source: Source) => + [source.group, ...(source.formerGroups ?? [])] + .map((name) => groups.find((g) => isObject(g) && g.name === name)) + .find((g) => g !== undefined); + const isObject = (x: unknown): x is Ranking => typeof x === "object" && x !== null && !Array.isArray(x); const positive = (x: unknown): x is number => @@ -224,8 +245,9 @@ const measured = (r: Row) => r.score !== null && !(r.measured !== undefined && r.total !== undefined && r.measured < r.total); -// Bench's coverage passes through: results.js shows a system measured -// on only part of the workload as "Not measured". +// Bench's coverage passes through, unless the source shows partial scores: +// results.js shows a system measured on only part of the workload as +// "Not measured". function rowFor(source: Source, rankings: Ranking[], format: string) { const name = source.name(format); const where = Object.entries(source.where ?? {}); @@ -234,6 +256,7 @@ function rowFor(source: Source, rankings: Ranking[], format: string) { ); if (!r) return { score: null }; const score = r[source.score]; + if (source.partialScores) return { score: positive(score) ? score : null }; const measured = count(r.measured); const total = count(r.total); return { @@ -340,7 +363,7 @@ function panelFor( groups: unknown[], charts: Record ): Panel | null { - const group = groups.find((g) => isObject(g) && g.name === source.group); + const group = findGroup(groups, source); const summary = isObject(group) ? group.summary : undefined; const list = isObject(summary) ? source.rankings(summary) : undefined; if (!Array.isArray(list)) return null; @@ -473,7 +496,7 @@ export async function getBenchSummary( const slugs: [string, string][] = []; for (const source of SOURCES) { if (!("times" in source) || !source.times) continue; - const group = groups.find((g) => isObject(g) && g.name === source.group); + const group = findGroup(groups, source); const slug = isObject(group) ? group.slug : undefined; if (typeof slug !== "string" || !/^[\w.-]+$/.test(slug)) return fail(`group ${source.group} has no usable slug`); diff --git a/tests/unit/bench-summary.test.ts b/tests/unit/bench-summary.test.ts index 33a4568..fb4c0a2 100644 --- a/tests/unit/bench-summary.test.ts +++ b/tests/unit/bench-summary.test.ts @@ -342,7 +342,7 @@ describe("panelsFrom", () => { it("drops a score that is not a positive number", () => { const gs = groups(); - const parquet = group(gs, "Compression Size").summary.rankings?.find( + const parquet = group(gs, "File Sizes").summary.rankings?.find( (r) => r.name === "parquet" ); assert.ok(parquet); @@ -353,10 +353,36 @@ describe("panelsFrom", () => { }); }); + it("keeps a file size score bench measured on part of the datasets", () => { + // The SQL suites write no Lance or Arrow IPC files, so bench's file size + // summary has those two on part of its datasets. A size ratio is to + // Parquet on the same dataset, so the card keeps it. + const gs = groups(); + const lance = group(gs, "File Sizes").summary.rankings?.find( + (r) => r.name === "lance" + ); + assert.ok(lance); + lance.measured = 12; + lance.total = 13; + assert.deepEqual(panelsFrom(gs, charts())?.[6]?.rows[2], { + system: "Lance", + score: 1.9876533869960349 + }); + }); + + it("reads the file size panel from the group's earlier name", () => { + // A deploy that renames a bench group builds against production bench, + // which still serves the old name. + const gs = groups(); + const sizes = group(gs, "File Sizes"); + sizes.name = "Compression Size"; + assert.deepEqual(panelsFrom(gs, charts()), panelsFrom(groups(), charts())); + }); + it("gives null when a group is missing, so the card never shows part of the table", () => { for (const name of [ "Compression", - "Compression Size", + "File Sizes", "Random Access (NVMe)", "Clickbench", "TPC-H (NVMe) (SF=10)" diff --git a/tests/unit/fixtures/bench-groups.json b/tests/unit/fixtures/bench-groups.json index 6b4a158..bcc6f54 100644 --- a/tests/unit/fixtures/bench-groups.json +++ b/tests/unit/fixtures/bench-groups.json @@ -48,11 +48,11 @@ } }, { - "name": "Compression Size", + "name": "File Sizes", "slug": "csg.eyJrIjoiQ29tcHJlc3Npb25TaXplR3JvdXAifQ", "summary": { "type": "compressionSize", - "title": "Compression Size Summary", + "title": "File Sizes Summary", "rankings": [ { "name": "parquet",