From 3158ffea9acc19c29618f03799f6c683b328f7ee Mon Sep 17 00:00:00 2001 From: Joe Isaacs Date: Fri, 13 Nov 2026 14:23:32 +0000 Subject: [PATCH 1/2] Chart SQL benchmark file sizes on bench Adds `file_sizes` (migration 014) as the sixth family. The new `SQL File Size` group has one chart per benchmark and scale factor, and each chart has one line per Vortex format, summed over the benchmark's files. A line per file would be thousands of lines; this is about 16 charts of at most 2 lines. - Discovery and the summary skip-scan `idx_file_sizes_baseline` for the scopes rather than reading every file row. - The summary reuses the compression-size card: each format's total and its min/geomean/max ratio to vortex-file-compressed, each scope at its newest commit with a baseline, so nightly-only scopes count too. - A size summary with its own baseline gets no headline. The Parquet headline read "same size as Parquet" for this group. Signed-off-by: Joe Isaacs Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01UeniB9quRiv5fCx8AhzH9s Signed-off-by: Joe Isaacs --- CLAUDE.md | 2 +- docs/bench/CONTRACT.md | 6 +- docs/bench/architecture/design-decisions.md | 4 +- src/bench/lib/benchmark-docs.ts | 1 + src/bench/lib/descriptions.ts | 5 + src/bench/lib/families.test.ts | 4 +- src/bench/lib/families.ts | 29 ++-- src/bench/lib/group-charts.test.ts | 53 +++++++- src/bench/lib/groups.test.ts | 62 ++++++++- src/bench/lib/headline.test.ts | 15 +++ src/bench/lib/headline.ts | 5 +- src/bench/lib/queries.ts | 142 +++++++++++++++++++- src/bench/lib/site-files.ts | 1 + src/bench/lib/slug.ts | 23 +++- src/bench/lib/summary.test.ts | 1 + src/bench/lib/summary.ts | 92 ++++++++++++- 16 files changed, 420 insertions(+), 25 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 87c50023..41d4cf9e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -82,7 +82,7 @@ The previous site generations (v2 and v3) are documented in [`docs/bench/legacy. - **Numeric `?n=` snaps up to a rung of 25, 50, 100, 200, 500, or 1000; `?n=all` is the uncapped escape hatch.** Pages hydrate from the materialized latest-100 shard artifact by default; `?n=all` is an explicit opt-in (the chart's full-history zoom-out hop uses it once, and curl power users can request it). The numeric path is bounded by `MAX_NUMERIC_COMMIT_WINDOW` in [`src/bench/lib/window.ts`](src/bench/lib/window.ts) as a DoS-protection floor against `?n=99999999`, and a numeric `n` snaps up to the next rung of `COMMIT_WINDOW_RUNGS` there, so a chart or group has seven window URLs rather than a thousand, each a direct query that the CDN caches per URL. Do NOT raise the numeric cap, add rungs, or remove either without thinking about the DoS surface, and do NOT re-introduce a server-side commit cap on `?n=all` — visual downsampling happens client-side via LTTB on the visible commit range only. Only the default window is in the Data Cache, so every other window is a direct query that only the CDN shields. The read APIs and the chart page redirect (308) a non-default window to its one URL, with `n` spelled canonically and only the parameters the route reads (`canonicalWindowQuery` in `window.ts`), so adding a parameter or respelling `n` cannot get a request past the CDN. They also redirect every other spelling of a slug to the canonical one (`chartKeyToSlug`, `groupKeyToSlug`), since the CDN keys on the spelling, and the chart page cuts `engine` and `format` to the values bench knows, sorted. - **Don't refetch on every scope change.** Once a chart's payload is in memory, pan/zoom/slider/range-strip all rebuild in place via the in-memory LTTB pass on the cached payload ([`src/bench/components/chart/controller.ts`](src/bench/components/chart/controller.ts)). The single exception is the latest-100 to full-history zoom-out path: when the user first zooms past the materialized window, the chart lazy-fetches `?n=all` once and replaces the payload in place. - **Group docs links mirror the monorepo's `Benchmark::doc_path`.** [`src/bench/lib/benchmark-docs.ts`](src/bench/lib/benchmark-docs.ts) maps each SQL dataset name to the benchmark's explainer doc in `vortex-data/vortex`, which is the same path that the monorepo's PR benchmark comment uses. The producer does not put `doc_path` on the wire, and upstream does not enforce a path convention, so update the explicit mapping when a suite is added or its doc moves. Groups without a matching explainer render no docs link. Do not guess a path or link to an unrelated general guide. -- **Every group kind has a summary, by default.** [`src/bench/lib/summary.ts`](src/bench/lib/summary.ts) has no allowlist gate: a suite that lands in one of the five fact tables gets a rollup card from its first ingest. The three timing families (query, random access, vector search) rank through one `rankSeries` model, so a new suite in any of them needs no summary code at all. `collectGroupSummary`'s exhaustive switch makes a missing arm for a sixth fact table a compile error rather than a silently blank card. Do NOT reintroduce a per-dataset allowlist; the v2-era one is what left `spatialbench`, `fineweb`, `gharchive`, `appian`, `public-bi`, `clickbench-sorted`, and every vector-search group with no card. +- **Every group kind has a summary, by default.** [`src/bench/lib/summary.ts`](src/bench/lib/summary.ts) has no allowlist gate: a suite that lands in one of the six fact tables gets a rollup card from its first ingest. The three timing families (query, random access, vector search) rank through one `rankSeries` model, so a new suite in any of them needs no summary code at all. `collectGroupSummary`'s exhaustive switch makes a missing arm for a seventh fact table a compile error rather than a silently blank card. Do NOT reintroduce a per-dataset allowlist; the v2-era one is what left `spatialbench`, `fineweb`, `gharchive`, `appian`, `public-bi`, `clickbench-sorted`, and every vector-search group with no card. - **A summary ranks the whole group, never one chart.** Random access is the cautionary case: the producer emits `dataset` as `{dataset}/{pattern}`, so the group holds ~nine charts, and the old summary published the alphabetically first chart's raw times under the group-wide title "Random Access Performance" — reporting `lance` at 352us when it is over 1ms on most of the other charts. Sum chart medians by dataset, then rank the dataset totals. Include the legacy taxi chart in the taxi total. Apply a penalty when a series lacks one dataset total. Report dataset coverage through `measured`/`total`. The secondary runtime is the arithmetic mean of complete dataset totals. - **Don't write a server-side classifier for live ingest.** The emitter produces structured records directly. Classifying loose name strings at read time was the v2-era weakness every later generation existed to escape; it belongs nowhere in the live pipeline. - **Every root-relative bench URL goes through `benchPath()`.** A link, a fetch, an icon or a selector that names one of bench's paths must work on bench.vortex.dev (from the root) and everywhere else (under `/bench`); see "Repository layout". A path read from `location.pathname` already carries the base. diff --git a/docs/bench/CONTRACT.md b/docs/bench/CONTRACT.md index c2e8baa9..e27c06ee 100644 --- a/docs/bench/CONTRACT.md +++ b/docs/bench/CONTRACT.md @@ -75,7 +75,8 @@ rows in the `String Encoding` groups and compares each encoder with `fsst`, so t `file_size` records are not written by `vortex-bench`: the monorepo's `scripts/capture-file-sizes.py` writes one per Vortex file a SQL benchmark leaves on disk, with `benchmark` (the `vx-bench` subcommand), `scale_factor`, `format`, `file` (the path below the format -directory) and `value_bytes`. The read service does not chart them; CI reads them back as the develop +directory) and `value_bytes`. The read service charts each benchmark and scale factor in the +`SQL File Size` group, one line per format summed over its files. CI reads them back as the develop baseline for the PR comment's file-size section. `random_access_time` records include `open_mode`. The value is `cached` for a reused accessor @@ -160,6 +161,5 @@ For wire/record *shape* changes, also update the producer snapshot fixtures in t the same commit. A new record kind with its own table, as `file_size` was in migration 014, changes no existing -shape, so it needs no bump: the read service never reads the new table, and older writers never -send the kind. Its migration must be applied before a writer sends it, because each JSONL file +shape, so it needs no bump: no existing read changes, and older writers never send the kind. Its migration must be applied before a writer sends it, because each JSONL file lands in one transaction and an unknown table fails the whole file. diff --git a/docs/bench/architecture/design-decisions.md b/docs/bench/architecture/design-decisions.md index f5177909..b57d8600 100644 --- a/docs/bench/architecture/design-decisions.md +++ b/docs/bench/architecture/design-decisions.md @@ -18,9 +18,9 @@ reasoning behind them. Each entry is "decision → why → where." Topic docs ## Data model **One fact table per measurement family (not one wide table).** -The five families have genuinely different dimensions. A single wide table would +The six families have genuinely different dimensions. A single wide table would bloat every row with NULL columns it doesn't use, or split one chart's data across -multiple rows that then need re-joining. Five focused tables keep each row dense +multiple rows that then need re-joining. Six focused tables keep each row dense and each chart query a single-table scan. → `migrations/001_initial_schema.sql`, `server/src/schema.rs`. diff --git a/src/bench/lib/benchmark-docs.ts b/src/bench/lib/benchmark-docs.ts index 096d3397..8fd75d70 100644 --- a/src/bench/lib/benchmark-docs.ts +++ b/src/bench/lib/benchmark-docs.ts @@ -88,6 +88,7 @@ export function groupDocPath(key: GroupKey): string | null { case 'QueryGroup': return QUERY_DOCS[key.dataset] ?? null; case 'VectorSearchGroup': + case 'FileSizeGroup': return null; } } diff --git a/src/bench/lib/descriptions.ts b/src/bench/lib/descriptions.ts index 5fe7d638..2aaa6de0 100644 --- a/src/bench/lib/descriptions.ts +++ b/src/bench/lib/descriptions.ts @@ -55,6 +55,11 @@ function staticDescription(name: string): string | null { return 'Measures Vortex file write and scan throughput for each string encoder'; case 'String Encoding Size': return 'Compares Vortex file sizes for each string encoder'; + case 'SQL File Size': + return ( + 'Tracks the total size of the Vortex files each SQL benchmark writes, per benchmark and ' + + 'scale factor' + ); case 'Clickbench': return ( "ClickHouse's analytical benchmark suite testing real-world query patterns on web " + diff --git a/src/bench/lib/families.test.ts b/src/bench/lib/families.test.ts index d8de9884..86c3dedc 100644 --- a/src/bench/lib/families.test.ts +++ b/src/bench/lib/families.test.ts @@ -5,13 +5,14 @@ import { describe, expect, it } from 'vitest'; import { FAMILIES, familyForChartKind, familyForGroupKind } from './families'; describe('FAMILIES registry', () => { - it('lists the five fact tables in family.rs declaration order', () => { + it('lists the six fact tables in family.rs declaration order, then file_sizes', () => { expect(FAMILIES.map((f) => f.tableName)).toEqual([ 'query_measurements', 'compression_times', 'compression_sizes', 'random_access_times', 'vector_search_runs', + 'file_sizes', ]); }); @@ -29,6 +30,7 @@ describe('FAMILIES registry', () => { ['cs', 'csg'], ['rat', 'rag'], ['vsr', 'vsg'], + ['fs', 'fsg'], ]); }); }); diff --git a/src/bench/lib/families.ts b/src/bench/lib/families.ts index 9f8eb8a0..6e196ee8 100644 --- a/src/bench/lib/families.ts +++ b/src/bench/lib/families.ts @@ -4,23 +4,25 @@ /** * Per-fact-table registry, the TypeScript port of `server/src/family.rs`. * - * Each of the five fact tables is a [`Family`] that ties together the Postgres + * Each of the six fact tables is a [`Family`] that ties together the Postgres * table name and the chart and group slug prefixes. The read endpoints * dispatch through this registry rather than hand-listing the families, so the * slug prefixes consumed by [`./slug`] have a single source of truth. The read * queries name their tables in static SQL. * * The order of [`FAMILIES`] mirrors `family.rs`'s `FAMILIES` constant and the - * DDL apply order in `migrations/001_initial_schema.sql`. + * DDL apply order in `migrations/001_initial_schema.sql`, with `file_sizes` + * (`migrations/014`) last. */ -/** The five fact-table names, spelled exactly as in `migrations/001`. */ +/** The six fact-table names, spelled exactly as in their migrations. */ type FamilyTable = | 'query_measurements' | 'compression_times' | 'compression_sizes' | 'random_access_times' - | 'vector_search_runs'; + | 'vector_search_runs' + | 'file_sizes'; /** Discriminant (`k`) of a [`./slug`].`ChartKey`; one per family. */ export type ChartKind = @@ -28,7 +30,8 @@ export type ChartKind = | 'CompressionTime' | 'CompressionSize' | 'RandomAccess' - | 'VectorSearch'; + | 'VectorSearch' + | 'FileSize'; /** Discriminant (`k`) of a [`./slug`].`GroupKey`; one per family. */ export type GroupKind = @@ -36,11 +39,12 @@ export type GroupKind = | 'CompressionTimeGroup' | 'CompressionSizeGroup' | 'RandomAccessGroup' - | 'VectorSearchGroup'; + | 'VectorSearchGroup' + | 'FileSizeGroup'; /** One fact-table family: its table name, slug prefixes, and key discriminants. */ export interface Family { - /** Postgres table name, matching `migrations/001_initial_schema.sql`. */ + /** Postgres table name, matching its migration. */ tableName: FamilyTable; /** Slug prefix for individual charts in this family (e.g. `qm`). */ chartSlugPrefix: string; @@ -53,8 +57,8 @@ export interface Family { } /** - * All five fact-table families, in the order `family.rs`'s `FAMILIES` declares - * them (which is also the `migrations/001` DDL apply order). + * All six fact-table families, in the order `family.rs`'s `FAMILIES` declares + * them (which is also the `migrations/001` DDL apply order), then `file_sizes`. */ export const FAMILIES: readonly Family[] = [ { @@ -92,6 +96,13 @@ export const FAMILIES: readonly Family[] = [ chartKind: 'VectorSearch', groupKind: 'VectorSearchGroup', }, + { + tableName: 'file_sizes', + chartSlugPrefix: 'fs', + groupSlugPrefix: 'fsg', + chartKind: 'FileSize', + groupKind: 'FileSizeGroup', + }, ]; /** Look up the family that owns a chart-key discriminant. */ diff --git a/src/bench/lib/group-charts.test.ts b/src/bench/lib/group-charts.test.ts index 820ce302..dc4f5b2d 100644 --- a/src/bench/lib/group-charts.test.ts +++ b/src/bench/lib/group-charts.test.ts @@ -88,6 +88,11 @@ describe('seriesFor', () => { { flavor: 'vortex-turboquant' }, { key: 'vortex-turboquant', engine: undefined, format: undefined }, ], + [ + 'FileSize', + { format: 'vortex-compact' }, + { key: 'vortex-compact', engine: undefined, format: 'vortex-compact' }, + ], ] as const)('names the %s series', (kind, cols, expected) => { expect(seriesFor(kind, { commit_sha: 'a', value: 1, ...cols })).toEqual(expected); }); @@ -468,6 +473,29 @@ describe.skipIf(!databaseAvailable)('groupChartPayloads at the window edges', () [measurementId, sha, dataset, variant, valueBytes], ); } + // File sizes: TPC-H SF=1 has two files per format on C0, E and F (compact only from E); + // SF=10 has one file on F. + for (const [sha, scaleFactor, format, file, valueBytes] of [ + [C0, '1.0', 'vortex-file-compressed', 'a.vortex', 100], + [C0, '1.0', 'vortex-file-compressed', 'b.vortex', 10], + [E, '1.0', 'vortex-file-compressed', 'a.vortex', 101], + [E, '1.0', 'vortex-file-compressed', 'b.vortex', 11], + [E, '1.0', 'vortex-compact', 'a.vortex', 80], + [E, '1.0', 'vortex-compact', 'b.vortex', 8], + [F, '1.0', 'vortex-file-compressed', 'a.vortex', 102], + [F, '1.0', 'vortex-file-compressed', 'b.vortex', 12], + [F, '1.0', 'vortex-compact', 'a.vortex', 79], + [F, '1.0', 'vortex-compact', 'b.vortex', 7], + [F, '10.0', 'vortex-file-compressed', 'a.vortex', 1000], + ] as const) { + measurementId += 1; + await getPool().query( + `INSERT INTO file_sizes + (measurement_id, commit_sha, benchmark, scale_factor, format, file, value_bytes) + VALUES ($1, $2, 'tpch', $3, $4, $5, $6)`, + [measurementId, sha, scaleFactor, format, file, valueBytes], + ); + } }); afterAll(async () => { @@ -563,10 +591,17 @@ describe.skipIf(!databaseAvailable)('groupChartPayloads at the window edges', () { k: 'CompressionSize', dataset: 'sy', dataset_variant: 'v2' }, ], ], + [ + { k: 'FileSizeGroup' }, + [ + { k: 'FileSize', benchmark: 'tpch', scale_factor: '1.0' }, + { k: 'FileSize', benchmark: 'tpch', scale_factor: '10.0' }, + ], + ], ]; it.each(['1', '2', '3', '4', 'all'])( - 'matches chartPayload for random-access, vector-search and compression-size groups at ?n=%s', + 'matches chartPayload for random-access, vector-search, compression-size and file-size groups at ?n=%s', async (n) => { const window = commitWindow(n); for (const [group, keys] of OTHER_FAMILIES) { @@ -608,6 +643,22 @@ describe.skipIf(!databaseAvailable)('groupChartPayloads at the window edges', () expect(short?.series).toEqual({ fl: [600] }); }); + it('sums each file-size chart over its files, one line per format', async () => { + const [sf1, sf10] = await groupChartPayloads( + ...defined(OTHER_FAMILIES[3]), + parseCommitWindow('all'), + ); + expect(sf1?.display_name).toBe('tpch'); + expect(sf1?.unit_kind).toBe('bytes'); + expect(sf1?.series).toEqual({ + 'vortex-compact': [null, null, null, null, 88, 86], + 'vortex-file-compressed': [110, null, null, null, 112, 114], + }); + expect(sf10?.display_name).toBe('tpch sf=10.0'); + expect(sf10?.commits.map((c) => c.sha)).toEqual([F]); + expect(sf10?.series).toEqual({ 'vortex-file-compressed': [1000] }); + }); + it('keeps sibling groups apart', async () => { const window = parseCommitWindow('all'); const [g] = await groupChartPayloads(G, [queryChart(G, 1)], window); diff --git a/src/bench/lib/groups.test.ts b/src/bench/lib/groups.test.ts index 3784b8d7..f4fb364f 100644 --- a/src/bench/lib/groups.test.ts +++ b/src/bench/lib/groups.test.ts @@ -26,7 +26,7 @@ import { import { groupDescription } from './descriptions'; import { READ_API_CACHE_CONTROL } from './cache'; import { collectGroupSummary } from './summary'; -import { chartKeyToSlug, groupKeyFromSlug, groupKeyToSlug } from './slug'; +import { chartKeyFromSlug, chartKeyToSlug, groupKeyFromSlug, groupKeyToSlug } from './slug'; import { type BenchDatabase, commitUrl, @@ -445,7 +445,7 @@ describe.skipIf(!databaseAvailable)('summary math fidelity (testcontainers Postg beforeEach(async () => { await getPool().query( `TRUNCATE compression_times, compression_sizes, query_measurements, - random_access_times, vector_search_runs, commits`, + random_access_times, vector_search_runs, file_sizes, commits`, ); }); @@ -626,6 +626,64 @@ describe.skipIf(!databaseAvailable)('summary math fidelity (testcontainers Postg ]); }); + it('discovers one file-size chart per benchmark and scale factor, with a summary', async () => { + const develop = 'a'.repeat(40); + const nightly = 'b'.repeat(40); + await insertCommit(develop, '2026-04-23T12:00:00Z'); + await insertCommit(nightly, '2026-04-24T03:00:00Z'); + // TPC-H at SF 1 and 10 from develop, and SF 100 from nightly, the newest commit. Clickbench + // has no scale-factor directory, so it is recorded as 1.0. + for (const [sha, benchmark, scaleFactor, format, file, bytes] of [ + [develop, 'tpch', '1.0', 'vortex-file-compressed', 'lineitem.vortex', 300], + [develop, 'tpch', '1.0', 'vortex-file-compressed', 'orders.vortex', 100], + [develop, 'tpch', '1.0', 'vortex-compact', 'lineitem.vortex', 150], + [develop, 'tpch', '1.0', 'vortex-compact', 'orders.vortex', 50], + [develop, 'tpch', '10.0', 'vortex-file-compressed', 'lineitem.vortex', 4000], + [develop, 'tpch', '10.0', 'vortex-compact', 'lineitem.vortex', 1000], + [nightly, 'tpch', '100', 'vortex-file-compressed', 'lineitem.vortex', 40000], + [develop, 'clickbench', '1.0', 'vortex-file-compressed', 'hits_0.vortex', 1000], + ] as const) { + await getPool().query( + `INSERT INTO file_sizes + (measurement_id, commit_sha, benchmark, scale_factor, format, file, value_bytes) + VALUES ($1, $2, $3, $4, $5, $6, $7)`, + [nextId(), sha, benchmark, scaleFactor, format, file, bytes], + ); + } + + const groups = await collectGroups(); + expect(groups.map((group) => group.name)).toEqual(['SQL File Size']); + const [group] = groups; + expect(groupKeyFromSlug(defined(group).slug)).toEqual({ k: 'FileSizeGroup' }); + expect(defined(group).charts.map((chart) => chart.name)).toEqual([ + 'clickbench', + 'tpch', + 'tpch sf=10.0', + 'tpch sf=100', + ]); + expect(chartKeyFromSlug(defined(defined(group).charts[3]).slug)).toEqual({ + k: 'FileSize', + benchmark: 'tpch', + scale_factor: '100', + }); + // Compact is half the size at SF 1 and a quarter at SF 10. Totals cover the scopes both + // formats have: 200 + 1000 against 400 + 4000. + expect(defined(group).summary).toMatchObject({ + type: 'compressionSize', + baseline: 'vortex', + rankings: [ + { + name: 'vortex-compact', + minRatio: 0.25, + ratio: expect.closeTo(0.5 ** 1.5), + maxRatio: 0.5, + totalBytes: 1200, + }, + { name: 'vortex-file-compressed', ratio: 1, compressionRatio: null, totalBytes: 4400 }, + ], + }); + }); + it('ranks random access over every chart, not the alphabetically first one', async () => { // The bug this pins: the old summary walked the group's chart links and // published the first populated chart's raw times under the group-wide diff --git a/src/bench/lib/headline.test.ts b/src/bench/lib/headline.test.ts index 7db1ee86..e9897649 100644 --- a/src/bench/lib/headline.test.ts +++ b/src/bench/lib/headline.test.ts @@ -193,6 +193,21 @@ describe('groupHeadline', () => { expect(h?.parts[0]?.tone).toBe(tone); }); + it('has no headline for a size summary with its own baseline', () => { + // The SQL file size summary compares with Vortex itself, which would read "same size as Parquet". + expect( + groupHeadline({ + type: 'compressionSize', + title: 's', + explanation: 'e', + baseline: 'vortex', + rankings: [ + { name: VX, minRatio: 1, ratio: 1, maxRatio: 1, compressionRatio: null, totalBytes: 1 }, + ], + }), + ).toBeNull(); + }); + it('reads hot and cold random access against Parquet', () => { expect(line(ra([r(VX, 1), r('parquet', 60.4)], [r(VX, 1), r('parquet', 32)]))).toBe( 'hot 60× · cold 32× faster than Parquet', diff --git a/src/bench/lib/headline.ts b/src/bench/lib/headline.ts index a08bd4ca..3d95f2e8 100644 --- a/src/bench/lib/headline.ts +++ b/src/bench/lib/headline.ts @@ -91,7 +91,10 @@ export function groupHeadline(summary: Summary | undefined): Headline | null { })), ); case 'compressionSize': - return sizeHeadline(summary.rankings.find((row) => row.name === VORTEX)?.ratio); + // A summary with its own baseline compares with that, not Parquet, so it has no headline. + return summary.baseline === undefined + ? sizeHeadline(summary.rankings.find((row) => row.name === VORTEX)?.ratio) + : null; case 'vectorSearch': return vectorSearchHeadline(summary.rankings); default: diff --git a/src/bench/lib/queries.ts b/src/bench/lib/queries.ts index 5688f605..cc67bf42 100644 --- a/src/bench/lib/queries.ts +++ b/src/bench/lib/queries.ts @@ -354,6 +354,7 @@ type CompressionTimeKey = Extract; type CompressionSizeKey = Extract; type RandomAccessKey = Extract; type VectorSearchKey = Extract; +type FileSizeKey = Extract; /** Row shape of a fact-table pass; `value` is read `::float8`. */ type ValueRow = { commit_sha: string; value: number }; @@ -396,6 +397,8 @@ export function seriesFor( } case 'VectorSearch': return { key: `${row.flavor}`, engine: undefined, format: undefined }; + case 'FileSize': + return { key: `${row.format}`, engine: undefined, format: row.format }; } } @@ -430,12 +433,22 @@ function chartDisplayName(key: ChartKey): string { return key.dataset; case 'VectorSearch': return `${key.dataset} / ${key.layout} (threshold=${key.threshold})`; + case 'FileSize': + return fileSizeChartName(key.benchmark, key.scale_factor); } } +/** + * A file-size chart's name: the benchmark, with its scale factor unless it is the default `1.0`. + * The monorepo's capture script records `1.0` for a benchmark with no scale-factor directory. + */ +function fileSizeChartName(benchmark: string, scaleFactor: string): string { + return scaleFactor === '1.0' ? benchmark : `${benchmark} sf=${scaleFactor}`; +} + /** A chart's base `unit_kind`: sizes are bytes, everything else is a time. */ function chartUnitKind(key: ChartKey): UnitKind { - return key.k === 'CompressionSize' ? 'bytes' : 'time_ns'; + return key.k === 'CompressionSize' || key.k === 'FileSize' ? 'bytes' : 'time_ns'; } async function collectQueryChart( @@ -636,6 +649,48 @@ async function collectVectorSearchChart( return acc.finish(chartDisplayName(key), chartUnitKind(key), seeded.history); } +/** + * One line per Vortex format: the total size of the benchmark's files at each commit. The table + * has a row per file, which would be thousands of lines across the group. + */ +async function collectFileSizeChart( + key: FileSizeKey, + window: CommitWindow, +): Promise { + const { benchmark, scale_factor } = key; + const seeded = await seededCommitsInWindow( + (p) => + `SELECT MIN(c2.timestamp) + FROM file_sizes f2 + JOIN ${BRANCH_COMMITS} c2 ON c2.commit_sha = f2.commit_sha + WHERE f2.benchmark = ${p.bind(benchmark)} + AND f2.scale_factor = ${p.bind(scale_factor)}`, + window, + ); + if (seeded.commits.length === 0) { + return null; + } + const acc = new SeriesAccumulator(); + acc.seedCommits(seeded.commits); + + const params = new QueryParams(); + const text = ` + SELECT f.commit_sha, + f.format, SUM(f.value_bytes)::float8 AS value + FROM file_sizes f + JOIN ${BRANCH_COMMITS} c USING (commit_sha) + WHERE f.benchmark = ${params.bind(benchmark)} + AND f.scale_factor = ${params.bind(scale_factor)}${factWindowFilter(params, window)} + GROUP BY f.commit_sha, c.timestamp, f.format + ORDER BY c.timestamp, f.format + `; + const rows = (await getPool().query(text, params.values)).rows; + for (const row of rows) { + recordRow(acc, key.k, row); + } + return acc.finish(chartDisplayName(key), chartUnitKind(key), seeded.history); +} + /** * Build the JSON payload for one chart by key, or `null` when the chart has no * data (callers render a 404). The shared implementation behind @@ -658,6 +713,8 @@ export function chartPayload(key: ChartKey, window: CommitWindow): Promise + `${alias}.benchmark = ch.benchmark AND ${alias}.scale_factor = ch.scale_factor`; + block( + ['format'], + keys.map((_, i) => i + 1), + [ + ['benchmark', 'text', keys.map((k) => k.benchmark)], + ['scale_factor', 'text', keys.map((k) => k.scale_factor)], + ], + `SELECT MIN(c2.timestamp) + FROM file_sizes f2 + JOIN ${BRANCH_COMMITS} c2 ON c2.commit_sha = f2.commit_sha + WHERE ${pins('f2')}`, + `SELECT f.commit_sha, + f.format, SUM(f.value_bytes)::float8 AS value + FROM file_sizes f + JOIN ${BRANCH_COMMITS} c USING (commit_sha) + WHERE ${pins('f')}${factWindowFilter(params, window)} + GROUP BY f.commit_sha, f.format`, + ); + break; + } } const text = ` WITH commit_times AS ( @@ -1159,6 +1242,7 @@ const GROUP_ORDER: readonly string[] = [ 'Compression Size', 'String Encoding', 'String Encoding Size', + 'SQL File Size', 'Random Access (NVMe)', 'Random Access (S3)', 'Clickbench', @@ -1573,6 +1657,60 @@ async function collectVectorSearchGroups(): Promise { return groups; } +type FileSizeScopeRow = { benchmark: string; scale_factor: string }; + +/** + * The `SQL File Size` group: one chart per benchmark and scale factor, or `[]` if the table is + * empty. Charts order by benchmark, then by numeric scale factor, so `10.0` follows `1.0`. + */ +async function collectFileSizeGroup(): Promise { + // A recursive skip scan of `idx_file_sizes_baseline`: one descent per scope instead of a scan + // of every file of every commit. + const text = ` + WITH RECURSIVE scopes AS ( + (SELECT f.benchmark, f.scale_factor + FROM file_sizes f + ORDER BY f.benchmark, f.scale_factor + LIMIT 1) + UNION ALL + SELECT nxt.benchmark, nxt.scale_factor + FROM scopes s + CROSS JOIN LATERAL ( + SELECT f.benchmark, f.scale_factor + FROM file_sizes f + WHERE (f.benchmark, f.scale_factor) > (s.benchmark, s.scale_factor) + ORDER BY f.benchmark, f.scale_factor + LIMIT 1 + ) nxt + ) + SELECT benchmark, scale_factor FROM scopes + `; + const rows = (await getPool().query(text)).rows; + if (rows.length === 0) { + return []; + } + rows.sort( + (a, b) => + compareCodeUnits(a.benchmark, b.benchmark) || + Number(a.scale_factor) - Number(b.scale_factor) || + compareCodeUnits(a.scale_factor, b.scale_factor), + ); + return [ + { + name: 'SQL File Size', + slug: groupKeyToSlug({ k: 'FileSizeGroup' }), + charts: rows.map((row) => ({ + name: fileSizeChartName(row.benchmark, row.scale_factor), + slug: chartKeyToSlug({ + k: 'FileSize', + benchmark: row.benchmark, + scale_factor: row.scale_factor, + }), + })), + }, + ]; +} + /** Dispatch to the discovery pass for one family, the TS analogue of the Rust * `(family.collect_groups)(conn)` registry indirection. */ function discoverGroups(groupKind: GroupKind): Promise { @@ -1587,6 +1725,8 @@ function discoverGroups(groupKind: GroupKind): Promise { return collectRandomAccessGroup(); case 'VectorSearchGroup': return collectVectorSearchGroups(); + case 'FileSizeGroup': + return collectFileSizeGroup(); } } diff --git a/src/bench/lib/site-files.ts b/src/bench/lib/site-files.ts index 61d70c2c..8a5dd95b 100644 --- a/src/bench/lib/site-files.ts +++ b/src/bench/lib/site-files.ts @@ -231,6 +231,7 @@ export function chartSeriesLabel(kind: ChartKind, name: string): string { return `${FORMATS.get(format) ?? format} (${detail})`; case 'CompressionSize': case 'VectorSearch': + case 'FileSize': return FORMATS.get(name) ?? name; } } diff --git a/src/bench/lib/slug.ts b/src/bench/lib/slug.ts index 6373b6f6..89d6a97d 100644 --- a/src/bench/lib/slug.ts +++ b/src/bench/lib/slug.ts @@ -32,7 +32,9 @@ export type ChartKey = | { k: 'CompressionTime'; dataset: string; dataset_variant: string | null } | { k: 'CompressionSize'; dataset: string; dataset_variant: string | null } | { k: 'RandomAccess'; dataset: string } - | { k: 'VectorSearch'; dataset: string; layout: string; threshold: number }; + | { k: 'VectorSearch'; dataset: string; layout: string; threshold: number } + // One SQL benchmark's data at one scale factor; its series are the Vortex formats. + | { k: 'FileSize'; benchmark: string; scale_factor: string }; /** Strongly-typed group key parsed from a slug. Discriminated on `k`. */ export type GroupKey = @@ -47,7 +49,8 @@ export type GroupKey = | { k: 'CompressionTimeGroup'; suite?: 'strings' } | { k: 'CompressionSizeGroup'; suite?: 'strings' } | { k: 'RandomAccessGroup'; storage: RandomAccessStorage } - | { k: 'VectorSearchGroup'; dataset: string; layout: string }; + | { k: 'VectorSearchGroup'; dataset: string; layout: string } + | { k: 'FileSizeGroup' }; const CHART_KINDS: readonly ChartKind[] = [ 'QueryMeasurement', @@ -55,6 +58,7 @@ const CHART_KINDS: readonly ChartKind[] = [ 'CompressionSize', 'RandomAccess', 'VectorSearch', + 'FileSize', ]; const GROUP_KINDS: readonly GroupKind[] = [ @@ -63,6 +67,7 @@ const GROUP_KINDS: readonly GroupKind[] = [ 'CompressionSizeGroup', 'RandomAccessGroup', 'VectorSearchGroup', + 'FileSizeGroup', ]; function encodePayload(json: string): string { @@ -111,6 +116,9 @@ export function chartKeyToSlug(key: ChartKey): string { case 'VectorSearch': ordered = { k: key.k, dataset: key.dataset, layout: key.layout, threshold: key.threshold }; break; + case 'FileSize': + ordered = { k: key.k, benchmark: key.benchmark, scale_factor: key.scale_factor }; + break; } return `${prefix}.${encodePayload(JSON.stringify(ordered))}`; } @@ -213,6 +221,12 @@ export function chartKeyFromSlug(slug: string): ChartKey { layout: reqString(obj, 'layout'), threshold: reqNumber(obj, 'threshold'), }; + case 'FileSize': + return { + k: kind, + benchmark: reqString(obj, 'benchmark'), + scale_factor: reqString(obj, 'scale_factor'), + }; } } @@ -243,6 +257,9 @@ export function groupKeyToSlug(key: GroupKey): string { case 'CompressionSizeGroup': ordered = key.suite === undefined ? { k: key.k } : { k: key.k, suite: key.suite }; break; + case 'FileSizeGroup': + ordered = { k: key.k }; + break; } return `${prefix}.${encodePayload(JSON.stringify(ordered))}`; } @@ -287,5 +304,7 @@ export function groupKeyFromSlug(slug: string): GroupKey { } return { k: kind, suite: 'strings' }; } + case 'FileSizeGroup': + return { k: kind }; } } diff --git a/src/bench/lib/summary.test.ts b/src/bench/lib/summary.test.ts index 63d2918e..d053b1f4 100644 --- a/src/bench/lib/summary.test.ts +++ b/src/bench/lib/summary.test.ts @@ -753,6 +753,7 @@ describe('timing summaries (shared ranking model)', () => { { k: 'CompressionSizeGroup' }, { k: 'RandomAccessGroup', storage: 'nvme' }, { k: 'VectorSearchGroup', dataset: 'd', layout: 'l' }, + { k: 'FileSizeGroup' }, ]; expect(keys.map((key) => key.k).sort()).toEqual([...GROUP_KINDS].sort()); for (const key of keys) { diff --git a/src/bench/lib/summary.ts b/src/bench/lib/summary.ts index d58a176b..43edd969 100644 --- a/src/bench/lib/summary.ts +++ b/src/bench/lib/summary.ts @@ -8,10 +8,10 @@ * variant. Compression summaries compare the configured formats with Parquet. * * **Every group kind has a summary.** There is no allowlist gate: a benchmark - * suite that lands in one of the five fact tables gets a rollup card the moment + * suite that lands in one of the six fact tables gets a rollup card the moment * its first rows are ingested. The timing-shaped families (query, random * access, vector search) all rank through the one [`rankSeries`] model below, - * so a new suite in any of them needs no summary code at all. Adding a sixth + * so a new suite in any of them needs no summary code at all. Adding a seventh * fact table is the only case that needs a new arm here, and * [`collectGroupSummary`]'s exhaustive switch makes omitting it a compile * error rather than a silently missing card. @@ -208,6 +208,8 @@ export function collectGroupSummary(key: GroupKey): Promise { return collectRandomAccessSummary(key.storage); case 'VectorSearchGroup': return collectVectorSearchSummary(key.dataset, key.layout); + case 'FileSizeGroup': + return collectFileSizeSummary(); } } @@ -1021,6 +1023,92 @@ async function collectStringEncodingSizeSummary(): Promise { }; } +/** The format every other SQL benchmark file format is compared with. */ +const FILE_SIZE_BASELINE = 'vortex-file-compressed'; + +/** + * The `SQL File Size` rollup: each Vortex format's total file size against + * `vortex-file-compressed`, per benchmark scope (benchmark and scale factor) at that scope's + * newest commit with a baseline. Scopes come from nightly as well as develop runs, so each one + * uses its own newest commit. File sizes record no Arrow size, so the Arrow ratio is empty. + */ +async function collectFileSizeSummary(): Promise { + // `scopes` is a skip scan of `idx_file_sizes_baseline`. `latest` walks the read branch's + // commits newest first and probes that index, so it stops at the newest commit with a baseline. + const text = ` + WITH RECURSIVE scopes AS ( + (SELECT f.benchmark, f.scale_factor + FROM file_sizes f + ORDER BY f.benchmark, f.scale_factor + LIMIT 1) + UNION ALL + SELECT nxt.benchmark, nxt.scale_factor + FROM scopes s + CROSS JOIN LATERAL ( + SELECT f.benchmark, f.scale_factor + FROM file_sizes f + WHERE (f.benchmark, f.scale_factor) > (s.benchmark, s.scale_factor) + ORDER BY f.benchmark, f.scale_factor + LIMIT 1 + ) nxt + ), + latest AS ( + SELECT s.benchmark, s.scale_factor, l.commit_sha + FROM scopes s + CROSS JOIN LATERAL ( + SELECT c.commit_sha + FROM ${BRANCH_COMMITS} c + WHERE EXISTS ( + SELECT 1 + FROM file_sizes f + WHERE f.benchmark = s.benchmark + AND f.scale_factor = s.scale_factor + AND f.format = $1 + AND f.commit_sha = c.commit_sha) + ORDER BY c.timestamp DESC, c.commit_sha DESC + LIMIT 1 + ) l + ), + totals AS ( + SELECT f.benchmark, f.scale_factor, f.format, SUM(f.value_bytes)::float8 AS bytes + FROM file_sizes f + JOIN latest l + ON l.benchmark = f.benchmark + AND l.scale_factor = f.scale_factor + AND l.commit_sha = f.commit_sha + GROUP BY f.benchmark, f.scale_factor, f.format + ) + SELECT t.format AS format, + t.benchmark AS dataset, + t.scale_factor AS "datasetVariant", + t.bytes AS "valueBytes", + b.bytes AS "baselineBytes", + NULL::float8 AS "uncompressedBytes" + FROM totals t + JOIN totals b + ON b.benchmark = t.benchmark + AND b.scale_factor = t.scale_factor + AND b.format = $1 + WHERE t.bytes > 0 + AND b.bytes > 0 + ORDER BY t.format, t.benchmark, t.scale_factor + `; + const rows = (await getPool().query(text, [FILE_SIZE_BASELINE])).rows; + const rankings = compressionSizeRankings(rows); + if (rankings.length === 0) { + return null; + } + return { + type: 'compressionSize', + title: 'SQL File Size Summary', + rankings, + explanation: + 'Total size of the Vortex files each SQL benchmark writes, and geometric means of sizes versus vortex (lower is better)', + // The card names formats by their display names, which call `vortex-file-compressed` `vortex`. + baseline: 'vortex', + }; +} + async function collectQuerySummary( dataset: string, datasetVariant: string | null, From f61e347bc3638e81a0e89ffc0298efdf78423777 Mon Sep 17 00:00:00 2001 From: Joe Isaacs Date: Fri, 13 Nov 2026 14:42:41 +0000 Subject: [PATCH 2/2] Bump the group payload cache version for the file size group The group list and group bundles now include the `SQL File Size` group and its summary, so payloads cached under `v12` by the deployed code are not what this code returns for the same data. Without a new key, a deployment of this change serves the old cached group list, without the group, until the next ingest or the one-day backstop, and shares the key with the old deployment. Signed-off-by: Joe Isaacs Co-Authored-By: Claude Opus 5.5 Claude-Session: https://claude.ai/code/session_01UeniB9quRiv5fCx8AhzH9s Signed-off-by: Joe Isaacs --- src/bench/lib/data-cache.test.ts | 4 ++-- src/bench/lib/data-cache.ts | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/bench/lib/data-cache.test.ts b/src/bench/lib/data-cache.test.ts index c77e9b75..e1a8b18b 100644 --- a/src/bench/lib/data-cache.test.ts +++ b/src/bench/lib/data-cache.test.ts @@ -69,9 +69,9 @@ describe('data-cache wrappers', () => { expect(BENCH_DATA_TAG).toBe('bench-data'); expect(DATA_CACHE_BACKSTOP_SECONDS).toBe(86400); expect(cacheCalls.map((call) => call.keyParts)).toContainEqual([ - 'data-cache:group-charts:v12:n100', + 'data-cache:group-charts:v13:n100', ]); - expect(cacheCalls.map((call) => call.keyParts)).toContainEqual(['data-cache:groups:v12']); + expect(cacheCalls.map((call) => call.keyParts)).toContainEqual(['data-cache:groups:v13']); for (const call of cacheCalls) { expect(call.options.tags).toEqual([BENCH_DATA_TAG]); expect(call.options.revalidate).toBe(DATA_CACHE_BACKSTOP_SECONDS); diff --git a/src/bench/lib/data-cache.ts b/src/bench/lib/data-cache.ts index f77ea34d..baf9da9e 100644 --- a/src/bench/lib/data-cache.ts +++ b/src/bench/lib/data-cache.ts @@ -57,7 +57,7 @@ const CACHE_OPTIONS = { tags: [BENCH_DATA_TAG], revalidate: DATA_CACHE_BACKSTOP_ * code can serve payloads the old code computed until the next ingest. * It is independent of the producer-facing benchmark schema version. */ -const GROUP_PAYLOAD_CACHE_VERSION = 'v12'; +const GROUP_PAYLOAD_CACHE_VERSION = 'v13'; // The default last-100 group bundle, keyed by group slug. The slug is the cache // key (an `unstable_cache` argument), so one wrapper covers every group. A