diff --git a/CLAUDE.md b/CLAUDE.md index 87c5002..41d4cf9 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -82,7 +82,7 @@ The previous site generations (v2 and v3) are documented in [`docs/bench/legacy. - **Numeric `?n=` snaps up to a rung of 25, 50, 100, 200, 500, or 1000; `?n=all` is the uncapped escape hatch.** Pages hydrate from the materialized latest-100 shard artifact by default; `?n=all` is an explicit opt-in (the chart's full-history zoom-out hop uses it once, and curl power users can request it). The numeric path is bounded by `MAX_NUMERIC_COMMIT_WINDOW` in [`src/bench/lib/window.ts`](src/bench/lib/window.ts) as a DoS-protection floor against `?n=99999999`, and a numeric `n` snaps up to the next rung of `COMMIT_WINDOW_RUNGS` there, so a chart or group has seven window URLs rather than a thousand, each a direct query that the CDN caches per URL. Do NOT raise the numeric cap, add rungs, or remove either without thinking about the DoS surface, and do NOT re-introduce a server-side commit cap on `?n=all` — visual downsampling happens client-side via LTTB on the visible commit range only. Only the default window is in the Data Cache, so every other window is a direct query that only the CDN shields. The read APIs and the chart page redirect (308) a non-default window to its one URL, with `n` spelled canonically and only the parameters the route reads (`canonicalWindowQuery` in `window.ts`), so adding a parameter or respelling `n` cannot get a request past the CDN. They also redirect every other spelling of a slug to the canonical one (`chartKeyToSlug`, `groupKeyToSlug`), since the CDN keys on the spelling, and the chart page cuts `engine` and `format` to the values bench knows, sorted. - **Don't refetch on every scope change.** Once a chart's payload is in memory, pan/zoom/slider/range-strip all rebuild in place via the in-memory LTTB pass on the cached payload ([`src/bench/components/chart/controller.ts`](src/bench/components/chart/controller.ts)). The single exception is the latest-100 to full-history zoom-out path: when the user first zooms past the materialized window, the chart lazy-fetches `?n=all` once and replaces the payload in place. - **Group docs links mirror the monorepo's `Benchmark::doc_path`.** [`src/bench/lib/benchmark-docs.ts`](src/bench/lib/benchmark-docs.ts) maps each SQL dataset name to the benchmark's explainer doc in `vortex-data/vortex`, which is the same path that the monorepo's PR benchmark comment uses. The producer does not put `doc_path` on the wire, and upstream does not enforce a path convention, so update the explicit mapping when a suite is added or its doc moves. Groups without a matching explainer render no docs link. Do not guess a path or link to an unrelated general guide. -- **Every group kind has a summary, by default.** [`src/bench/lib/summary.ts`](src/bench/lib/summary.ts) has no allowlist gate: a suite that lands in one of the five fact tables gets a rollup card from its first ingest. The three timing families (query, random access, vector search) rank through one `rankSeries` model, so a new suite in any of them needs no summary code at all. `collectGroupSummary`'s exhaustive switch makes a missing arm for a sixth fact table a compile error rather than a silently blank card. Do NOT reintroduce a per-dataset allowlist; the v2-era one is what left `spatialbench`, `fineweb`, `gharchive`, `appian`, `public-bi`, `clickbench-sorted`, and every vector-search group with no card. +- **Every group kind has a summary, by default.** [`src/bench/lib/summary.ts`](src/bench/lib/summary.ts) has no allowlist gate: a suite that lands in one of the six fact tables gets a rollup card from its first ingest. The three timing families (query, random access, vector search) rank through one `rankSeries` model, so a new suite in any of them needs no summary code at all. `collectGroupSummary`'s exhaustive switch makes a missing arm for a seventh fact table a compile error rather than a silently blank card. Do NOT reintroduce a per-dataset allowlist; the v2-era one is what left `spatialbench`, `fineweb`, `gharchive`, `appian`, `public-bi`, `clickbench-sorted`, and every vector-search group with no card. - **A summary ranks the whole group, never one chart.** Random access is the cautionary case: the producer emits `dataset` as `{dataset}/{pattern}`, so the group holds ~nine charts, and the old summary published the alphabetically first chart's raw times under the group-wide title "Random Access Performance" — reporting `lance` at 352us when it is over 1ms on most of the other charts. Sum chart medians by dataset, then rank the dataset totals. Include the legacy taxi chart in the taxi total. Apply a penalty when a series lacks one dataset total. Report dataset coverage through `measured`/`total`. The secondary runtime is the arithmetic mean of complete dataset totals. - **Don't write a server-side classifier for live ingest.** The emitter produces structured records directly. Classifying loose name strings at read time was the v2-era weakness every later generation existed to escape; it belongs nowhere in the live pipeline. - **Every root-relative bench URL goes through `benchPath()`.** A link, a fetch, an icon or a selector that names one of bench's paths must work on bench.vortex.dev (from the root) and everywhere else (under `/bench`); see "Repository layout". A path read from `location.pathname` already carries the base. diff --git a/docs/bench/CONTRACT.md b/docs/bench/CONTRACT.md index c2e8baa..e27c06e 100644 --- a/docs/bench/CONTRACT.md +++ b/docs/bench/CONTRACT.md @@ -75,7 +75,8 @@ rows in the `String Encoding` groups and compares each encoder with `fsst`, so t `file_size` records are not written by `vortex-bench`: the monorepo's `scripts/capture-file-sizes.py` writes one per Vortex file a SQL benchmark leaves on disk, with `benchmark` (the `vx-bench` subcommand), `scale_factor`, `format`, `file` (the path below the format -directory) and `value_bytes`. The read service does not chart them; CI reads them back as the develop +directory) and `value_bytes`. The read service charts each benchmark and scale factor in the +`SQL File Size` group, one line per format summed over its files. CI reads them back as the develop baseline for the PR comment's file-size section. `random_access_time` records include `open_mode`. The value is `cached` for a reused accessor @@ -160,6 +161,5 @@ For wire/record *shape* changes, also update the producer snapshot fixtures in t the same commit. A new record kind with its own table, as `file_size` was in migration 014, changes no existing -shape, so it needs no bump: the read service never reads the new table, and older writers never -send the kind. Its migration must be applied before a writer sends it, because each JSONL file +shape, so it needs no bump: no existing read changes, and older writers never send the kind. Its migration must be applied before a writer sends it, because each JSONL file lands in one transaction and an unknown table fails the whole file. diff --git a/docs/bench/architecture/design-decisions.md b/docs/bench/architecture/design-decisions.md index f517790..b57d860 100644 --- a/docs/bench/architecture/design-decisions.md +++ b/docs/bench/architecture/design-decisions.md @@ -18,9 +18,9 @@ reasoning behind them. Each entry is "decision → why → where." Topic docs ## Data model **One fact table per measurement family (not one wide table).** -The five families have genuinely different dimensions. A single wide table would +The six families have genuinely different dimensions. A single wide table would bloat every row with NULL columns it doesn't use, or split one chart's data across -multiple rows that then need re-joining. Five focused tables keep each row dense +multiple rows that then need re-joining. Six focused tables keep each row dense and each chart query a single-table scan. → `migrations/001_initial_schema.sql`, `server/src/schema.rs`. diff --git a/src/bench/lib/benchmark-docs.ts b/src/bench/lib/benchmark-docs.ts index 096d339..8fd75d7 100644 --- a/src/bench/lib/benchmark-docs.ts +++ b/src/bench/lib/benchmark-docs.ts @@ -88,6 +88,7 @@ export function groupDocPath(key: GroupKey): string | null { case 'QueryGroup': return QUERY_DOCS[key.dataset] ?? null; case 'VectorSearchGroup': + case 'FileSizeGroup': return null; } } diff --git a/src/bench/lib/data-cache.test.ts b/src/bench/lib/data-cache.test.ts index c77e9b7..e1a8b18 100644 --- a/src/bench/lib/data-cache.test.ts +++ b/src/bench/lib/data-cache.test.ts @@ -69,9 +69,9 @@ describe('data-cache wrappers', () => { expect(BENCH_DATA_TAG).toBe('bench-data'); expect(DATA_CACHE_BACKSTOP_SECONDS).toBe(86400); expect(cacheCalls.map((call) => call.keyParts)).toContainEqual([ - 'data-cache:group-charts:v12:n100', + 'data-cache:group-charts:v13:n100', ]); - expect(cacheCalls.map((call) => call.keyParts)).toContainEqual(['data-cache:groups:v12']); + expect(cacheCalls.map((call) => call.keyParts)).toContainEqual(['data-cache:groups:v13']); for (const call of cacheCalls) { expect(call.options.tags).toEqual([BENCH_DATA_TAG]); expect(call.options.revalidate).toBe(DATA_CACHE_BACKSTOP_SECONDS); diff --git a/src/bench/lib/data-cache.ts b/src/bench/lib/data-cache.ts index f77ea34..baf9da9 100644 --- a/src/bench/lib/data-cache.ts +++ b/src/bench/lib/data-cache.ts @@ -57,7 +57,7 @@ const CACHE_OPTIONS = { tags: [BENCH_DATA_TAG], revalidate: DATA_CACHE_BACKSTOP_ * code can serve payloads the old code computed until the next ingest. * It is independent of the producer-facing benchmark schema version. */ -const GROUP_PAYLOAD_CACHE_VERSION = 'v12'; +const GROUP_PAYLOAD_CACHE_VERSION = 'v13'; // The default last-100 group bundle, keyed by group slug. The slug is the cache // key (an `unstable_cache` argument), so one wrapper covers every group. A diff --git a/src/bench/lib/descriptions.ts b/src/bench/lib/descriptions.ts index 5fe7d63..2aaa6de 100644 --- a/src/bench/lib/descriptions.ts +++ b/src/bench/lib/descriptions.ts @@ -55,6 +55,11 @@ function staticDescription(name: string): string | null { return 'Measures Vortex file write and scan throughput for each string encoder'; case 'String Encoding Size': return 'Compares Vortex file sizes for each string encoder'; + case 'SQL File Size': + return ( + 'Tracks the total size of the Vortex files each SQL benchmark writes, per benchmark and ' + + 'scale factor' + ); case 'Clickbench': return ( "ClickHouse's analytical benchmark suite testing real-world query patterns on web " + diff --git a/src/bench/lib/families.test.ts b/src/bench/lib/families.test.ts index d8de988..86c3ded 100644 --- a/src/bench/lib/families.test.ts +++ b/src/bench/lib/families.test.ts @@ -5,13 +5,14 @@ import { describe, expect, it } from 'vitest'; import { FAMILIES, familyForChartKind, familyForGroupKind } from './families'; describe('FAMILIES registry', () => { - it('lists the five fact tables in family.rs declaration order', () => { + it('lists the six fact tables in family.rs declaration order, then file_sizes', () => { expect(FAMILIES.map((f) => f.tableName)).toEqual([ 'query_measurements', 'compression_times', 'compression_sizes', 'random_access_times', 'vector_search_runs', + 'file_sizes', ]); }); @@ -29,6 +30,7 @@ describe('FAMILIES registry', () => { ['cs', 'csg'], ['rat', 'rag'], ['vsr', 'vsg'], + ['fs', 'fsg'], ]); }); }); diff --git a/src/bench/lib/families.ts b/src/bench/lib/families.ts index 9f8eb8a..6e196ee 100644 --- a/src/bench/lib/families.ts +++ b/src/bench/lib/families.ts @@ -4,23 +4,25 @@ /** * Per-fact-table registry, the TypeScript port of `server/src/family.rs`. * - * Each of the five fact tables is a [`Family`] that ties together the Postgres + * Each of the six fact tables is a [`Family`] that ties together the Postgres * table name and the chart and group slug prefixes. The read endpoints * dispatch through this registry rather than hand-listing the families, so the * slug prefixes consumed by [`./slug`] have a single source of truth. The read * queries name their tables in static SQL. * * The order of [`FAMILIES`] mirrors `family.rs`'s `FAMILIES` constant and the - * DDL apply order in `migrations/001_initial_schema.sql`. + * DDL apply order in `migrations/001_initial_schema.sql`, with `file_sizes` + * (`migrations/014`) last. */ -/** The five fact-table names, spelled exactly as in `migrations/001`. */ +/** The six fact-table names, spelled exactly as in their migrations. */ type FamilyTable = | 'query_measurements' | 'compression_times' | 'compression_sizes' | 'random_access_times' - | 'vector_search_runs'; + | 'vector_search_runs' + | 'file_sizes'; /** Discriminant (`k`) of a [`./slug`].`ChartKey`; one per family. */ export type ChartKind = @@ -28,7 +30,8 @@ export type ChartKind = | 'CompressionTime' | 'CompressionSize' | 'RandomAccess' - | 'VectorSearch'; + | 'VectorSearch' + | 'FileSize'; /** Discriminant (`k`) of a [`./slug`].`GroupKey`; one per family. */ export type GroupKind = @@ -36,11 +39,12 @@ export type GroupKind = | 'CompressionTimeGroup' | 'CompressionSizeGroup' | 'RandomAccessGroup' - | 'VectorSearchGroup'; + | 'VectorSearchGroup' + | 'FileSizeGroup'; /** One fact-table family: its table name, slug prefixes, and key discriminants. */ export interface Family { - /** Postgres table name, matching `migrations/001_initial_schema.sql`. */ + /** Postgres table name, matching its migration. */ tableName: FamilyTable; /** Slug prefix for individual charts in this family (e.g. `qm`). */ chartSlugPrefix: string; @@ -53,8 +57,8 @@ export interface Family { } /** - * All five fact-table families, in the order `family.rs`'s `FAMILIES` declares - * them (which is also the `migrations/001` DDL apply order). + * All six fact-table families, in the order `family.rs`'s `FAMILIES` declares + * them (which is also the `migrations/001` DDL apply order), then `file_sizes`. */ export const FAMILIES: readonly Family[] = [ { @@ -92,6 +96,13 @@ export const FAMILIES: readonly Family[] = [ chartKind: 'VectorSearch', groupKind: 'VectorSearchGroup', }, + { + tableName: 'file_sizes', + chartSlugPrefix: 'fs', + groupSlugPrefix: 'fsg', + chartKind: 'FileSize', + groupKind: 'FileSizeGroup', + }, ]; /** Look up the family that owns a chart-key discriminant. */ diff --git a/src/bench/lib/group-charts.test.ts b/src/bench/lib/group-charts.test.ts index 820ce30..dc4f5b2 100644 --- a/src/bench/lib/group-charts.test.ts +++ b/src/bench/lib/group-charts.test.ts @@ -88,6 +88,11 @@ describe('seriesFor', () => { { flavor: 'vortex-turboquant' }, { key: 'vortex-turboquant', engine: undefined, format: undefined }, ], + [ + 'FileSize', + { format: 'vortex-compact' }, + { key: 'vortex-compact', engine: undefined, format: 'vortex-compact' }, + ], ] as const)('names the %s series', (kind, cols, expected) => { expect(seriesFor(kind, { commit_sha: 'a', value: 1, ...cols })).toEqual(expected); }); @@ -468,6 +473,29 @@ describe.skipIf(!databaseAvailable)('groupChartPayloads at the window edges', () [measurementId, sha, dataset, variant, valueBytes], ); } + // File sizes: TPC-H SF=1 has two files per format on C0, E and F (compact only from E); + // SF=10 has one file on F. + for (const [sha, scaleFactor, format, file, valueBytes] of [ + [C0, '1.0', 'vortex-file-compressed', 'a.vortex', 100], + [C0, '1.0', 'vortex-file-compressed', 'b.vortex', 10], + [E, '1.0', 'vortex-file-compressed', 'a.vortex', 101], + [E, '1.0', 'vortex-file-compressed', 'b.vortex', 11], + [E, '1.0', 'vortex-compact', 'a.vortex', 80], + [E, '1.0', 'vortex-compact', 'b.vortex', 8], + [F, '1.0', 'vortex-file-compressed', 'a.vortex', 102], + [F, '1.0', 'vortex-file-compressed', 'b.vortex', 12], + [F, '1.0', 'vortex-compact', 'a.vortex', 79], + [F, '1.0', 'vortex-compact', 'b.vortex', 7], + [F, '10.0', 'vortex-file-compressed', 'a.vortex', 1000], + ] as const) { + measurementId += 1; + await getPool().query( + `INSERT INTO file_sizes + (measurement_id, commit_sha, benchmark, scale_factor, format, file, value_bytes) + VALUES ($1, $2, 'tpch', $3, $4, $5, $6)`, + [measurementId, sha, scaleFactor, format, file, valueBytes], + ); + } }); afterAll(async () => { @@ -563,10 +591,17 @@ describe.skipIf(!databaseAvailable)('groupChartPayloads at the window edges', () { k: 'CompressionSize', dataset: 'sy', dataset_variant: 'v2' }, ], ], + [ + { k: 'FileSizeGroup' }, + [ + { k: 'FileSize', benchmark: 'tpch', scale_factor: '1.0' }, + { k: 'FileSize', benchmark: 'tpch', scale_factor: '10.0' }, + ], + ], ]; it.each(['1', '2', '3', '4', 'all'])( - 'matches chartPayload for random-access, vector-search and compression-size groups at ?n=%s', + 'matches chartPayload for random-access, vector-search, compression-size and file-size groups at ?n=%s', async (n) => { const window = commitWindow(n); for (const [group, keys] of OTHER_FAMILIES) { @@ -608,6 +643,22 @@ describe.skipIf(!databaseAvailable)('groupChartPayloads at the window edges', () expect(short?.series).toEqual({ fl: [600] }); }); + it('sums each file-size chart over its files, one line per format', async () => { + const [sf1, sf10] = await groupChartPayloads( + ...defined(OTHER_FAMILIES[3]), + parseCommitWindow('all'), + ); + expect(sf1?.display_name).toBe('tpch'); + expect(sf1?.unit_kind).toBe('bytes'); + expect(sf1?.series).toEqual({ + 'vortex-compact': [null, null, null, null, 88, 86], + 'vortex-file-compressed': [110, null, null, null, 112, 114], + }); + expect(sf10?.display_name).toBe('tpch sf=10.0'); + expect(sf10?.commits.map((c) => c.sha)).toEqual([F]); + expect(sf10?.series).toEqual({ 'vortex-file-compressed': [1000] }); + }); + it('keeps sibling groups apart', async () => { const window = parseCommitWindow('all'); const [g] = await groupChartPayloads(G, [queryChart(G, 1)], window); diff --git a/src/bench/lib/groups.test.ts b/src/bench/lib/groups.test.ts index 3784b8d..f4fb364 100644 --- a/src/bench/lib/groups.test.ts +++ b/src/bench/lib/groups.test.ts @@ -26,7 +26,7 @@ import { import { groupDescription } from './descriptions'; import { READ_API_CACHE_CONTROL } from './cache'; import { collectGroupSummary } from './summary'; -import { chartKeyToSlug, groupKeyFromSlug, groupKeyToSlug } from './slug'; +import { chartKeyFromSlug, chartKeyToSlug, groupKeyFromSlug, groupKeyToSlug } from './slug'; import { type BenchDatabase, commitUrl, @@ -445,7 +445,7 @@ describe.skipIf(!databaseAvailable)('summary math fidelity (testcontainers Postg beforeEach(async () => { await getPool().query( `TRUNCATE compression_times, compression_sizes, query_measurements, - random_access_times, vector_search_runs, commits`, + random_access_times, vector_search_runs, file_sizes, commits`, ); }); @@ -626,6 +626,64 @@ describe.skipIf(!databaseAvailable)('summary math fidelity (testcontainers Postg ]); }); + it('discovers one file-size chart per benchmark and scale factor, with a summary', async () => { + const develop = 'a'.repeat(40); + const nightly = 'b'.repeat(40); + await insertCommit(develop, '2026-04-23T12:00:00Z'); + await insertCommit(nightly, '2026-04-24T03:00:00Z'); + // TPC-H at SF 1 and 10 from develop, and SF 100 from nightly, the newest commit. Clickbench + // has no scale-factor directory, so it is recorded as 1.0. + for (const [sha, benchmark, scaleFactor, format, file, bytes] of [ + [develop, 'tpch', '1.0', 'vortex-file-compressed', 'lineitem.vortex', 300], + [develop, 'tpch', '1.0', 'vortex-file-compressed', 'orders.vortex', 100], + [develop, 'tpch', '1.0', 'vortex-compact', 'lineitem.vortex', 150], + [develop, 'tpch', '1.0', 'vortex-compact', 'orders.vortex', 50], + [develop, 'tpch', '10.0', 'vortex-file-compressed', 'lineitem.vortex', 4000], + [develop, 'tpch', '10.0', 'vortex-compact', 'lineitem.vortex', 1000], + [nightly, 'tpch', '100', 'vortex-file-compressed', 'lineitem.vortex', 40000], + [develop, 'clickbench', '1.0', 'vortex-file-compressed', 'hits_0.vortex', 1000], + ] as const) { + await getPool().query( + `INSERT INTO file_sizes + (measurement_id, commit_sha, benchmark, scale_factor, format, file, value_bytes) + VALUES ($1, $2, $3, $4, $5, $6, $7)`, + [nextId(), sha, benchmark, scaleFactor, format, file, bytes], + ); + } + + const groups = await collectGroups(); + expect(groups.map((group) => group.name)).toEqual(['SQL File Size']); + const [group] = groups; + expect(groupKeyFromSlug(defined(group).slug)).toEqual({ k: 'FileSizeGroup' }); + expect(defined(group).charts.map((chart) => chart.name)).toEqual([ + 'clickbench', + 'tpch', + 'tpch sf=10.0', + 'tpch sf=100', + ]); + expect(chartKeyFromSlug(defined(defined(group).charts[3]).slug)).toEqual({ + k: 'FileSize', + benchmark: 'tpch', + scale_factor: '100', + }); + // Compact is half the size at SF 1 and a quarter at SF 10. Totals cover the scopes both + // formats have: 200 + 1000 against 400 + 4000. + expect(defined(group).summary).toMatchObject({ + type: 'compressionSize', + baseline: 'vortex', + rankings: [ + { + name: 'vortex-compact', + minRatio: 0.25, + ratio: expect.closeTo(0.5 ** 1.5), + maxRatio: 0.5, + totalBytes: 1200, + }, + { name: 'vortex-file-compressed', ratio: 1, compressionRatio: null, totalBytes: 4400 }, + ], + }); + }); + it('ranks random access over every chart, not the alphabetically first one', async () => { // The bug this pins: the old summary walked the group's chart links and // published the first populated chart's raw times under the group-wide diff --git a/src/bench/lib/headline.test.ts b/src/bench/lib/headline.test.ts index 7db1ee8..e989764 100644 --- a/src/bench/lib/headline.test.ts +++ b/src/bench/lib/headline.test.ts @@ -193,6 +193,21 @@ describe('groupHeadline', () => { expect(h?.parts[0]?.tone).toBe(tone); }); + it('has no headline for a size summary with its own baseline', () => { + // The SQL file size summary compares with Vortex itself, which would read "same size as Parquet". + expect( + groupHeadline({ + type: 'compressionSize', + title: 's', + explanation: 'e', + baseline: 'vortex', + rankings: [ + { name: VX, minRatio: 1, ratio: 1, maxRatio: 1, compressionRatio: null, totalBytes: 1 }, + ], + }), + ).toBeNull(); + }); + it('reads hot and cold random access against Parquet', () => { expect(line(ra([r(VX, 1), r('parquet', 60.4)], [r(VX, 1), r('parquet', 32)]))).toBe( 'hot 60× · cold 32× faster than Parquet', diff --git a/src/bench/lib/headline.ts b/src/bench/lib/headline.ts index a08bd4c..3d95f2e 100644 --- a/src/bench/lib/headline.ts +++ b/src/bench/lib/headline.ts @@ -91,7 +91,10 @@ export function groupHeadline(summary: Summary | undefined): Headline | null { })), ); case 'compressionSize': - return sizeHeadline(summary.rankings.find((row) => row.name === VORTEX)?.ratio); + // A summary with its own baseline compares with that, not Parquet, so it has no headline. + return summary.baseline === undefined + ? sizeHeadline(summary.rankings.find((row) => row.name === VORTEX)?.ratio) + : null; case 'vectorSearch': return vectorSearchHeadline(summary.rankings); default: diff --git a/src/bench/lib/queries.ts b/src/bench/lib/queries.ts index 5688f60..cc67bf4 100644 --- a/src/bench/lib/queries.ts +++ b/src/bench/lib/queries.ts @@ -354,6 +354,7 @@ type CompressionTimeKey = Extract; type CompressionSizeKey = Extract; type RandomAccessKey = Extract; type VectorSearchKey = Extract; +type FileSizeKey = Extract; /** Row shape of a fact-table pass; `value` is read `::float8`. */ type ValueRow = { commit_sha: string; value: number }; @@ -396,6 +397,8 @@ export function seriesFor( } case 'VectorSearch': return { key: `${row.flavor}`, engine: undefined, format: undefined }; + case 'FileSize': + return { key: `${row.format}`, engine: undefined, format: row.format }; } } @@ -430,12 +433,22 @@ function chartDisplayName(key: ChartKey): string { return key.dataset; case 'VectorSearch': return `${key.dataset} / ${key.layout} (threshold=${key.threshold})`; + case 'FileSize': + return fileSizeChartName(key.benchmark, key.scale_factor); } } +/** + * A file-size chart's name: the benchmark, with its scale factor unless it is the default `1.0`. + * The monorepo's capture script records `1.0` for a benchmark with no scale-factor directory. + */ +function fileSizeChartName(benchmark: string, scaleFactor: string): string { + return scaleFactor === '1.0' ? benchmark : `${benchmark} sf=${scaleFactor}`; +} + /** A chart's base `unit_kind`: sizes are bytes, everything else is a time. */ function chartUnitKind(key: ChartKey): UnitKind { - return key.k === 'CompressionSize' ? 'bytes' : 'time_ns'; + return key.k === 'CompressionSize' || key.k === 'FileSize' ? 'bytes' : 'time_ns'; } async function collectQueryChart( @@ -636,6 +649,48 @@ async function collectVectorSearchChart( return acc.finish(chartDisplayName(key), chartUnitKind(key), seeded.history); } +/** + * One line per Vortex format: the total size of the benchmark's files at each commit. The table + * has a row per file, which would be thousands of lines across the group. + */ +async function collectFileSizeChart( + key: FileSizeKey, + window: CommitWindow, +): Promise { + const { benchmark, scale_factor } = key; + const seeded = await seededCommitsInWindow( + (p) => + `SELECT MIN(c2.timestamp) + FROM file_sizes f2 + JOIN ${BRANCH_COMMITS} c2 ON c2.commit_sha = f2.commit_sha + WHERE f2.benchmark = ${p.bind(benchmark)} + AND f2.scale_factor = ${p.bind(scale_factor)}`, + window, + ); + if (seeded.commits.length === 0) { + return null; + } + const acc = new SeriesAccumulator(); + acc.seedCommits(seeded.commits); + + const params = new QueryParams(); + const text = ` + SELECT f.commit_sha, + f.format, SUM(f.value_bytes)::float8 AS value + FROM file_sizes f + JOIN ${BRANCH_COMMITS} c USING (commit_sha) + WHERE f.benchmark = ${params.bind(benchmark)} + AND f.scale_factor = ${params.bind(scale_factor)}${factWindowFilter(params, window)} + GROUP BY f.commit_sha, c.timestamp, f.format + ORDER BY c.timestamp, f.format + `; + const rows = (await getPool().query(text, params.values)).rows; + for (const row of rows) { + recordRow(acc, key.k, row); + } + return acc.finish(chartDisplayName(key), chartUnitKind(key), seeded.history); +} + /** * Build the JSON payload for one chart by key, or `null` when the chart has no * data (callers render a 404). The shared implementation behind @@ -658,6 +713,8 @@ export function chartPayload(key: ChartKey, window: CommitWindow): Promise + `${alias}.benchmark = ch.benchmark AND ${alias}.scale_factor = ch.scale_factor`; + block( + ['format'], + keys.map((_, i) => i + 1), + [ + ['benchmark', 'text', keys.map((k) => k.benchmark)], + ['scale_factor', 'text', keys.map((k) => k.scale_factor)], + ], + `SELECT MIN(c2.timestamp) + FROM file_sizes f2 + JOIN ${BRANCH_COMMITS} c2 ON c2.commit_sha = f2.commit_sha + WHERE ${pins('f2')}`, + `SELECT f.commit_sha, + f.format, SUM(f.value_bytes)::float8 AS value + FROM file_sizes f + JOIN ${BRANCH_COMMITS} c USING (commit_sha) + WHERE ${pins('f')}${factWindowFilter(params, window)} + GROUP BY f.commit_sha, f.format`, + ); + break; + } } const text = ` WITH commit_times AS ( @@ -1159,6 +1242,7 @@ const GROUP_ORDER: readonly string[] = [ 'Compression Size', 'String Encoding', 'String Encoding Size', + 'SQL File Size', 'Random Access (NVMe)', 'Random Access (S3)', 'Clickbench', @@ -1573,6 +1657,60 @@ async function collectVectorSearchGroups(): Promise { return groups; } +type FileSizeScopeRow = { benchmark: string; scale_factor: string }; + +/** + * The `SQL File Size` group: one chart per benchmark and scale factor, or `[]` if the table is + * empty. Charts order by benchmark, then by numeric scale factor, so `10.0` follows `1.0`. + */ +async function collectFileSizeGroup(): Promise { + // A recursive skip scan of `idx_file_sizes_baseline`: one descent per scope instead of a scan + // of every file of every commit. + const text = ` + WITH RECURSIVE scopes AS ( + (SELECT f.benchmark, f.scale_factor + FROM file_sizes f + ORDER BY f.benchmark, f.scale_factor + LIMIT 1) + UNION ALL + SELECT nxt.benchmark, nxt.scale_factor + FROM scopes s + CROSS JOIN LATERAL ( + SELECT f.benchmark, f.scale_factor + FROM file_sizes f + WHERE (f.benchmark, f.scale_factor) > (s.benchmark, s.scale_factor) + ORDER BY f.benchmark, f.scale_factor + LIMIT 1 + ) nxt + ) + SELECT benchmark, scale_factor FROM scopes + `; + const rows = (await getPool().query(text)).rows; + if (rows.length === 0) { + return []; + } + rows.sort( + (a, b) => + compareCodeUnits(a.benchmark, b.benchmark) || + Number(a.scale_factor) - Number(b.scale_factor) || + compareCodeUnits(a.scale_factor, b.scale_factor), + ); + return [ + { + name: 'SQL File Size', + slug: groupKeyToSlug({ k: 'FileSizeGroup' }), + charts: rows.map((row) => ({ + name: fileSizeChartName(row.benchmark, row.scale_factor), + slug: chartKeyToSlug({ + k: 'FileSize', + benchmark: row.benchmark, + scale_factor: row.scale_factor, + }), + })), + }, + ]; +} + /** Dispatch to the discovery pass for one family, the TS analogue of the Rust * `(family.collect_groups)(conn)` registry indirection. */ function discoverGroups(groupKind: GroupKind): Promise { @@ -1587,6 +1725,8 @@ function discoverGroups(groupKind: GroupKind): Promise { return collectRandomAccessGroup(); case 'VectorSearchGroup': return collectVectorSearchGroups(); + case 'FileSizeGroup': + return collectFileSizeGroup(); } } diff --git a/src/bench/lib/site-files.ts b/src/bench/lib/site-files.ts index 61d70c2..8a5dd95 100644 --- a/src/bench/lib/site-files.ts +++ b/src/bench/lib/site-files.ts @@ -231,6 +231,7 @@ export function chartSeriesLabel(kind: ChartKind, name: string): string { return `${FORMATS.get(format) ?? format} (${detail})`; case 'CompressionSize': case 'VectorSearch': + case 'FileSize': return FORMATS.get(name) ?? name; } } diff --git a/src/bench/lib/slug.ts b/src/bench/lib/slug.ts index 6373b6f..89d6a97 100644 --- a/src/bench/lib/slug.ts +++ b/src/bench/lib/slug.ts @@ -32,7 +32,9 @@ export type ChartKey = | { k: 'CompressionTime'; dataset: string; dataset_variant: string | null } | { k: 'CompressionSize'; dataset: string; dataset_variant: string | null } | { k: 'RandomAccess'; dataset: string } - | { k: 'VectorSearch'; dataset: string; layout: string; threshold: number }; + | { k: 'VectorSearch'; dataset: string; layout: string; threshold: number } + // One SQL benchmark's data at one scale factor; its series are the Vortex formats. + | { k: 'FileSize'; benchmark: string; scale_factor: string }; /** Strongly-typed group key parsed from a slug. Discriminated on `k`. */ export type GroupKey = @@ -47,7 +49,8 @@ export type GroupKey = | { k: 'CompressionTimeGroup'; suite?: 'strings' } | { k: 'CompressionSizeGroup'; suite?: 'strings' } | { k: 'RandomAccessGroup'; storage: RandomAccessStorage } - | { k: 'VectorSearchGroup'; dataset: string; layout: string }; + | { k: 'VectorSearchGroup'; dataset: string; layout: string } + | { k: 'FileSizeGroup' }; const CHART_KINDS: readonly ChartKind[] = [ 'QueryMeasurement', @@ -55,6 +58,7 @@ const CHART_KINDS: readonly ChartKind[] = [ 'CompressionSize', 'RandomAccess', 'VectorSearch', + 'FileSize', ]; const GROUP_KINDS: readonly GroupKind[] = [ @@ -63,6 +67,7 @@ const GROUP_KINDS: readonly GroupKind[] = [ 'CompressionSizeGroup', 'RandomAccessGroup', 'VectorSearchGroup', + 'FileSizeGroup', ]; function encodePayload(json: string): string { @@ -111,6 +116,9 @@ export function chartKeyToSlug(key: ChartKey): string { case 'VectorSearch': ordered = { k: key.k, dataset: key.dataset, layout: key.layout, threshold: key.threshold }; break; + case 'FileSize': + ordered = { k: key.k, benchmark: key.benchmark, scale_factor: key.scale_factor }; + break; } return `${prefix}.${encodePayload(JSON.stringify(ordered))}`; } @@ -213,6 +221,12 @@ export function chartKeyFromSlug(slug: string): ChartKey { layout: reqString(obj, 'layout'), threshold: reqNumber(obj, 'threshold'), }; + case 'FileSize': + return { + k: kind, + benchmark: reqString(obj, 'benchmark'), + scale_factor: reqString(obj, 'scale_factor'), + }; } } @@ -243,6 +257,9 @@ export function groupKeyToSlug(key: GroupKey): string { case 'CompressionSizeGroup': ordered = key.suite === undefined ? { k: key.k } : { k: key.k, suite: key.suite }; break; + case 'FileSizeGroup': + ordered = { k: key.k }; + break; } return `${prefix}.${encodePayload(JSON.stringify(ordered))}`; } @@ -287,5 +304,7 @@ export function groupKeyFromSlug(slug: string): GroupKey { } return { k: kind, suite: 'strings' }; } + case 'FileSizeGroup': + return { k: kind }; } } diff --git a/src/bench/lib/summary.test.ts b/src/bench/lib/summary.test.ts index 63d2918..d053b1f 100644 --- a/src/bench/lib/summary.test.ts +++ b/src/bench/lib/summary.test.ts @@ -753,6 +753,7 @@ describe('timing summaries (shared ranking model)', () => { { k: 'CompressionSizeGroup' }, { k: 'RandomAccessGroup', storage: 'nvme' }, { k: 'VectorSearchGroup', dataset: 'd', layout: 'l' }, + { k: 'FileSizeGroup' }, ]; expect(keys.map((key) => key.k).sort()).toEqual([...GROUP_KINDS].sort()); for (const key of keys) { diff --git a/src/bench/lib/summary.ts b/src/bench/lib/summary.ts index d58a176..43edd96 100644 --- a/src/bench/lib/summary.ts +++ b/src/bench/lib/summary.ts @@ -8,10 +8,10 @@ * variant. Compression summaries compare the configured formats with Parquet. * * **Every group kind has a summary.** There is no allowlist gate: a benchmark - * suite that lands in one of the five fact tables gets a rollup card the moment + * suite that lands in one of the six fact tables gets a rollup card the moment * its first rows are ingested. The timing-shaped families (query, random * access, vector search) all rank through the one [`rankSeries`] model below, - * so a new suite in any of them needs no summary code at all. Adding a sixth + * so a new suite in any of them needs no summary code at all. Adding a seventh * fact table is the only case that needs a new arm here, and * [`collectGroupSummary`]'s exhaustive switch makes omitting it a compile * error rather than a silently missing card. @@ -208,6 +208,8 @@ export function collectGroupSummary(key: GroupKey): Promise { return collectRandomAccessSummary(key.storage); case 'VectorSearchGroup': return collectVectorSearchSummary(key.dataset, key.layout); + case 'FileSizeGroup': + return collectFileSizeSummary(); } } @@ -1021,6 +1023,92 @@ async function collectStringEncodingSizeSummary(): Promise { }; } +/** The format every other SQL benchmark file format is compared with. */ +const FILE_SIZE_BASELINE = 'vortex-file-compressed'; + +/** + * The `SQL File Size` rollup: each Vortex format's total file size against + * `vortex-file-compressed`, per benchmark scope (benchmark and scale factor) at that scope's + * newest commit with a baseline. Scopes come from nightly as well as develop runs, so each one + * uses its own newest commit. File sizes record no Arrow size, so the Arrow ratio is empty. + */ +async function collectFileSizeSummary(): Promise { + // `scopes` is a skip scan of `idx_file_sizes_baseline`. `latest` walks the read branch's + // commits newest first and probes that index, so it stops at the newest commit with a baseline. + const text = ` + WITH RECURSIVE scopes AS ( + (SELECT f.benchmark, f.scale_factor + FROM file_sizes f + ORDER BY f.benchmark, f.scale_factor + LIMIT 1) + UNION ALL + SELECT nxt.benchmark, nxt.scale_factor + FROM scopes s + CROSS JOIN LATERAL ( + SELECT f.benchmark, f.scale_factor + FROM file_sizes f + WHERE (f.benchmark, f.scale_factor) > (s.benchmark, s.scale_factor) + ORDER BY f.benchmark, f.scale_factor + LIMIT 1 + ) nxt + ), + latest AS ( + SELECT s.benchmark, s.scale_factor, l.commit_sha + FROM scopes s + CROSS JOIN LATERAL ( + SELECT c.commit_sha + FROM ${BRANCH_COMMITS} c + WHERE EXISTS ( + SELECT 1 + FROM file_sizes f + WHERE f.benchmark = s.benchmark + AND f.scale_factor = s.scale_factor + AND f.format = $1 + AND f.commit_sha = c.commit_sha) + ORDER BY c.timestamp DESC, c.commit_sha DESC + LIMIT 1 + ) l + ), + totals AS ( + SELECT f.benchmark, f.scale_factor, f.format, SUM(f.value_bytes)::float8 AS bytes + FROM file_sizes f + JOIN latest l + ON l.benchmark = f.benchmark + AND l.scale_factor = f.scale_factor + AND l.commit_sha = f.commit_sha + GROUP BY f.benchmark, f.scale_factor, f.format + ) + SELECT t.format AS format, + t.benchmark AS dataset, + t.scale_factor AS "datasetVariant", + t.bytes AS "valueBytes", + b.bytes AS "baselineBytes", + NULL::float8 AS "uncompressedBytes" + FROM totals t + JOIN totals b + ON b.benchmark = t.benchmark + AND b.scale_factor = t.scale_factor + AND b.format = $1 + WHERE t.bytes > 0 + AND b.bytes > 0 + ORDER BY t.format, t.benchmark, t.scale_factor + `; + const rows = (await getPool().query(text, [FILE_SIZE_BASELINE])).rows; + const rankings = compressionSizeRankings(rows); + if (rankings.length === 0) { + return null; + } + return { + type: 'compressionSize', + title: 'SQL File Size Summary', + rankings, + explanation: + 'Total size of the Vortex files each SQL benchmark writes, and geometric means of sizes versus vortex (lower is better)', + // The card names formats by their display names, which call `vortex-file-compressed` `vortex`. + baseline: 'vortex', + }; +} + async function collectQuerySummary( dataset: string, datasetVariant: string | null,