Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ The previous site generations (v2 and v3) are documented in [`docs/bench/legacy.
- **Numeric `?n=` snaps up to a rung of 25, 50, 100, 200, 500, or 1000; `?n=all` is the uncapped escape hatch.** Pages hydrate from the materialized latest-100 shard artifact by default; `?n=all` is an explicit opt-in (the chart's full-history zoom-out hop uses it once, and curl power users can request it). The numeric path is bounded by `MAX_NUMERIC_COMMIT_WINDOW` in [`src/bench/lib/window.ts`](src/bench/lib/window.ts) as a DoS-protection floor against `?n=99999999`, and a numeric `n` snaps up to the next rung of `COMMIT_WINDOW_RUNGS` there, so a chart or group has seven window URLs rather than a thousand, each a direct query that the CDN caches per URL. Do NOT raise the numeric cap, add rungs, or remove either without thinking about the DoS surface, and do NOT re-introduce a server-side commit cap on `?n=all` — visual downsampling happens client-side via LTTB on the visible commit range only. Only the default window is in the Data Cache, so every other window is a direct query that only the CDN shields. The read APIs and the chart page redirect (308) a non-default window to its one URL, with `n` spelled canonically and only the parameters the route reads (`canonicalWindowQuery` in `window.ts`), so adding a parameter or respelling `n` cannot get a request past the CDN. They also redirect every other spelling of a slug to the canonical one (`chartKeyToSlug`, `groupKeyToSlug`), since the CDN keys on the spelling, and the chart page cuts `engine` and `format` to the values bench knows, sorted.
- **Don't refetch on every scope change.** Once a chart's payload is in memory, pan/zoom/slider/range-strip all rebuild in place via the in-memory LTTB pass on the cached payload ([`src/bench/components/chart/controller.ts`](src/bench/components/chart/controller.ts)). The single exception is the latest-100 to full-history zoom-out path: when the user first zooms past the materialized window, the chart lazy-fetches `?n=all` once and replaces the payload in place.
- **Group docs links mirror the monorepo's `Benchmark::doc_path`.** [`src/bench/lib/benchmark-docs.ts`](src/bench/lib/benchmark-docs.ts) maps each SQL dataset name to the benchmark's explainer doc in `vortex-data/vortex`, which is the same path that the monorepo's PR benchmark comment uses. The producer does not put `doc_path` on the wire, and upstream does not enforce a path convention, so update the explicit mapping when a suite is added or its doc moves. Groups without a matching explainer render no docs link. Do not guess a path or link to an unrelated general guide.
- **Every group kind has a summary, by default.** [`src/bench/lib/summary.ts`](src/bench/lib/summary.ts) has no allowlist gate: a suite that lands in one of the five fact tables gets a rollup card from its first ingest. The three timing families (query, random access, vector search) rank through one `rankSeries` model, so a new suite in any of them needs no summary code at all. `collectGroupSummary`'s exhaustive switch makes a missing arm for a sixth fact table a compile error rather than a silently blank card. Do NOT reintroduce a per-dataset allowlist; the v2-era one is what left `spatialbench`, `fineweb`, `gharchive`, `appian`, `public-bi`, `clickbench-sorted`, and every vector-search group with no card.
- **Every group kind has a summary, by default.** [`src/bench/lib/summary.ts`](src/bench/lib/summary.ts) has no allowlist gate: a suite that lands in one of the six fact tables gets a rollup card from its first ingest. The three timing families (query, random access, vector search) rank through one `rankSeries` model, so a new suite in any of them needs no summary code at all. `collectGroupSummary`'s exhaustive switch makes a missing arm for a seventh fact table a compile error rather than a silently blank card. Do NOT reintroduce a per-dataset allowlist; the v2-era one is what left `spatialbench`, `fineweb`, `gharchive`, `appian`, `public-bi`, `clickbench-sorted`, and every vector-search group with no card.
- **A summary ranks the whole group, never one chart.** Random access is the cautionary case: the producer emits `dataset` as `{dataset}/{pattern}`, so the group holds ~nine charts, and the old summary published the alphabetically first chart's raw times under the group-wide title "Random Access Performance" — reporting `lance` at 352us when it is over 1ms on most of the other charts. Sum chart medians by dataset, then rank the dataset totals. Include the legacy taxi chart in the taxi total. Apply a penalty when a series lacks one dataset total. Report dataset coverage through `measured`/`total`. The secondary runtime is the arithmetic mean of complete dataset totals.
- **Don't write a server-side classifier for live ingest.** The emitter produces structured records directly. Classifying loose name strings at read time was the v2-era weakness every later generation existed to escape; it belongs nowhere in the live pipeline.
- **Every root-relative bench URL goes through `benchPath()`.** A link, a fetch, an icon or a selector that names one of bench's paths must work on bench.vortex.dev (from the root) and everywhere else (under `/bench`); see "Repository layout". A path read from `location.pathname` already carries the base.
Expand Down
6 changes: 3 additions & 3 deletions docs/bench/CONTRACT.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,8 @@ rows in the `String Encoding` groups and compares each encoder with `fsst`, so t
`file_size` records are not written by `vortex-bench`: the monorepo's
`scripts/capture-file-sizes.py` writes one per Vortex file a SQL benchmark leaves on disk, with
`benchmark` (the `vx-bench` subcommand), `scale_factor`, `format`, `file` (the path below the format
directory) and `value_bytes`. The read service does not chart them; CI reads them back as the develop
directory) and `value_bytes`. The read service charts each benchmark and scale factor in the
`SQL File Size` group, one line per format summed over its files. CI reads them back as the develop
baseline for the PR comment's file-size section.

`random_access_time` records include `open_mode`. The value is `cached` for a reused accessor
Expand Down Expand Up @@ -160,6 +161,5 @@ For wire/record *shape* changes, also update the producer snapshot fixtures in t
the same commit.

A new record kind with its own table, as `file_size` was in migration 014, changes no existing
shape, so it needs no bump: the read service never reads the new table, and older writers never
send the kind. Its migration must be applied before a writer sends it, because each JSONL file
shape, so it needs no bump: no existing read changes, and older writers never send the kind. Its migration must be applied before a writer sends it, because each JSONL file
lands in one transaction and an unknown table fails the whole file.
4 changes: 2 additions & 2 deletions docs/bench/architecture/design-decisions.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@ reasoning behind them. Each entry is "decision → why → where." Topic docs
## Data model

**One fact table per measurement family (not one wide table).**
The five families have genuinely different dimensions. A single wide table would
The six families have genuinely different dimensions. A single wide table would
bloat every row with NULL columns it doesn't use, or split one chart's data across
multiple rows that then need re-joining. Five focused tables keep each row dense
multiple rows that then need re-joining. Six focused tables keep each row dense
and each chart query a single-table scan.
→ `migrations/001_initial_schema.sql`, `server/src/schema.rs`.

Expand Down
1 change: 1 addition & 0 deletions src/bench/lib/benchmark-docs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ export function groupDocPath(key: GroupKey): string | null {
case 'QueryGroup':
return QUERY_DOCS[key.dataset] ?? null;
case 'VectorSearchGroup':
case 'FileSizeGroup':
return null;
}
}
4 changes: 2 additions & 2 deletions src/bench/lib/data-cache.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -69,9 +69,9 @@ describe('data-cache wrappers', () => {
expect(BENCH_DATA_TAG).toBe('bench-data');
expect(DATA_CACHE_BACKSTOP_SECONDS).toBe(86400);
expect(cacheCalls.map((call) => call.keyParts)).toContainEqual([
'data-cache:group-charts:v12:n100',
'data-cache:group-charts:v13:n100',
]);
expect(cacheCalls.map((call) => call.keyParts)).toContainEqual(['data-cache:groups:v12']);
expect(cacheCalls.map((call) => call.keyParts)).toContainEqual(['data-cache:groups:v13']);
for (const call of cacheCalls) {
expect(call.options.tags).toEqual([BENCH_DATA_TAG]);
expect(call.options.revalidate).toBe(DATA_CACHE_BACKSTOP_SECONDS);
Expand Down
2 changes: 1 addition & 1 deletion src/bench/lib/data-cache.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ const CACHE_OPTIONS = { tags: [BENCH_DATA_TAG], revalidate: DATA_CACHE_BACKSTOP_
* code can serve payloads the old code computed until the next ingest.
* It is independent of the producer-facing benchmark schema version.
*/
const GROUP_PAYLOAD_CACHE_VERSION = 'v12';
const GROUP_PAYLOAD_CACHE_VERSION = 'v13';

// The default last-100 group bundle, keyed by group slug. The slug is the cache
// key (an `unstable_cache` argument), so one wrapper covers every group. A
Expand Down
5 changes: 5 additions & 0 deletions src/bench/lib/descriptions.ts
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,11 @@ function staticDescription(name: string): string | null {
return 'Measures Vortex file write and scan throughput for each string encoder';
case 'String Encoding Size':
return 'Compares Vortex file sizes for each string encoder';
case 'SQL File Size':
return (
'Tracks the total size of the Vortex files each SQL benchmark writes, per benchmark and ' +
'scale factor'
);
case 'Clickbench':
return (
"ClickHouse's analytical benchmark suite testing real-world query patterns on web " +
Expand Down
4 changes: 3 additions & 1 deletion src/bench/lib/families.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,14 @@ import { describe, expect, it } from 'vitest';
import { FAMILIES, familyForChartKind, familyForGroupKind } from './families';

describe('FAMILIES registry', () => {
it('lists the five fact tables in family.rs declaration order', () => {
it('lists the six fact tables in family.rs declaration order, then file_sizes', () => {
expect(FAMILIES.map((f) => f.tableName)).toEqual([
'query_measurements',
'compression_times',
'compression_sizes',
'random_access_times',
'vector_search_runs',
'file_sizes',
]);
});

Expand All @@ -29,6 +30,7 @@ describe('FAMILIES registry', () => {
['cs', 'csg'],
['rat', 'rag'],
['vsr', 'vsg'],
['fs', 'fsg'],
]);
});
});
Expand Down
29 changes: 20 additions & 9 deletions src/bench/lib/families.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,43 +4,47 @@
/**
* Per-fact-table registry, the TypeScript port of `server/src/family.rs`.
*
* Each of the five fact tables is a [`Family`] that ties together the Postgres
* Each of the six fact tables is a [`Family`] that ties together the Postgres
* table name and the chart and group slug prefixes. The read endpoints
* dispatch through this registry rather than hand-listing the families, so the
* slug prefixes consumed by [`./slug`] have a single source of truth. The read
* queries name their tables in static SQL.
*
* The order of [`FAMILIES`] mirrors `family.rs`'s `FAMILIES` constant and the
* DDL apply order in `migrations/001_initial_schema.sql`.
* DDL apply order in `migrations/001_initial_schema.sql`, with `file_sizes`
* (`migrations/014`) last.
*/

/** The five fact-table names, spelled exactly as in `migrations/001`. */
/** The six fact-table names, spelled exactly as in their migrations. */
type FamilyTable =
| 'query_measurements'
| 'compression_times'
| 'compression_sizes'
| 'random_access_times'
| 'vector_search_runs';
| 'vector_search_runs'
| 'file_sizes';

/** Discriminant (`k`) of a [`./slug`].`ChartKey`; one per family. */
export type ChartKind =
| 'QueryMeasurement'
| 'CompressionTime'
| 'CompressionSize'
| 'RandomAccess'
| 'VectorSearch';
| 'VectorSearch'
| 'FileSize';

/** Discriminant (`k`) of a [`./slug`].`GroupKey`; one per family. */
export type GroupKind =
| 'QueryGroup'
| 'CompressionTimeGroup'
| 'CompressionSizeGroup'
| 'RandomAccessGroup'
| 'VectorSearchGroup';
| 'VectorSearchGroup'
| 'FileSizeGroup';

/** One fact-table family: its table name, slug prefixes, and key discriminants. */
export interface Family {
/** Postgres table name, matching `migrations/001_initial_schema.sql`. */
/** Postgres table name, matching its migration. */
tableName: FamilyTable;
/** Slug prefix for individual charts in this family (e.g. `qm`). */
chartSlugPrefix: string;
Expand All @@ -53,8 +57,8 @@ export interface Family {
}

/**
* All five fact-table families, in the order `family.rs`'s `FAMILIES` declares
* them (which is also the `migrations/001` DDL apply order).
* All six fact-table families, in the order `family.rs`'s `FAMILIES` declares
* them (which is also the `migrations/001` DDL apply order), then `file_sizes`.
*/
export const FAMILIES: readonly Family[] = [
{
Expand Down Expand Up @@ -92,6 +96,13 @@ export const FAMILIES: readonly Family[] = [
chartKind: 'VectorSearch',
groupKind: 'VectorSearchGroup',
},
{
tableName: 'file_sizes',
chartSlugPrefix: 'fs',
groupSlugPrefix: 'fsg',
chartKind: 'FileSize',
groupKind: 'FileSizeGroup',
},
];

/** Look up the family that owns a chart-key discriminant. */
Expand Down
53 changes: 52 additions & 1 deletion src/bench/lib/group-charts.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,11 @@ describe('seriesFor', () => {
{ flavor: 'vortex-turboquant' },
{ key: 'vortex-turboquant', engine: undefined, format: undefined },
],
[
'FileSize',
{ format: 'vortex-compact' },
{ key: 'vortex-compact', engine: undefined, format: 'vortex-compact' },
],
] as const)('names the %s series', (kind, cols, expected) => {
expect(seriesFor(kind, { commit_sha: 'a', value: 1, ...cols })).toEqual(expected);
});
Expand Down Expand Up @@ -468,6 +473,29 @@ describe.skipIf(!databaseAvailable)('groupChartPayloads at the window edges', ()
[measurementId, sha, dataset, variant, valueBytes],
);
}
// File sizes: TPC-H SF=1 has two files per format on C0, E and F (compact only from E);
// SF=10 has one file on F.
for (const [sha, scaleFactor, format, file, valueBytes] of [
[C0, '1.0', 'vortex-file-compressed', 'a.vortex', 100],
[C0, '1.0', 'vortex-file-compressed', 'b.vortex', 10],
[E, '1.0', 'vortex-file-compressed', 'a.vortex', 101],
[E, '1.0', 'vortex-file-compressed', 'b.vortex', 11],
[E, '1.0', 'vortex-compact', 'a.vortex', 80],
[E, '1.0', 'vortex-compact', 'b.vortex', 8],
[F, '1.0', 'vortex-file-compressed', 'a.vortex', 102],
[F, '1.0', 'vortex-file-compressed', 'b.vortex', 12],
[F, '1.0', 'vortex-compact', 'a.vortex', 79],
[F, '1.0', 'vortex-compact', 'b.vortex', 7],
[F, '10.0', 'vortex-file-compressed', 'a.vortex', 1000],
] as const) {
measurementId += 1;
await getPool().query(
`INSERT INTO file_sizes
(measurement_id, commit_sha, benchmark, scale_factor, format, file, value_bytes)
VALUES ($1, $2, 'tpch', $3, $4, $5, $6)`,
[measurementId, sha, scaleFactor, format, file, valueBytes],
);
}
});

afterAll(async () => {
Expand Down Expand Up @@ -563,10 +591,17 @@ describe.skipIf(!databaseAvailable)('groupChartPayloads at the window edges', ()
{ k: 'CompressionSize', dataset: 'sy', dataset_variant: 'v2' },
],
],
[
{ k: 'FileSizeGroup' },
[
{ k: 'FileSize', benchmark: 'tpch', scale_factor: '1.0' },
{ k: 'FileSize', benchmark: 'tpch', scale_factor: '10.0' },
],
],
];

it.each(['1', '2', '3', '4', 'all'])(
'matches chartPayload for random-access, vector-search and compression-size groups at ?n=%s',
'matches chartPayload for random-access, vector-search, compression-size and file-size groups at ?n=%s',
async (n) => {
const window = commitWindow(n);
for (const [group, keys] of OTHER_FAMILIES) {
Expand Down Expand Up @@ -608,6 +643,22 @@ describe.skipIf(!databaseAvailable)('groupChartPayloads at the window edges', ()
expect(short?.series).toEqual({ fl: [600] });
});

it('sums each file-size chart over its files, one line per format', async () => {
const [sf1, sf10] = await groupChartPayloads(
...defined(OTHER_FAMILIES[3]),
parseCommitWindow('all'),
);
expect(sf1?.display_name).toBe('tpch');
expect(sf1?.unit_kind).toBe('bytes');
expect(sf1?.series).toEqual({
'vortex-compact': [null, null, null, null, 88, 86],
'vortex-file-compressed': [110, null, null, null, 112, 114],
});
expect(sf10?.display_name).toBe('tpch sf=10.0');
expect(sf10?.commits.map((c) => c.sha)).toEqual([F]);
expect(sf10?.series).toEqual({ 'vortex-file-compressed': [1000] });
});

it('keeps sibling groups apart', async () => {
const window = parseCommitWindow('all');
const [g] = await groupChartPayloads(G, [queryChart(G, 1)], window);
Expand Down
Loading
Loading