Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ The previous site generations (v2 and v3) are documented in [`docs/bench/legacy.
- **Don't refetch on every scope change.** Once a chart's payload is in memory, pan/zoom/slider/range-strip all rebuild in place via the in-memory LTTB pass on the cached payload ([`src/bench/components/chart/controller.ts`](src/bench/components/chart/controller.ts)). The single exception is the latest-100 to full-history zoom-out path: when the user first zooms past the materialized window, the chart lazy-fetches `?n=all` once and replaces the payload in place.
- **Group docs links mirror the monorepo's `Benchmark::doc_path`.** [`src/bench/lib/benchmark-docs.ts`](src/bench/lib/benchmark-docs.ts) maps each SQL dataset name to the benchmark's explainer doc in `vortex-data/vortex`, which is the same path that the monorepo's PR benchmark comment uses. The producer does not put `doc_path` on the wire, and upstream does not enforce a path convention, so update the explicit mapping when a suite is added or its doc moves. Groups without a matching explainer render no docs link. Do not guess a path or link to an unrelated general guide.
- **Every group kind has a summary, by default.** [`src/bench/lib/summary.ts`](src/bench/lib/summary.ts) has no allowlist gate: a suite that lands in one of the six fact tables gets a rollup card from its first ingest. The three timing families (query, random access, vector search) rank through one `rankSeries` model, so a new suite in any of them needs no summary code at all. `collectGroupSummary`'s exhaustive switch makes a missing arm for a seventh fact table a compile error rather than a silently blank card. Do NOT reintroduce a per-dataset allowlist; the v2-era one is what left `spatialbench`, `fineweb`, `gharchive`, `appian`, `public-bi`, `clickbench-sorted`, and every vector-search group with no card.
- **A summary ranks the whole group, never one chart.** Random access is the cautionary case: the producer emits `dataset` as `{dataset}/{pattern}`, so the group holds ~nine charts, and the old summary published the alphabetically first chart's raw times under the group-wide title "Random Access Performance" — reporting `lance` at 352us when it is over 1ms on most of the other charts. Sum chart medians by dataset, then rank the dataset totals. Include the legacy taxi chart in the taxi total. Apply a penalty when a series lacks one dataset total. Report dataset coverage through `measured`/`total`. The secondary runtime is the arithmetic mean of complete dataset totals.
- **A summary ranks the whole group, never one chart.** Random access is the cautionary case: the producer emits `dataset` as `{dataset}/{pattern}`, so the group holds ~nine charts, and the old summary published the alphabetically first chart's raw times under the group-wide title "Random Access Performance" — reporting `lance` at 352us when it is over 1ms on most of the other charts. Sum chart medians by dataset, then rank the dataset totals. Include the legacy taxi chart in the taxi total. Apply a penalty when a series lacks one dataset total. Report dataset coverage through `measured`/`total`. The secondary runtime is the arithmetic mean of complete dataset totals. In a query summary, the penalty is not enough: a series that misses a query ranks below every series that ran them all, so a series cannot rank first by failing or skipping the hardest query (Lance skips TPC-DS q72).
- **Don't write a server-side classifier for live ingest.** The emitter produces structured records directly. Classifying loose name strings at read time was the v2-era weakness every later generation existed to escape; it belongs nowhere in the live pipeline.
- **Every root-relative bench URL goes through `benchPath()`.** A link, a fetch, an icon or a selector that names one of bench's paths must work on bench.vortex.dev (from the root) and everywhere else (under `/bench`); see "Repository layout". A path read from `location.pathname` already carries the base.

Expand Down
27 changes: 23 additions & 4 deletions docs/bench/CONTRACT.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,10 @@ column-by-column; an unknown `kind` or a missing dimension field fails the inges

`compression_size` records include `uncompressed_bytes`, the Arrow memory size after the
source Parquet file is decoded. The column is nullable so historical rows remain valid.
`compress-bench` also writes two size-only formats, with no `compression_time` records:
`parquet-snappy` (Parquet as DuckDB writes it, with Snappy; `parquet` is zstd level 3) and
`vortex-compact` (Vortex with the compact strategy). Each is one write per dataset that is not
timed.

`string-bench` shares the two compression tables with `compress-bench`. Its records use the
dataset `strings/<column>`, the string encoder (`onpair-12`, `fsst`, ...) as `dataset_variant`,
Expand All @@ -73,11 +77,26 @@ rows in the `String Encoding` groups and compares each encoder with `fsst`, so t
`fsst` label are part of the contract.

`file_size` records are not written by `vortex-bench`: the monorepo's
`scripts/capture-file-sizes.py` writes one per Vortex file a SQL benchmark leaves on disk, with
`scripts/capture-file-sizes.py` writes one per data file a SQL benchmark leaves on disk, with
`benchmark` (the `vx-bench` subcommand), `scale_factor`, `format`, `file` (the path below the format
directory) and `value_bytes`. The read service charts each benchmark and scale factor in the
`SQL File Size` group, one line per format summed over its files. CI reads them back as the develop
baseline for the PR comment's file-size section.
directory) and `value_bytes`. A file's `format` is the name of its format directory, not its
codec. `parquet` is the suite's base Parquet: zstd level 3 where the benchmarks write it, and the
dataset's published files where the suite prescribes them, as ClickBench's are Snappy.
`parquet-snappy` is Parquet as the DuckDB CLI writes it, with Snappy, from the base Parquet. CI
reads the records back as the develop baseline for the PR comment's file-size section.

The read service charts both tables in the `File Sizes` group, one line per format. It charts every
current `compress-bench` dataset in `compression_sizes`, and three `file_sizes` scopes, each summed
over its files: ClickBench, TPC-H at SF10, and TPC-DS at SF1 (`src/bench/lib/file-sizes.ts`). Its
summary compares each format with zstd Parquet on the `compress-bench` datasets and on TPC-DS.
On develop, TPC-DS SF1 has every format `compress-bench` measures. It queries DuckDB's Snappy
Parquet (`parquet-snappy`) on DataFusion and DuckDB and Lance on DataFusion, and writes Arrow IPC after its
queries, for its size only. Lance skips q72, which it cannot run; the query summary ranks a series
that misses a query below every series that ran them all.

The `SQL File Sizes` group charts every `file_sizes` scope, at every scale factor, and leaves out
`compression_sizes`. Its summary compares each format with `vortex-file-compressed`, per scope at
that scope's newest commit with a Vortex size.

`random_access_time` records include `open_mode`. The value is `cached` for a reused accessor
or `reopen` when each timed take includes a new accessor. Historical rows use `cached`.
Expand Down
2 changes: 1 addition & 1 deletion docs/bench/architecture/read-path.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,7 +120,7 @@ the v3 Axum server (so the frontend is generation-agnostic). The landing-page
order is the curated `GROUP_ORDER` in `queries.ts`:

```
Compression, Compression Size, Random Access, Clickbench,
Compression, File Sizes, SQL File Sizes, Random Access, Clickbench,
TPC-H (NVMe/S3) SF=1, SF=10, SF=100 (alternating),
TPC-DS (NVMe) SF=1,
Statistical and Population Genetics, Polar Signals Profiling,
Expand Down
8 changes: 7 additions & 1 deletion lighthouserc.cjs
Original file line number Diff line number Diff line change
Expand Up @@ -145,7 +145,13 @@ module.exports = {
settings: {
preset: "desktop",
throttlingMethod: "simulate",
skipAudits: ["uses-http2"]
skipAudits: ["uses-http2"],
// Vercel adds its Toolbar to preview deployments, even with the
// proxy's `x-vercel-skip-toolbar` header. It loads from vercel.live
// (and Pusher, for live comments), and its third-party cookies and
// issues cut Best Practices to 0.78 on every page from 2026-11-20.
// The site never loads either host, so block them.
blockedUrlPatterns: ["*vercel.live*", "*pusher.com*"]
}
},
assert: {
Expand Down
36 changes: 36 additions & 0 deletions src/bench/components/SummaryCard.test.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -152,6 +152,8 @@ describe('SummaryCard', () => {
maxRatio: 1,
compressionRatio: 4,
totalBytes: null,
measured: 1,
total: 1,
},
],
explanation: 'e',
Expand All @@ -160,6 +162,30 @@ describe('SummaryCard', () => {
expect(html).toContain('class="compression-size-heading compression-size-total-unit"></span>');
});

it('says how many datasets a partly measured size ranking covers', () => {
const ranking = (name: string, measured: number) => ({
name,
minRatio: 1,
ratio: 1,
maxRatio: 1,
compressionRatio: null,
totalBytes: measured === 2 ? 10 : null,
measured,
total: 2,
});
const html = render({
type: 'compressionSize',
title: 'File Sizes Summary',
rankings: [ranking('parquet', 2), ranking('vortex-compact', 1)],
explanation: 'e',
});
expect(html).toContain('title="vortex-compact · measured on 1 of 2 datasets"');
expect(html).toContain('<span aria-hidden="true"> (1/2)</span>');
expect(html).toContain('<span class="vx-sr-only">, measured on 1 of 2 datasets</span>');
// A complete ranking carries no coverage.
expect(html.match(/measured on/g)).toHaveLength(2);
});

it('renders nothing for a compression card with no rankings', () => {
expect(render({ type: 'compression', title: 't', rankings: [], explanation: 'e' })).toBe('');
});
Expand All @@ -176,6 +202,8 @@ describe('SummaryCard', () => {
maxRatio: 1.55,
compressionRatio: 8.25,
totalBytes: 1_603_980_000,
measured: 1,
total: 1,
},
{
name: 'parquet',
Expand All @@ -184,6 +212,8 @@ describe('SummaryCard', () => {
maxRatio: 1,
compressionRatio: 4.5,
totalBytes: 1_428_150_000,
measured: 1,
total: 1,
},
{
name: 'lance',
Expand All @@ -192,6 +222,8 @@ describe('SummaryCard', () => {
maxRatio: 1.3,
compressionRatio: null,
totalBytes: 2_500_000,
measured: 1,
total: 1,
},
],
explanation: 'higher is better',
Expand Down Expand Up @@ -311,6 +343,8 @@ describe('SummaryCard', () => {
maxRatio: 1,
compressionRatio: 8,
totalBytes: 100,
measured: 1,
total: 1,
},
{
name: 'lance',
Expand All @@ -319,6 +353,8 @@ describe('SummaryCard', () => {
maxRatio: 1,
compressionRatio: 2,
totalBytes: 200,
measured: 1,
total: 1,
},
],
explanation: 'e',
Expand Down
15 changes: 14 additions & 1 deletion src/bench/components/SummaryCard.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -144,14 +144,27 @@ export function SummaryCard({ summary }: { summary?: Summary | undefined }) {
<div className="compression-size-arrow-list">
{summary.rankings.map((item, idx) => {
const label = displayFormat(item.name);
// A format measured on only some datasets has its means over those alone, so
// the row says how many, and it ranks after the complete ones.
const partial = item.measured < item.total;
const coverage = `measured on ${item.measured} of ${item.total} datasets`;
return (
<div
className={rowClass('score-item compression-size-arrow-row', item.name)}
key={item.name}
>
<span className="score-rank">#{idx + 1}</span>
<span className="score-series" title={label}>
<span
className="score-series"
title={partial ? `${label} · ${coverage}` : label}
>
{label}
{partial ? (
<>
<span aria-hidden="true">{` (${item.measured}/${item.total})`}</span>
<span className="vx-sr-only">{`, ${coverage}`}</span>
</>
) : null}
</span>
<span className="score-runtime compression-size-value">
{item.totalBytes === null || totalUnit === null ? (
Expand Down
24 changes: 24 additions & 0 deletions src/bench/lib/chart-format.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -139,6 +139,30 @@ describe('seriesStyle', () => {
}
});

it("draws a Parquet codec variant dotted, in Parquet's color, thinner, behind Parquet", () => {
for (const theme of ['dark', 'light'] as const) {
const zstd = seriesStyle('parquet', { format: 'parquet' }, theme);
const snappy = seriesStyle('parquet-snappy', { format: 'parquet-snappy' }, theme);
expect(snappy.color).toBe(zstd.color);
expect(snappy.dash).toEqual([2, 3]);
expect(snappy.width).toBeLessThan(zstd.width);
}
const order = (format: string) => seriesOrder(format, { format });
expect(order('parquet')).toBeLessThan(order('parquet-snappy'));
expect(order('parquet-snappy')).toBeLessThan(order('vortex-compact'));
});

it("keeps a query chart's four Parquet lines apart", () => {
const styles = ['parquet', 'parquet-snappy'].flatMap((format) =>
['datafusion', 'duckdb'].map((engine) =>
seriesStyle(`${engine}:${format}`, { engine, format }, 'dark'),
),
);
const looks = new Set(styles.map((style) => `${style.width}/${style.dash.join(',')}`));
expect(looks.size).toBe(4);
expect(seriesStyle('duckdb:parquet-snappy', null, 'dark').dash).toEqual([6, 3, 2, 3]);
});

it('draws a cold random-access run dashed and a hot one solid', () => {
const cold = seriesStyle('parquet:cold', { format: 'parquet' }, 'dark');
const hot = seriesStyle('parquet:hot', { format: 'parquet' }, 'dark');
Expand Down
43 changes: 38 additions & 5 deletions src/bench/lib/chart-format.ts
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,8 @@ export const MAX_VISIBLE_POINTS = 500;
// purple, DuckDB green, and Arrow orange. The color means the format and
// nothing else, so a format looks the same on every engine: the line style
// shows the engine instead. A duckdb run is dashed, as is a cold random-access
// run (the two never share a chart). Widths carry the importance tiers.
// run (the two never share a chart). A Parquet codec variant is dotted, or
// dash-dotted on duckdb. Widths carry the importance tiers.
//
// The palette has one set of colors per card theme, because one set cannot
// stay legible on both cards: a color with 3:1 contrast against both the
Expand Down Expand Up @@ -113,6 +114,10 @@ export interface SeriesStyle {

/** The dash of a duckdb run and of a cold random-access run. */
const DASH: readonly number[] = [6, 4];
/** The dash of a Parquet codec variant, which a query chart can hold on both
* engines beside zstd Parquet: dotted, and dash-dotted on duckdb. */
const DOT: readonly number[] = [2, 3];
const DASH_DOT: readonly number[] = [6, 3, 2, 3];

/** Line widths per importance tier, reinforcing the color hierarchy. */
const HERO_WIDTH = 2.6;
Expand Down Expand Up @@ -160,6 +165,17 @@ const FORMAT: Record<string, FormatStyle> = {
'arrow-ipc': ARROW,
};

/**
* The format a codec variant is drawn as: Parquet with a codec other than zstd
* (`parquet-snappy`, `parquet-uncompressed`, as the size tables and TPC-DS's
* queries name them) is Parquet's teal, thinner and dotted, so it reads as a
* kind of Parquet without a color of its own. Its dots differ from duckdb's
* dash, so a query chart's four Parquet lines stay apart.
*/
function codecVariantOf(format: string): string | undefined {
return format.startsWith('parquet-') ? 'parquet' : undefined;
}

/** The catch-all style for an otherwise-unknown series: a neutral gray with
* 3:1 contrast on both cards. */
const FALLBACK_COLOR = '#8a8a87';
Expand Down Expand Up @@ -190,12 +206,24 @@ export function seriesStyle(
theme: ChartTheme,
): SeriesStyle {
const { engine, format } = seriesDims(name, meta);
const dash = engine === 'duckdb' || name.endsWith(':cold') ? [...DASH] : [];
const style = format ? FORMAT[format] : undefined;
const variantOf = format === undefined ? undefined : codecVariantOf(format);
const duckdbOrCold = engine === 'duckdb' || name.endsWith(':cold');
const dash =
variantOf !== undefined
? [...(engine === 'duckdb' ? DASH_DOT : DOT)]
: duckdbOrCold
? [...DASH]
: [];
const base = variantOf ?? format;
const style = base ? FORMAT[base] : undefined;
if (!style) {
return { color: FALLBACK_COLOR, width: MUTED_WIDTH, dash };
}
return { color: style.color[theme], width: style.width, dash };
return {
color: style.color[theme],
width: variantOf === undefined ? style.width : MUTED_WIDTH,
dash,
};
}

/** Format draw priority, front (low) to back (high) -- Chart.js renders a lower
Expand Down Expand Up @@ -231,7 +259,12 @@ export function seriesOrder(
meta: { engine?: string; format?: string } | null | undefined,
): number {
const { engine, format } = seriesDims(name, meta);
const formatRank = rankIn(FORMAT_ORDER, format, FORMAT_ORDER_FALLBACK);
// A codec variant sits just behind its format.
const variantOf = format === undefined ? undefined : codecVariantOf(format);
const formatRank =
variantOf === undefined
? rankIn(FORMAT_ORDER, format, FORMAT_ORDER_FALLBACK)
: rankIn(FORMAT_ORDER, variantOf, FORMAT_ORDER_FALLBACK) + 0.5;
const engineRank = rankIn(ENGINE_ORDER, engine, 0);
return formatRank * 10 + engineRank;
}
Expand Down
Loading