diff --git a/CHANGELOG.md b/CHANGELOG.md index 690c979..fc92509 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,14 @@ # Changelog +## November 2026: parser refresh, build cost, and allocation pressure + +- Benchmarked parser versions: sqlparser-rs 0.64 (git 8a176d98), pg_query.rs 18.1.0 on libpg_query 18 (git 20f3a182), polyglot-sql 0.14.1 (git 866e97bd), qusql-parse 0.12, sqlglot-rust 0.11.1, turso_parser 0.8.3, with databend-common-ast 0.2.5 and sqlite3-parser 0.17 already at their newest releases. +- The time machine gains seven release points (sqlparser-rs 0.64, polyglot-sql 0.12.1, 0.13.5 and 0.14.1, sqlglot-rust 0.11.1, qusql-parse 0.12, turso_parser 0.8.3) for 102 versions in total, and the sqlglot-rust 0.10 milestone moves to 0.10.33. +- pg_query 18 takes explicit parser and deparse options. The benchmark passes `ParserOptions::DEFAULT` and `DeparseOptions::default()`, the standard grammar and plain deparse. +- Build cost: every benchmarked release, current and historical, is compiled as a minimal stdin-parse program from a clean target directory with the workspace lockfile. The site reports CPU seconds of the whole build, wall seconds at a fixed job count, the parser crate's own compile time, the number of crates compiled, and the stripped binary size the parser adds, on the overview, the parser pages, and the time-machine trends. +- Allocation pressure: the counting allocator also records allocator calls and bytes requested per statement, freed or not, next to peak and retained memory, in the dialect and parser tables, the batch figures, and the time-machine trends. +- The counting allocator lives once in `src/mem.rs`, shared by both memory benches, and the workspace no longer forbids `unsafe` code. + ## September 2026: parser refresh - Updated the benchmarked parsers to their latest versions: sqlparser-rs 0.62 to 0.63 (git 9296011a), polyglot-sql 0.5.1 to 0.11.0 (git 47342ba6), pg_query.rs 6.1.1 to 6.2.0 (git 0e382742), qusql-parse 0.8 to 0.11, sqlglot-rust 0.10.1 to 0.10.29, sqlite3-parser 0.16 to 0.17, turso_parser 0.6.1 to 0.7.2. databend-common-ast and orql were already at their newest releases. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 966acaf..94a70bb 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -10,7 +10,7 @@ cargo fmt --all cargo clippy --all-targets ``` -No unsafe code is allowed (`unsafe_code = "forbid"`). Clippy runs with pedantic and nursery lints enabled. The workspace has three crates: the native benchmark (root), `viz` (wasm-clean shared schema + plotters chart renderers), and `web` (the Dioxus viewer). `default-members` keeps bare `cargo test`/`bench`/`clippy` on the native crate only. +Clippy runs with pedantic and nursery lints enabled. The only `unsafe` is the counting global allocator in `src/mem.rs`, installed by the memory benches. The workspace has three crates: the native benchmark (root), `viz` (wasm-clean shared schema + plotters chart renderers), and `web` (the Dioxus viewer). `default-members` keeps bare `cargo test`/`bench`/`clippy` on the native crate only. ## Results website diff --git a/Cargo.toml b/Cargo.toml index 36aab8a..40833a2 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -21,9 +21,6 @@ categories = ["development-tools::profiling"] publish = false rust-version.workspace = true -[lints.rust] -unsafe_code = "forbid" - [lints.clippy] all = "warn" pedantic = "warn" @@ -36,12 +33,12 @@ cast_sign_loss = { level = "allow", priority = 1 } sqlparser = { git = "https://github.com/sqlparser-rs/sqlparser-rs" } # libpg_query, the PostgreSQL server's own parser, the reference for PostgreSQL. pg_query = { git = "https://github.com/pganalyze/pg_query.rs" } -qusql-parse = "0.11.0" +qusql-parse = "0.12.0" polyglot-sql = { git = "https://github.com/tobilg/polyglot" } databend-common-ast = "0.2.5" -sqlglot-rust = "0.10.29" +sqlglot-rust = "0.11.1" sqlite3-parser = "0.17.0" -turso_parser = "0.7.2" +turso_parser = "0.8.3" fallible-iterator = "0.3.0" viz = { path = "viz" } serde = { version = "1", features = ["derive"] } diff --git a/README.md b/README.md index 4e6fe28..0352fdc 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ Choosing a SQL parser for a Rust project means weighing dialect coverage, correc We evaluated eight parser libraries: [sqlparser-rs](https://github.com/sqlparser-rs/sqlparser-rs) (Apache DataFusion), [pg_query.rs](https://github.com/pganalyze/pg_query.rs) and its faster summary mode (Rust bindings to [libpg_query](https://github.com/pganalyze/libpg_query), PostgreSQL's own parser), [databend-common-ast](https://crates.io/crates/databend-common-ast), [polyglot-sql](https://github.com/tobilg/polyglot), [sqlglot-rust](https://crates.io/crates/sqlglot-rust), [qusql-parse](https://crates.io/crates/qusql-parse), [sqlite3-parser](https://crates.io/crates/sqlite3-parser) (lemon-rs), and [turso_parser](https://crates.io/crates/turso_parser) (the SQLite parser from Turso). We ran them against a corpus of 340,938 statements spanning 13 dialects, drawn from each engine's own regression suites and official samples and committed compressed so every run is reproducible. -We exercised each parser in the dialect that matches the corpus under test. Where a dialect has a runnable engine, we labelled each statement valid or invalid with the real database engine itself, run in Docker via [testcontainers](https://github.com/testcontainers/testcontainers-rs): a statement counts as valid unless the engine reports a syntax error, so a missing table or column still counts as parsed. Against that ground truth we scored the parsers on recall (valid statements accepted), false positives (invalid statements wrongly accepted), and display round-trip stability. The other dialects have no runnable engine, so their statements count as provenance-valid and the metric is simply the acceptance rate. Across all dialects, we captured speed as a per-statement parse-time distribution over every accepted statement, and memory as the peak and retained bytes per statement under a counting allocator. A batch axis additionally parses each parser's whole accepted set as a single script, showing what bulk parsing amortizes, and a time machine benchmarks the historical releases of every pure-Rust parser (95 versions across 7 families, including every sqlparser-rs minor since July 2020), so each parser page also charts how coverage, speed, and memory evolved across releases. +We exercised each parser in the dialect that matches the corpus under test. Where a dialect has a runnable engine, we labelled each statement valid or invalid with the real database engine itself, run in Docker via [testcontainers](https://github.com/testcontainers/testcontainers-rs): a statement counts as valid unless the engine reports a syntax error, so a missing table or column still counts as parsed. Against that ground truth we scored the parsers on recall (valid statements accepted), false positives (invalid statements wrongly accepted), and display round-trip stability. The other dialects have no runnable engine, so their statements count as provenance-valid and the metric is simply the acceptance rate. Across all dialects, we captured speed as a per-statement parse-time distribution over every accepted statement, and memory as the peak and retained bytes and the allocation pressure (allocator calls and bytes requested) per statement under a counting allocator. Build cost is the compile time and added binary size of a minimal program using each release. A batch axis additionally parses each parser's whole accepted set as a single script, showing what bulk parsing amortizes, and a time machine benchmarks the historical releases of every pure-Rust parser (102 versions across 7 families, including every sqlparser-rs minor since July 2020), so each parser page also charts how coverage, speed, memory, and build cost evolved across releases. On their home dialect the reference bindings are exact by construction, so the more telling comparison is among the pure-Rust parsers. There, [sqlparser-rs](https://github.com/sqlparser-rs/sqlparser-rs) is the most broadly capable. Permissiveness turns out to be expensive. [sqlglot-rust](https://crates.io/crates/sqlglot-rust) recalls 98 percent of valid statements yet wrongly accepts 85 percent of invalid ones, [polyglot-sql](https://github.com/tobilg/polyglot) reaches 92 percent recall at 50 percent false positives, and the two SQLite lemon parsers, [sqlite3-parser](https://crates.io/crates/sqlite3-parser) and [turso_parser](https://crates.io/crates/turso_parser), accept nearly every valid statement while still wrongly accepting six in ten. The strictest, [databend-common-ast](https://crates.io/crates/databend-common-ast) and [qusql-parse](https://crates.io/crates/qusql-parse), hold false positives to 6 and 12 percent but recall only 53 and 74 percent. Speed spans more than an order of magnitude, from half a microsecond to just under one for the fastest, [qusql-parse](https://crates.io/crates/qusql-parse) and [turso_parser](https://crates.io/crates/turso_parser), to the six to nine microsecond band occupied by [polyglot-sql](https://github.com/tobilg/polyglot), [sqlparser-rs](https://github.com/sqlparser-rs/sqlparser-rs) and [databend-common-ast](https://crates.io/crates/databend-common-ast). No parser leads on every axis, so the right choice comes down to what a given project values most: broad coverage, few false positives, or raw speed. @@ -22,14 +22,14 @@ On their home dialect the reference bindings are exact by construction, so the m | Parser | Version | Source | Implementation | Dialects | | --- | --- | --- | --- | --- | -| **[sqlparser-rs](https://github.com/sqlparser-rs/sqlparser-rs)** | 0.63.0 | git [`9296011a`](https://github.com/sqlparser-rs/sqlparser-rs/commit/9296011a) | Pure Rust, handwritten recursive descent | 14 dedicated dialects | -| **[sqlglot-rust](https://crates.io/crates/sqlglot-rust)** | 0.10.29 | crates.io | Pure Rust, standalone port of Python sqlglot | 30 (parser currently dialect-agnostic) | -| **[polyglot-sql](https://github.com/tobilg/polyglot)** | 0.11.0 | git [`47342ba6`](https://github.com/tobilg/polyglot/commit/47342ba6) | Pure Rust, transpiler | 32 | -| **[pg_query.rs](https://github.com/pganalyze/pg_query.rs)** | 6.2.0 | git [`0e382742`](https://github.com/pganalyze/pg_query.rs/commit/0e382742b36ee9592438c6791b91ce14f32f6233) | Rust FFI to C (libpg_query) | PostgreSQL | -| **[qusql-parse](https://crates.io/crates/qusql-parse)** | 0.11.0 | crates.io | Pure Rust, zero-copy | PostgreSQL, MariaDB/MySQL, SQLite | +| **[sqlparser-rs](https://github.com/apache/datafusion-sqlparser-rs)** | 0.64.0 | git [`8a176d98`](https://github.com/apache/datafusion-sqlparser-rs/commit/8a176d98) | Pure Rust, handwritten recursive descent | 14 dedicated dialects | +| **[sqlglot-rust](https://crates.io/crates/sqlglot-rust)** | 0.11.1 | crates.io | Pure Rust, standalone port of Python sqlglot | 30 (parser currently dialect-agnostic) | +| **[polyglot-sql](https://github.com/tobilg/polyglot)** | 0.14.1 | git [`866e97bd`](https://github.com/tobilg/polyglot/commit/866e97bd) | Pure Rust, transpiler | 32 | +| **[pg_query.rs](https://github.com/pganalyze/pg_query.rs)** | 18.1.0 | git [`20f3a182`](https://github.com/pganalyze/pg_query.rs/commit/20f3a182dc74ef0406752548bc6fb79fbfb591b1) | Rust FFI to C (libpg_query) | PostgreSQL | +| **[qusql-parse](https://crates.io/crates/qusql-parse)** | 0.12.0 | crates.io | Pure Rust, zero-copy | PostgreSQL, MariaDB/MySQL, SQLite | | **[databend-common-ast](https://github.com/datafuselabs/databend)** | 0.2.5 | crates.io | Pure Rust, zero-copy, Pratt | PostgreSQL, MySQL, Hive | | **[sqlite3-parser](https://crates.io/crates/sqlite3-parser)** (lemon-rs) | 0.17.0 | crates.io | Generated from SQLite's Lemon grammar | SQLite | -| **[turso_parser](https://crates.io/crates/turso_parser)** | 0.7.2 | crates.io | Pure Rust, handwritten recursive descent over a Lemon token table | SQLite | +| **[turso_parser](https://crates.io/crates/turso_parser)** | 0.8.3 | crates.io | Pure Rust, handwritten recursive descent over a Lemon token table | SQLite | Per-parser repository metadata (stars, contributors, fuzzing, test and benchmark suites, license) is shown on each parser page in the [explorer](https://sql-ast-benchmark.luca.phd). @@ -37,7 +37,7 @@ Per-parser repository metadata (stars, contributors, fuzzing, test and benchmark 340,938 statements across 32 files and 13 dialects, committed compressed as `datasets.tar.zst` (5.6 MB) and unpacked to `datasets/{dialect}/{name}.txt`, one statement per line. The commands below extract it automatically on first use. All sources are openly licensed (Apache-2.0, MIT, BSD, public domain or CC-BY), drawn from each engine's own regression suites and official samples. The SQLite corpus includes the SQLite project's own official test suite (public domain), which exercises SQLite-specific grammar such as PRAGMAs, virtual tables, recursive CTEs, and upsert. Natural-language-with-embedded-SQL datasets are intentionally excluded. -Correctness is defined per dialect. Dialects with a runnable engine are graded against that real database engine, run in Docker via testcontainers by the `oracle` crate: a statement is valid unless the engine reports a syntax error (a missing table or column still counts as parsed). The validity labels are computed once and committed under `oracle/labels`, so grading and CI need no Docker. That reference splits the corpus into valid and invalid and scores recall, false positives, and round-trip. Dialects with no runnable engine (cloud services, heavy JVM engines) have no reference, so their statements count as provenance-valid (sourced from each engine's own suites) and the metric is acceptance rate. Speed is a per-statement parse-time distribution over every accepted statement, timed with an adaptive iteration count on a no-`catch_unwind` path. Memory is measured separately with a counting allocator, as peak live bytes and retained (AST) bytes per statement. A companion batch axis parses each parser's whole accepted set as one script and normalizes the time and memory by the statement count, showing what bulk parsing amortizes against parsing one statement at a time. A batch that does not parse the whole set (a parser that bails out partway) is dropped rather than reported, and parsers without a multi-statement entry point (databend-common-ast) sit out the batch axis. +Correctness is defined per dialect. Dialects with a runnable engine are graded against that real database engine, run in Docker via testcontainers by the `oracle` crate: a statement is valid unless the engine reports a syntax error (a missing table or column still counts as parsed). The validity labels are computed once and committed under `oracle/labels`, so grading and CI need no Docker. That reference splits the corpus into valid and invalid and scores recall, false positives, and round-trip. Dialects with no runnable engine (cloud services, heavy JVM engines) have no reference, so their statements count as provenance-valid (sourced from each engine's own suites) and the metric is acceptance rate. Speed is a per-statement parse-time distribution over every accepted statement, timed with an adaptive iteration count on a no-`catch_unwind` path. Memory is measured separately with a counting allocator, as peak live bytes, retained (AST) bytes, allocator calls, and bytes requested per statement. The last two are the allocation pressure, which a small peak hides when a parser frees and reuses memory as it goes. A companion batch axis parses each parser's whole accepted set as one script and normalizes the time and memory by the statement count, showing what bulk parsing amortizes against parsing one statement at a time. A batch that does not parse the whole set (a parser that bails out partway) is dropped rather than reported, and parsers without a multi-statement entry point (databend-common-ast) sit out the batch axis. Build cost compiles, for every benchmarked release, a minimal program that parses stdin, from a clean target directory with the workspace lockfile, and records CPU and wall seconds, the parser crate's own compile time, and the stripped binary size it adds over an empty program. ## Running @@ -53,7 +53,8 @@ That is an alias (see `.cargo/config.toml`) for `cargo run --release --bin sqlbe cargo run --release --bin sqlbench correctness --per-file # per-file acceptance, every dialect cargo run --release --bin sqlbench correctness # reference + provenance correctness cargo bench # parse time (per-statement and batch), every dialect -cargo run --release -p membench # per-statement memory (peak + retained bytes) +cargo run --release -p featurescan --bin featurescan-build # build cost of every release (writes featurescan/data/build.json) +cargo run --release -p membench # per-statement memory and allocation pressure cargo run --release -p membench -- batch # whole-script (batch) memory, per statement cargo run --release -p timemachine --bin timemachine-mem -- --full # per-version memory (writes a sidecar) cargo run --release -p timemachine --bin timemachine -- --full # per-version time + correctness, writes history @@ -62,7 +63,7 @@ cargo run --release --bin sqlbench export # regenerate web `cargo bench` runs both the per-statement (`parsing`) and whole-script (`batch_parsing`) timing benches. Add `--bench batch_parsing` to run only the batch one. `export` reads whatever the benches left under `target/`, warning rather than failing for any missing source, so the memory and batch columns stay empty until their producers have run. -The `timemachine` crate benchmarks several historical versions of each pure-Rust parser at once (via `package`-rename aliases in `timemachine/Cargo.toml`) and writes a compressed `web/assets/history.json.zst` that the explorer embeds and decompresses in the browser. Each parser page then shows how that library's time, memory, and correctness changed across releases, with a version picker. Cargo can only host semver-incompatible versions side by side, so the milestones are the latest patch of every `0.x` minor the shared adapter compiles against: `sqlparser-rs` gets 58 points (every minor from 0.6.1, July 2020, through 0.63), `sqlite3-parser` nine, `qusql-parse` ten (its 0.1.0 release parses pathologically slowly on parts of the corpus and is excluded), `polyglot-sql` eleven, `databend-common-ast` three, `sqlglot-rust` two, `turso_parser` two. The FFI parsers (`pg_query`) are excluded because two builds of libpg_query collide at link. Without `--full` the runners use a small per-dialect sample, which is a fast pipeline check rather than publishable numbers. +The `timemachine` crate benchmarks several historical versions of each pure-Rust parser at once (via `package`-rename aliases in `timemachine/Cargo.toml`) and writes a compressed `web/assets/history.json.zst` that the explorer embeds and decompresses in the browser. Each parser page then shows how that library's time, memory, and correctness changed across releases, with a version picker. Cargo can only host semver-incompatible versions side by side, so the milestones are the latest patch of every `0.x` minor the shared adapter compiles against: `sqlparser-rs` gets 59 points (every minor from 0.6.1, July 2020, through 0.64), `sqlite3-parser` nine, `qusql-parse` eleven (its 0.1.0 release parses pathologically slowly on parts of the corpus and is excluded), `polyglot-sql` fourteen, `databend-common-ast` three, `sqlglot-rust` three, `turso_parser` three. The FFI parsers (`pg_query`) are excluded because two builds of libpg_query collide at link. Without `--full` the runners use a small per-dialect sample, which is a fast pipeline check rather than publishable numbers. Validity labels for the reference dialects are produced by the `oracle` crate (real engines in Docker via testcontainers) and committed under `oracle/labels`, so `correctness` and `export` need no Docker. Regenerate them with `cargo run --release -p oracle`. diff --git a/featurescan/Cargo.toml b/featurescan/Cargo.toml index c46de64..b2d2e10 100644 --- a/featurescan/Cargo.toml +++ b/featurescan/Cargo.toml @@ -2,7 +2,7 @@ name = "featurescan" version = "0.1.0" edition = "2021" -description = "Static source-feature scan of the benchmarked SQL parsers (panic discipline, unsafe, lint policy)" +description = "Static source-feature scan and build-cost measurement of the benchmarked SQL parsers (panic discipline, unsafe, lint policy, compile cost)" license = "MIT" publish = false rust-version.workspace = true @@ -13,7 +13,10 @@ default-run = "featurescan" # `src/` with `syn`, and records panic-inducing constructs, unsafe usage, and the # crate's own lint policy. The committed result (featurescan/data/featurescan.json) # is baked into the web metadata, so the wasm build stays free of network access. -# Not a default workspace member, so `cargo build` never pulls these deps. +# The build-cost bin (`featurescan-build`) compiles a minimal probe per parser +# release in a scratch crate under `target/buildscan/` and writes +# `featurescan/data/build.json`. Not a default workspace member, so `cargo build` +# never pulls these deps. [dependencies] cargo_metadata = "0.19" @@ -40,3 +43,7 @@ path = "src/main.rs" [[bin]] name = "featurescan-depth" path = "src/depth.rs" + +[[bin]] +name = "featurescan-build" +path = "src/build.rs" diff --git a/featurescan/data/build.json b/featurescan/data/build.json new file mode 100644 index 0000000..4f6a727 --- /dev/null +++ b/featurescan/data/build.json @@ -0,0 +1,7 @@ +{ + "note": "", + "rustc": "", + "jobs": 8, + "baseline_bytes": 0, + "entries": [] +} diff --git a/featurescan/src/build.rs b/featurescan/src/build.rs new file mode 100644 index 0000000..f84960a --- /dev/null +++ b/featurescan/src/build.rs @@ -0,0 +1,912 @@ +//! Build-cost scan of the benchmarked SQL parsers. +//! +//! Compiles a minimal probe (read stdin, parse it, print the statement count) +//! from a clean target directory for every benchmarked parser release, the +//! current root-manifest releases and every time-machine milestone, and records +//! the build CPU time, wall time, compiled unit count, parser-crate-only +//! compile time, and stripped binary size over the no-parser baseline. Writes a +//! committed snapshot at `featurescan/data/build.json` that the web metadata +//! bakes in. Run with `cargo run -p featurescan --bin featurescan-build`. + +use std::cmp::Ordering; +use std::io::Write; +use std::path::{Path, PathBuf}; +use std::process::{Command, Stdio}; +use std::time::Instant; + +use cargo_metadata::semver::Version; +use cargo_metadata::{DependencyKind, Metadata, MetadataCommand}; +use viz::{BuildCost, BuildScan}; + +/// One compiler job, recorded in the snapshot because the wall times assume it. +const JOBS: usize = 8; + +/// (display name, package name) of every current parser release; `pg_query` +/// lists twice, the parse and summary probes share the crate. +const CURRENT: &[(&str, &str)] = &[ + ("sqlparser-rs", "sqlparser"), + ("pg_query.rs", "pg_query"), + ("pg_query (summary)", "pg_query"), + ("qusql-parse", "qusql-parse"), + ("polyglot-sql", "polyglot-sql"), + ("databend-common-ast", "databend-common-ast"), + ("sqlglot-rust", "sqlglot-rust"), + ("sqlite3-parser", "sqlite3-parser"), + ("turso_parser", "turso_parser"), +]; + +/// The package names that are benchmarked parsers, so the time-machine +/// dependencies that are not (serde_json, zstd, ...) are skipped. +const PARSER_PACKAGES: &[&str] = &[ + "sqlparser", + "pg_query", + "qusql-parse", + "polyglot-sql", + "databend-common-ast", + "sqlglot-rust", + "sqlite3-parser", + "turso_parser", +]; + +const NOTE: &str = "Clean `--release` builds of a minimal probe (read stdin, parse it, \ +print the statement count), one per parser release, the current root-manifest releases \ +plus every time-machine milestone. Each build starts from an empty target directory with \ +the workspace `Cargo.lock` copied in and `cargo fetch`ed first, then runs `--frozen -j 8`, \ +so the dependency versions match the benchmarked ones and no network or resolution \ +enters the timed region. `cpu_secs` is the cutime plus cstime delta of the whole build \ +tree, `wall_secs` the same region, `crate_secs` the parser lib unit duration and `units` \ +the compiled unit count from cargo's `--timings` report (build-script execution units \ +are excluded, they run a script rather than compile it), and `binary_bytes` the \ +stripped probe size over the no-parser baseline. Regenerate with `cargo run -p \ +featurescan --bin featurescan-build`."; + +/// The probe `main`, shared by every probe source. +const PROBE_MAIN: &str = r##"fn main() { + let mut buf = String::new(); + std::io::stdin().read_to_string(&mut buf).expect("read stdin"); + println!("{}", parse(&buf)); +} +"##; + +/// The probe prolog without extra dependencies. +const PROBE_PROLOG: &str = "use std::io::Read;\n"; + +/// The sqlite3 probes need the fallible-iterator trait in scope for `next()`. +const PROBE_PROLOG_FALLIBLE: &str = + "use fallible_iterator::FallibleIterator as _;\nuse std::io::Read;\n"; + +/// sqlparser probe body, the generic dialect like the time machine's adapter. +const PARSE_SQLPARSER: &str = r##"fn parse(sql: &str) -> usize { + match p::parser::Parser::parse_sql(&p::dialect::GenericDialect {}, &sql) { + Ok(v) => v.len(), + Err(e) => { + eprintln!("sqlparser: {e}"); + std::process::exit(1); + } + } +} +"##; + +/// pg_query probe body, the full parse entry point. +const PARSE_PG_QUERY: &str = r##"fn parse(sql: &str) -> usize { + match p::parse(sql, p::ParserOptions::DEFAULT) { + Ok(_) => 1, + Err(e) => { + eprintln!("pg_query: {e}"); + std::process::exit(1); + } + } +} +"##; + +/// pg_query probe body, the summary entry point. +const PARSE_PG_QUERY_SUMMARY: &str = r##"fn parse(sql: &str) -> usize { + match p::summary(sql, p::ParserOptions::DEFAULT, -1) { + Ok(_) => 1, + Err(e) => { + eprintln!("pg_query summary: {e}"); + std::process::exit(1); + } + } +} +"##; + +/// qusql-parse probe body, the resilient API counts the statements it parsed. +const PARSE_QUSQL: &str = r##"fn parse(sql: &str) -> usize { + let opts = p::ParseOptions::new() + .dialect(p::SQLDialect::PostgreSQL) + .arguments(p::SQLArguments::Dollar); + let mut issues = p::Issues::new(sql); + p::parse_statements(sql, &mut issues, &opts).len() +} +"##; + +/// polyglot-sql probe body. +const PARSE_POLYGLOT: &str = r##"fn parse(sql: &str) -> usize { + match p::parse(&sql, p::DialectType::Generic) { + Ok(v) => v.len(), + Err(e) => { + eprintln!("polyglot: {e}"); + std::process::exit(1); + } + } +} +"##; + +/// databend-common-ast probe body, the single-statement tokenizer plus parser. +const PARSE_DATABEND: &str = r##"fn parse(sql: &str) -> usize { + let tokens = match p::parser::tokenize_sql(sql) { + Ok(t) => t, + Err(e) => { + eprintln!("databend tokenize: {e}"); + std::process::exit(1); + } + }; + match p::parser::parse_sql(&tokens, p::parser::Dialect::PostgreSQL) { + Ok(_) => 1, + Err(e) => { + eprintln!("databend parse: {e}"); + std::process::exit(1); + } + } +} +"##; + +/// sqlglot-rust probe body. +const PARSE_SQLGLOT: &str = r##"fn parse(sql: &str) -> usize { + match p::parser::parse_statements(&sql, p::Dialect::Ansi) { + Ok(v) => v.len(), + Err(e) => { + eprintln!("sqlglot: {e}"); + std::process::exit(1); + } + } +} +"##; + +/// sqlite3-parser probe body for releases before 0.17.0, the plain constructor. +const PARSE_SQLITE3: &str = r##"fn parse(sql: &str) -> usize { + let input = sql.as_bytes(); + let mut parser = p::lexer::sql::Parser::new(input); + let mut count = 0; + loop { + match parser.next() { + Ok(Some(_)) => count += 1, + Ok(None) => break count, + Err(e) => { + eprintln!("sqlite3: {e}"); + std::process::exit(1); + } + } + } +} +"##; + +/// sqlite3-parser probe body for 0.17.0 and later, which take a bump arena. +const PARSE_SQLITE3_ARENA: &str = r##"fn parse(sql: &str) -> usize { + let input = sql.as_bytes(); + let arena = p::Bump::new(); + let mut parser = p::lexer::sql::Parser::new(&arena, input); + let mut count = 0; + loop { + match parser.next() { + Ok(Some(_)) => count += 1, + Ok(None) => break count, + Err(e) => { + eprintln!("sqlite3: {e}"); + std::process::exit(1); + } + } + } +} +"##; + +/// turso_parser probe body, the streaming `next_cmd` loop. +const PARSE_TURSO: &str = r##"fn parse(sql: &str) -> usize { + let mut parser = p::parser::Parser::new(sql.as_bytes()); + let mut count = 0; + loop { + match parser.next_cmd() { + Ok(Some(_)) => count += 1, + Ok(None) => break count, + Err(e) => { + eprintln!("turso: {e}"); + std::process::exit(1); + } + } + } +} +"##; + +/// The baseline probe, the same stdin read with no parser to measure against. +const BASELINE_PROBE: &str = r##"use std::io::Read; + +fn parse(sql: &str) -> usize { + sql.trim().lines().count() +} + +fn main() { + let mut buf = String::new(); + std::io::stdin().read_to_string(&mut buf).expect("read stdin"); + println!("{}", parse(&buf)); +} +"##; + +/// The baseline scratch manifest, the probe manifest without a parser dep. +const BASELINE_MANIFEST: &str = + "[package]\nname = \"buildscan-probe\"\nversion = \"0.1.0\"\nedition = \"2021\"\npublish = false\n\n[workspace]\n\n[dependencies]\n\n[profile.release]\nstrip = true\n"; + +/// Where a release comes from, enough to rebuild its scratch dependency line. +enum ReleaseSource { + Registry, + Git { url: String, rev: String }, +} + +impl ReleaseSource { + /// Parse the locked `source` string, `crates.io` or `git+URL#`. + fn from_lock_source(source: &str) -> Self { + match source.strip_prefix("git+") { + Some(rest) => { + let (url, rev) = rest + .rsplit_once('#') + .expect("a git lock source carries a commit"); + ReleaseSource::Git { + url: url.to_string(), + rev: rev.to_string(), + } + } + None => ReleaseSource::Registry, + } + } + + /// `crates.io`, or the git URL and the first 7 hex digits of the commit. + fn display(&self) -> String { + match self { + ReleaseSource::Registry => "crates.io".to_string(), + ReleaseSource::Git { url, rev } => format!("{url}#{}", &rev[..7]), + } + } +} + +/// One benchmarked parser release to measure. +struct Release { + parser: String, + package: String, + version: String, + source: ReleaseSource, + current: bool, +} + +/// The wall and CPU seconds of one clean release build. +struct BuildTimes { + wall_secs: f64, + cpu_secs: f64, +} + +/// A `--only [@]` filter. +struct Only { + package: String, + version: Option, +} + +/// Split a package id into (source, name, version); the name is absent when a +/// path package's directory and package names agree. +fn id_parts(repr: &str) -> (&str, Option<&str>, &str) { + let (source, tail) = repr + .rsplit_once('#') + .expect("a package id carries a source"); + match tail.rsplit_once('@') { + Some((name, version)) => (source, Some(name), version), + None => (source, None, tail), + } +} + +/// The locked `source` string for one (name, version); a git id matches by +/// prefix, since the lock appends the commit. +fn locked_source(doc: &toml::Value, name: &str, version: &str, id_source: &str) -> String { + let packages = doc + .get("package") + .and_then(toml::Value::as_array) + .expect("the lockfile has packages"); + for entry in packages { + let matches_name = entry.get("name").and_then(toml::Value::as_str) == Some(name); + let matches_version = entry.get("version").and_then(toml::Value::as_str) == Some(version); + if !matches_name || !matches_version { + continue; + } + let Some(source) = entry.get("source").and_then(toml::Value::as_str) else { + continue; + }; + let matches_source = if source == id_source { + true + } else if id_source.starts_with("git+") { + source.starts_with(id_source) + } else { + false + }; + if matches_source { + return source.to_string(); + } + } + panic!("{name} {version} is not in Cargo.lock"); +} + +/// The display name of a parser release, matching the web metadata names. +fn display_name(package: &str) -> String { + match package { + "sqlparser" => "sqlparser-rs".to_string(), + "pg_query" => "pg_query.rs".to_string(), + other => other.to_string(), + } +} + +/// The root package's parser dependencies, resolved to the exact locked package. +fn current_releases(metadata: &Metadata, doc: &toml::Value) -> Vec { + let resolve = metadata + .resolve + .as_ref() + .expect("cargo metadata resolves the graph"); + let root_package = metadata + .packages + .iter() + .find(|p| p.name == "sql_ast_benchmark") + .expect("the root package is in the metadata"); + let root_node = resolve + .nodes + .iter() + .find(|n| n.id.repr == root_package.id.repr) + .expect("the root node is in the graph"); + let mut out = Vec::new(); + for (parser, package) in CURRENT { + assert!( + root_package + .dependencies + .iter() + .any(|d| d.name == *package && d.kind == DependencyKind::Normal), + "the root package does not depend on {package}" + ); + let node_dep = root_node + .deps + .iter() + .find(|d| id_parts(&d.pkg.repr).1 == Some(*package)) + .unwrap_or_else(|| panic!("{package} is not resolved")); + let (id_source, _, version) = id_parts(&node_dep.pkg.repr); + let source = + ReleaseSource::from_lock_source(&locked_source(doc, package, version, id_source)); + out.push(Release { + parser: (*parser).to_string(), + package: (*package).to_string(), + version: version.to_string(), + source, + current: true, + }); + } + out +} + +/// The timemachine package's renamed dependencies (one pinned release per +/// alias), resolved to the exact locked package. +fn milestone_releases(metadata: &Metadata, doc: &toml::Value) -> Vec { + let resolve = metadata + .resolve + .as_ref() + .expect("cargo metadata resolves the graph"); + let tm_package = metadata + .packages + .iter() + .find(|p| p.name == "timemachine") + .expect("timemachine is in the metadata"); + let tm_node = resolve + .nodes + .iter() + .find(|n| n.id.repr == tm_package.id.repr) + .expect("the timemachine node is in the graph"); + let mut out = Vec::new(); + for dep in tm_package + .dependencies + .iter() + .filter(|d| d.kind == DependencyKind::Normal && d.rename.is_some()) + { + let alias = dep.rename.as_deref().expect("the dependency is renamed"); + let node_dep = tm_node + .deps + .iter() + .find(|d| d.name == alias) + .unwrap_or_else(|| panic!("{alias} is not resolved")); + let (id_source, name, version) = id_parts(&node_dep.pkg.repr); + let Some(name) = name else { + continue; + }; + if !PARSER_PACKAGES.contains(&name) { + continue; + } + let source = ReleaseSource::from_lock_source(&locked_source(doc, name, version, id_source)); + out.push(Release { + parser: display_name(name), + package: name.to_string(), + version: version.to_string(), + source, + current: false, + }); + } + out +} + +/// Current releases first, then by parser and semver. +fn release_order(a: &Release, b: &Release) -> Ordering { + (!a.current) + .cmp(&!b.current) + .then_with(|| a.parser.cmp(&b.parser)) + .then_with(|| version_key(&a.version).cmp(&version_key(&b.version))) +} + +/// A comparable (major, minor, patch) key for a release version. +fn version_key(version: &str) -> (u64, u64, u64) { + match Version::parse(version) { + Ok(v) => (v.major, v.minor, v.patch), + Err(_) => (0, 0, 0), + } +} + +/// 0.17.0 moved the sqlite3 AST into a caller-owned bump arena. +fn uses_arena(version: &str) -> bool { + match Version::parse(version) { + Ok(v) => v >= Version::new(0, 17, 0), + Err(_) => false, + } +} + +/// The probe source for one release, mirroring the time-machine parse calls. +fn probe_source(release: &Release) -> String { + let (prolog, parse_fn) = match release.package.as_str() { + "sqlparser" => (PROBE_PROLOG, PARSE_SQLPARSER), + "pg_query" if release.parser == "pg_query (summary)" => { + (PROBE_PROLOG, PARSE_PG_QUERY_SUMMARY) + } + "pg_query" => (PROBE_PROLOG, PARSE_PG_QUERY), + "qusql-parse" => (PROBE_PROLOG, PARSE_QUSQL), + "polyglot-sql" => (PROBE_PROLOG, PARSE_POLYGLOT), + "databend-common-ast" => (PROBE_PROLOG, PARSE_DATABEND), + "sqlglot-rust" => (PROBE_PROLOG, PARSE_SQLGLOT), + "sqlite3-parser" if uses_arena(&release.version) => { + (PROBE_PROLOG_FALLIBLE, PARSE_SQLITE3_ARENA) + } + "sqlite3-parser" => (PROBE_PROLOG_FALLIBLE, PARSE_SQLITE3), + "turso_parser" => (PROBE_PROLOG, PARSE_TURSO), + _ => panic!("no probe for the parser package {}", release.package), + }; + format!("{prolog}\n{parse_fn}\n{PROBE_MAIN}") +} + +/// The scratch manifest, the parser pinned to the exact locked version or commit. +fn probe_manifest(release: &Release) -> String { + let dep = match &release.source { + ReleaseSource::Registry => { + format!( + "p = {{ package = \"{}\", version = \"={}\" }}", + release.package, release.version + ) + } + ReleaseSource::Git { url, rev } => { + format!( + "p = {{ package = \"{}\", git = \"{}\", rev = \"{}\" }}", + release.package, url, rev + ) + } + }; + let fallible = if release.package == "sqlite3-parser" { + "fallible-iterator = \"0.3\"\n" + } else { + "" + }; + format!( + "[package]\nname = \"buildscan-probe\"\nversion = \"0.1.0\"\nedition = \"2021\"\npublish = false\n\n[workspace]\n\n[dependencies]\n{dep}\n{fallible}\n[profile.release]\nstrip = true\n" + ) +} + +/// The scratch directory of one release, under the workspace `target/`. +fn scratch_dir(scan_root: &Path, release: &Release) -> PathBuf { + let suffix = if matches!(release.source, ReleaseSource::Git { .. }) { + "-git" + } else { + "" + }; + scan_root.join(format!("{}-{}{suffix}", release.package, release.version)) +} + +/// Write the scratch crate (manifest, probe, workspace lock) and clear the target. +fn write_scratch(dir: &Path, manifest: &str, probe: &str, lock: &Path) { + let src = dir.join("src"); + std::fs::create_dir_all(&src).expect("create the scratch src dir"); + std::fs::write(dir.join("Cargo.toml"), manifest).expect("write the scratch manifest"); + std::fs::write(src.join("main.rs"), probe).expect("write the scratch probe"); + std::fs::copy(lock, dir.join("Cargo.lock")).expect("copy the workspace Cargo.lock"); + std::fs::remove_dir_all(dir.join("target")).ok(); +} + +/// Download every locked dependency before the timed region. +fn cargo_fetch(dir: &Path) -> Result<(), String> { + let output = Command::new("cargo") + .arg("fetch") + .current_dir(dir) + .output() + .map_err(|e| format!("cargo fetch failed to start: {e}"))?; + if !output.status.success() { + return Err(format!( + "cargo fetch exited with {:?}\n{}", + output.status, + String::from_utf8_lossy(&output.stderr) + )); + } + Ok(()) +} + +/// The `cutime` and `cstime` fields of `/proc/self/stat`, in clock ticks, read +/// after the last `)` because the command field may hold spaces. They +/// accumulate every waited-on child's CPU time. +fn self_ticks() -> (u64, u64) { + let stat = std::fs::read_to_string("/proc/self/stat").expect("read /proc/self/stat"); + let rest = &stat[stat.rfind(')').expect("stat has a command field") + 1..]; + let fields: Vec<&str> = rest.split_whitespace().collect(); + // `rest` starts at field 3 (state), so `cutime` (16) and `cstime` (17) + // are at indexes 13 and 14. + ( + fields[13].parse().expect("cutime is numeric"), + fields[14].parse().expect("cstime is numeric"), + ) +} + +/// Run one clean `cargo build --release` in `dir` and time it, the scratch +/// `target/` removed first so `--timings` writes a fresh unit report. +fn clean_build(dir: &Path, clk_tck: f64) -> Result { + let target = dir.join("target"); + std::fs::remove_dir_all(&target).ok(); + let before = self_ticks(); + let start = Instant::now(); + let output = Command::new("cargo") + .arg("build") + .arg("--release") + .arg("--frozen") + .arg("-j") + .arg(JOBS.to_string()) + .arg("--timings") + // The user-level cargo config enables incremental builds, which a clean release build must not pick up. + .arg("--config") + .arg("build.incremental=false") + .current_dir(dir) + .env("CARGO_TARGET_DIR", &target) + .env_remove("RUSTC_WRAPPER") + .env_remove("CARGO_BUILD_RUSTC_WRAPPER") + .env_remove("RUSTFLAGS") + .env_remove("CARGO_INCREMENTAL") + .output() + .map_err(|e| format!("cargo build failed to start: {e}"))?; + let wall_secs = start.elapsed().as_secs_f64(); + let after = self_ticks(); + if !output.status.success() { + return Err(format!( + "cargo build exited with {:?}\n{}", + output.status, + String::from_utf8_lossy(&output.stderr) + )); + } + // Clock ticks to seconds, a deliberately lossy float here. + let ticks = after + .0 + .saturating_add(after.1) + .saturating_sub(before.0.saturating_add(before.1)); + Ok(BuildTimes { + wall_secs, + cpu_secs: ticks as f64 / clk_tck, + }) +} + +/// Read the units the `--timings` report embeds as `const UNIT_DATA = [...]`. +fn unit_data(html: &str) -> Option> { + const MARKER: &str = "const UNIT_DATA = "; + let start = html.find(MARKER)?.checked_add(MARKER.len())?; + let end = html[start..].find(";\n")?.checked_add(start)?; + let value: serde_json::Value = serde_json::from_str(&html[start..end]).ok()?; + let mut out = Vec::new(); + for unit in value.as_array()? { + let name = unit.get("name")?.as_str()?.to_string(); + let target = unit.get("target")?.as_str()?.to_string(); + let duration = unit.get("duration")?.as_f64()?; + out.push((name, target, duration)); + } + Some(out) +} + +/// The parser lib unit duration and the compiled unit count from the report, +/// excluding build-script execution units, which run a script rather than compile. +fn timing_stats(report: &Path, package: &str) -> (Option, usize) { + let Ok(html) = std::fs::read_to_string(report) else { + return (None, 0); + }; + let Some(units) = unit_data(&html) else { + return (None, 0); + }; + let mut crate_secs = None; + let mut count = 0; + for (name, target, duration) in units { + let label = target.trim(); + if label == "build-script (run)" { + continue; + } + count += 1; + // The lib unit's target label is empty, the package name identifies it. + if name == package && label.is_empty() && crate_secs.is_none() { + crate_secs = Some(duration); + } + } + (crate_secs, count) +} + +/// Run the built probe on `SELECT 1` and require a clean exit. +fn smoke(bin: &Path) -> Result<(), String> { + let mut child = Command::new(bin) + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .spawn() + .map_err(|e| format!("the smoke run failed to start: {e}"))?; + let mut stdin = child.stdin.take().expect("the stdin pipe is open"); + stdin + .write_all(b"SELECT 1\n") + .expect("write the smoke input"); + // Closing stdin sends the probe its EOF. + drop(stdin); + let output = child + .wait_with_output() + .map_err(|e| format!("the smoke run failed to finish: {e}"))?; + if !output.status.success() { + return Err(format!( + "the smoke run exited with {:?}\n{}", + output.status, + String::from_utf8_lossy(&output.stderr) + )); + } + Ok(()) +} + +/// Build one release's scratch crate, measure it, smoke it, and clear the target. +fn measure_release( + dir: &Path, + release: &Release, + baseline_bytes: u64, + clk_tck: f64, +) -> Result { + let times = clean_build(dir, clk_tck)?; + let bin = dir.join("target/release/buildscan-probe"); + let size = std::fs::metadata(&bin) + .map_err(|e| format!("no release binary: {e}"))? + .len(); + let (crate_secs, units) = timing_stats( + &dir.join("target/cargo-timings/cargo-timing.html"), + &release.package, + ); + smoke(&bin)?; + std::fs::remove_dir_all(dir.join("target")).ok(); + Ok(BuildCost { + parser: release.parser.clone(), + package: release.package.clone(), + version: release.version.clone(), + source: release.source.display(), + current: release.current, + cpu_secs: times.cpu_secs, + wall_secs: times.wall_secs, + crate_secs, + units, + binary_bytes: size.saturating_sub(baseline_bytes), + }) +} + +/// A one-line identity for logs, checkpoint reuse, and failure reports. +fn release_key(release: &Release) -> String { + format!( + "{} {} {} {} current={}", + release.parser, + release.package, + release.version, + release.source.display(), + release.current + ) +} + +/// Whether a checkpointed entry is the same release as one being planned. +fn cost_matches(cost: &BuildCost, release: &Release) -> bool { + cost.parser == release.parser + && cost.package == release.package + && cost.version == release.version + && cost.source == release.source.display() + && cost.current == release.current +} + +/// Load the checkpoint, the `BuildCost` lines written as releases finish. +fn load_checkpoint(path: &Path) -> Vec { + let Ok(content) = std::fs::read_to_string(path) else { + return Vec::new(); + }; + content + .lines() + .filter_map(|line| serde_json::from_str(line).ok()) + .collect() +} + +/// Append one finished `BuildCost` to the checkpoint. +fn append_checkpoint(path: &Path, cost: &BuildCost) { + let Ok(mut file) = std::fs::OpenOptions::new() + .create(true) + .append(true) + .open(path) + else { + eprintln!("warning: cannot append to the checkpoint {path:?}"); + return; + }; + let line = serde_json::to_string(cost).expect("serialize the entry"); + file.write_all(format!("{line}\n").as_bytes()) + .expect("append the checkpoint line"); +} + +/// The clock tick rate of this machine, run once at startup. +fn clk_tck() -> f64 { + let output = Command::new("getconf") + .arg("CLK_TCK") + .output() + .expect("getconf CLK_TCK"); + String::from_utf8_lossy(&output.stdout) + .trim() + .parse() + .expect("CLK_TCK is numeric") +} + +/// The `rustc -V` string, so the snapshot records the toolchain. +fn rustc_version() -> String { + let output = Command::new("rustc").arg("-V").output().expect("rustc -V"); + String::from_utf8_lossy(&output.stdout).trim().to_string() +} + +fn main() { + let args = std::env::args().skip(1).collect::>(); + let mut fresh = false; + let mut only: Option = None; + let mut i = 0; + while i < args.len() { + let arg = &args[i]; + if arg == "--fresh" { + fresh = true; + } else if arg == "--only" { + i += 1; + let Some(spec) = args.get(i) else { + eprintln!("--only needs [@]"); + std::process::exit(2); + }; + let (package, version) = spec + .split_once('@') + .map(|(p, v)| (p.to_string(), Some(v.to_string()))) + .unwrap_or_else(|| (spec.clone(), None)); + only = Some(Only { package, version }); + } else { + eprintln!("unknown flag {arg}"); + std::process::exit(2); + } + i += 1; + } + + let metadata = MetadataCommand::new() + .current_dir(Path::new(env!("CARGO_MANIFEST_DIR"))) + .exec() + .expect("cargo metadata failed"); + let lock_path = metadata + .workspace_root + .join("Cargo.lock") + .as_std_path() + .to_path_buf(); + let lock_doc: toml::Value = + toml::from_str(&std::fs::read_to_string(&lock_path).expect("read Cargo.lock")) + .expect("parse Cargo.lock"); + + let mut planned = current_releases(&metadata, &lock_doc); + planned.extend(milestone_releases(&metadata, &lock_doc)); + if let Some(only) = &only { + planned.retain(|r| { + r.package == only.package && only.version.as_deref().is_none_or(|v| r.version == v) + }); + } + if planned.is_empty() { + eprintln!("no releases match the --only filter"); + std::process::exit(1); + } + planned.sort_by(release_order); + + let scan_root = metadata.target_directory.as_std_path().join("buildscan"); + std::fs::create_dir_all(&scan_root).expect("create the buildscan dir"); + let checkpoint = scan_root.join("results.jsonl"); + if fresh { + std::fs::remove_file(&checkpoint).ok(); + } + let mut stored = load_checkpoint(&checkpoint); + + let clk_tck = clk_tck(); + let rustc = rustc_version(); + eprintln!( + "toolchain {rustc}, CLK_TCK {clk_tck}, planning {} releases ({} reused)", + planned.len(), + planned + .iter() + .filter(|r| stored.iter().any(|c| cost_matches(c, r))) + .count() + ); + + // The baseline is the no-parser probe built the same way, its stripped + // size subtracted from every binary. + let baseline_dir = scan_root.join("baseline"); + write_scratch(&baseline_dir, BASELINE_MANIFEST, BASELINE_PROBE, &lock_path); + cargo_fetch(&baseline_dir).expect("the baseline cargo fetch failed"); + clean_build(&baseline_dir, clk_tck).expect("the baseline build failed"); + let baseline_bytes = std::fs::metadata(baseline_dir.join("target/release/buildscan-probe")) + .expect("the baseline binary exists") + .len(); + std::fs::remove_dir_all(baseline_dir.join("target")).ok(); + eprintln!("baseline probe is {baseline_bytes} bytes stripped"); + + let mut entries: Vec = Vec::new(); + let mut failures: Vec = Vec::new(); + for release in &planned { + if let Some(pos) = stored.iter().position(|c| cost_matches(c, release)) { + eprintln!("reuse {}", release_key(release)); + entries.push(stored.remove(pos)); + continue; + } + let dir = scratch_dir(&scan_root, release); + write_scratch( + &dir, + &probe_manifest(release), + &probe_source(release), + &lock_path, + ); + let result = + cargo_fetch(&dir).and_then(|_| measure_release(&dir, release, baseline_bytes, clk_tck)); + match result { + Ok(cost) => { + eprintln!( + "built {} (cpu {:.1}s, wall {:.1}s, binary {} bytes)", + release_key(release), + cost.cpu_secs, + cost.wall_secs, + cost.binary_bytes + ); + append_checkpoint(&checkpoint, &cost); + entries.push(cost); + } + Err(e) => failures.push(format!("{}: {e}", release_key(release))), + } + } + + if only.is_some() { + println!( + "{}", + serde_json::to_string_pretty(&entries).expect("serialize the entries") + ); + } else { + let scan = BuildScan { + note: NOTE.to_string(), + rustc, + jobs: JOBS, + baseline_bytes, + entries, + }; + let out_path = Path::new(env!("CARGO_MANIFEST_DIR")).join("data/build.json"); + let json = serde_json::to_string_pretty(&scan).expect("serialize the snapshot"); + std::fs::write(&out_path, json).expect("write the snapshot"); + eprintln!("wrote {}", out_path.display()); + } + + if !failures.is_empty() { + eprintln!("failed releases"); + for failure in &failures { + eprintln!(" {failure}"); + } + std::process::exit(1); + } +} diff --git a/membench/Cargo.toml b/membench/Cargo.toml index 34ad9fd..7846892 100644 --- a/membench/Cargo.toml +++ b/membench/Cargo.toml @@ -7,9 +7,6 @@ license = "MIT" publish = false rust-version.workspace = true -# This crate installs a counting global allocator, which requires `unsafe`, so -# unlike the main crate it does not forbid unsafe code. - [dependencies] sql_ast_benchmark = { path = ".." } diff --git a/membench/src/main.rs b/membench/src/main.rs index 4ba900d..871d558 100644 --- a/membench/src/main.rs +++ b/membench/src/main.rs @@ -1,10 +1,11 @@ //! Per-statement memory benchmark. //! -//! Installs a counting global allocator that feeds the safe counters in -//! `sql_ast_benchmark::mem`, then, for every (parser, dialect) pair, measures -//! the peak live bytes and the retained (AST) bytes for each accepted statement. -//! Results are written one value per line to `target/mem_dist/`, consumed by -//! `sqlbench export` to build the memory section of `web/assets/bench.json.zst`. +//! Installs the counting global allocator from `sql_ast_benchmark::mem`, then, +//! for every (parser, dialect) pair, measures for each accepted statement the +//! peak live bytes, the retained (AST) bytes, and the allocation pressure +//! (allocator calls and bytes requested). Results are written one value per +//! line to `target/mem_dist/`, consumed by `sqlbench export` to build the +//! memory section of `web/assets/bench.json.zst`. //! //! Measurement is single-threaded by design: the allocator counters are //! process-wide, so concurrent allocations from other threads would corrupt a @@ -13,14 +14,13 @@ //! //! A `batch` subcommand measures whole-script memory instead: per (parser, //! dialect) it concatenates the accepted set into one script, parses it holding -//! every AST live, and records peak/retained bytes normalized per statement to -//! `target/batch_mem_dist/summary.csv`. Databend has no batch entry point and -//! is skipped there. +//! every AST live, and records the same four figures normalized per statement +//! to `target/batch_mem_dist/summary.csv`. Databend has no batch entry point +//! and is skipped there. //! //! Run locally: `cargo run --release -p membench` (per-statement) //! `cargo run --release -p membench -- batch` (whole-script) -use std::alloc::{GlobalAlloc, Layout, System}; use std::fmt::Write as _; use std::fs; use std::io::Write as _; @@ -28,43 +28,12 @@ use std::path::Path; use sql_ast_benchmark::batch::{batch_eligible, evaluate_batches, reports_statement_count}; use sql_ast_benchmark::datasets::{ensure_corpus, Dialect}; +use sql_ast_benchmark::mem::CountingAlloc; use sql_ast_benchmark::stats::slug; use sql_ast_benchmark::BenchParser; -/// System allocator that records every allocation into `sql_ast_benchmark::mem`. -struct Counting; - -// SAFETY: a thin pass-through to the system allocator that only adds atomic -// bookkeeping (no allocation of its own) around each call. -unsafe impl GlobalAlloc for Counting { - unsafe fn alloc(&self, layout: Layout) -> *mut u8 { - let ptr = System.alloc(layout); - if !ptr.is_null() { - sql_ast_benchmark::mem::record_alloc(layout.size()); - } - ptr - } - - unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) { - System.dealloc(ptr, layout); - sql_ast_benchmark::mem::record_dealloc(layout.size()); - } - - unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 { - let new_ptr = System.realloc(ptr, layout, new_size); - if !new_ptr.is_null() { - if new_size >= layout.size() { - sql_ast_benchmark::mem::record_alloc(new_size - layout.size()); - } else { - sql_ast_benchmark::mem::record_dealloc(layout.size() - new_size); - } - } - new_ptr - } -} - #[global_allocator] -static GLOBAL: Counting = Counting; +static GLOBAL: CountingAlloc = CountingAlloc; const OUT_DIR: &str = "target/mem_dist"; const BATCH_OUT_DIR: &str = "target/batch_mem_dist"; @@ -154,14 +123,21 @@ fn run() { let mut peaks = Vec::with_capacity(accepted.len()); let mut retained = Vec::with_capacity(accepted.len()); + let mut allocs = Vec::with_capacity(accepted.len()); + let mut allocated = Vec::with_capacity(accepted.len()); for s in &accepted { - if let Some((pk, rt)) = parser.measure_mem(s, dialect) { - peaks.push(pk); - retained.push(rt); + if let Some(m) = parser.measure_mem(s, dialect) { + peaks.push(m.peak); + retained.push(m.retained); + allocs.push(m.allocs); + allocated.push(m.allocated); } } - write_raw(dialect.dir_name(), parser.name(), "peak", &peaks); - write_raw(dialect.dir_name(), parser.name(), "retained", &retained); + let (dir, name) = (dialect.dir_name(), parser.name()); + write_raw(dir, name, "peak", &peaks); + write_raw(dir, name, "retained", &retained); + write_raw(dir, name, "allocs", &allocs); + write_raw(dir, name, "allocated", &allocated); eprintln!( "mem {} {}: n={}", dialect.dir_name(), @@ -182,17 +158,18 @@ fn safe_count(parser: BenchParser, sql: &str, dialect: Dialect) -> usize { } /// Batch memory: per (parser, dialect) it samples the same random batches as the -/// time bench (deterministic seed), measures peak/retained over the batches that -/// reparse correctly, and records them normalized per statement to -/// `target/batch_mem_dist/summary.csv`. Only parsers whose memory is visible to -/// the Rust allocator take part (the libpg_query bindings report `None`). +/// time bench (deterministic seed), measures peak, retained, and allocation +/// pressure over the batches that reparse correctly, and records them normalized +/// per statement to `target/batch_mem_dist/summary.csv`. Only parsers whose +/// memory is visible to the Rust allocator take part (the libpg_query bindings +/// report `None`). fn run_batch() { fs::create_dir_all(BATCH_OUT_DIR).expect("create batch_mem_dist dir"); let mut summary = fs::File::create(format!("{BATCH_OUT_DIR}/summary.csv")).expect("create summary.csv"); writeln!( summary, - "dialect,parser,n_eligible,k,n_correct,accuracy_pct,peak_per_stmt,retained_per_stmt" + "dialect,parser,n_eligible,k,n_correct,accuracy_pct,peak_per_stmt,retained_per_stmt,allocs_per_stmt,allocated_per_stmt" ) .expect("write header"); @@ -226,22 +203,24 @@ fn run_batch() { let label = format!("{}/{}", dialect.dir_name(), parser.name()); let eval = evaluate_batches(&eligible, &label, |s| safe_count(parser, s, dialect)); - let (peak_per_stmt, retained_per_stmt) = if eval.n_correct == 0 { - (String::new(), String::new()) + let [peak_per_stmt, retained_per_stmt, allocs_per_stmt, allocated_per_stmt]: [String; + 4] = if eval.n_correct == 0 { + Default::default() } else { - let mut peak_sum = 0u128; - let mut ret_sum = 0u128; + let mut sums = [0usize; 4]; for s in &eval.correct_scripts { - if let Some((peak, retained)) = parser.measure_mem_batch(s, dialect) { - peak_sum += peak as u128; - ret_sum += retained as u128; + if let Some(m) = parser.measure_mem_batch(s, dialect) { + let sample = [m.peak, m.retained, m.allocs, m.allocated]; + for (sum, v) in sums.iter_mut().zip(sample) { + *sum = sum + .checked_add(v) + .expect("batch memory sum overflows usize"); + } } } + // Precision loss above 2^53 is irrelevant for a per-statement mean. let denom = (eval.n_correct * eval.effective_m) as f64; - ( - format!("{:.1}", peak_sum as f64 / denom), - format!("{:.1}", ret_sum as f64 / denom), - ) + sums.map(|sum| format!("{:.1}", sum as f64 / denom)) }; let acc = eval @@ -249,7 +228,7 @@ fn run_batch() { .map_or_else(String::new, |a| format!("{a:.3}")); writeln!( summary, - "{},{},{},{},{},{acc},{peak_per_stmt},{retained_per_stmt}", + "{},{},{},{},{},{acc},{peak_per_stmt},{retained_per_stmt},{allocs_per_stmt},{allocated_per_stmt}", dialect.dir_name(), parser.name(), eval.n_eligible, @@ -259,7 +238,7 @@ fn run_batch() { .expect("write row"); summary.flush().expect("flush summary"); eprintln!( - "batch-mem {} {}: elig={} ok={}/{} peak/stmt={peak_per_stmt} ret/stmt={retained_per_stmt}", + "batch-mem {} {}: elig={} ok={}/{} peak/stmt={peak_per_stmt} ret/stmt={retained_per_stmt} allocs/stmt={allocs_per_stmt}", dialect.dir_name(), parser.name(), eval.n_eligible, diff --git a/src/bin/sqlbench.rs b/src/bin/sqlbench.rs index b83d945..dc92b06 100644 --- a/src/bin/sqlbench.rs +++ b/src/bin/sqlbench.rs @@ -200,7 +200,7 @@ fn run_regen() { // straight to web/assets/history.json.zst. The memory passes install a global // allocator, so they are separate processes. The time-machine memory pass // runs before its timing pass, which merges the memory sidecar. - let steps: [(&str, &[&str]); 7] = [ + let steps: [(&str, &[&str]); 8] = [ // Static source-feature scan and recursion-depth probe, writing the // committed featurescan/data/*.json the web bakes in. Independent of the // benches, so run first. @@ -226,6 +226,19 @@ fn run_regen() { "featurescan-depth", ], ), // featurescan/data/depth.json + // Build cost of every parser release: one clean release build at a + // time, before the benches so no measurement shares the machine with it. + ( + "cargo", + &[ + "run", + "--release", + "-p", + "featurescan", + "--bin", + "featurescan-build", + ], + ), // featurescan/data/build.json ("cargo", &["bench"]), // target/bench_dist/ + target/batch_dist/ ("cargo", &["run", "--release", "-p", "membench"]), // target/mem_dist/ ( @@ -300,7 +313,7 @@ fn usage() -> ! { eprintln!("usage: sqlbench "); eprintln!(" correctness [--per-file] grade parsers over datasets/"); eprintln!(" export write web/assets/bench.json.zst for the site"); - eprintln!(" regen run feature scan + depth probe + benches + time machine, then export"); + eprintln!(" regen run feature scan + depth probe + build cost + benches + time machine, then export"); std::process::exit(2); } diff --git a/src/export.rs b/src/export.rs index e79fb10..38759a4 100644 --- a/src/export.rs +++ b/src/export.rs @@ -210,12 +210,14 @@ fn mem_for(dir: &str, parsers: &[BenchParser]) -> Vec { if peak.is_empty() { continue; } - let retained = bench_dist::load_mem(dir, name, "retained"); + let load = |kind| dist_from(&bench_dist::load_mem(dir, name, kind)); out.push(ParserMem { parser: name.to_string(), n: peak.len(), peak: dist_from(&peak), - retained: dist_from(&retained), + retained: load("retained"), + allocs: load("allocs"), + allocated: load("allocated"), }); } out @@ -233,13 +235,15 @@ struct BatchPerfRow { } /// One row of the batch memory summary (`batch_mem_dist/summary.csv`): -/// `dialect,parser,n_eligible,k,n_correct,accuracy_pct,peak_per_stmt,retained_per_stmt`. +/// `dialect,parser,n_eligible,k,n_correct,accuracy_pct,peak_per_stmt,retained_per_stmt,allocs_per_stmt,allocated_per_stmt`. /// The per-statement figures may be blank (no batch parsed correctly). struct BatchMemRow { dialect: String, parser: String, peak_per_stmt: Option, retained_per_stmt: Option, + allocs_per_stmt: Option, + allocated_per_stmt: Option, } /// Parse a possibly-blank CSV float field into `Option`. @@ -274,7 +278,7 @@ fn parse_batch_mem(content: &str) -> Vec { .skip(1) .filter_map(|line| { let f: Vec<&str> = line.split(',').collect(); - if f.len() < 8 { + if f.len() < 10 { return None; } Some(BatchMemRow { @@ -282,6 +286,8 @@ fn parse_batch_mem(content: &str) -> Vec { parser: f[1].to_string(), peak_per_stmt: opt_f64(f[6]), retained_per_stmt: opt_f64(f[7]), + allocs_per_stmt: opt_f64(f[8]), + allocated_per_stmt: opt_f64(f[9]), }) }) .collect() @@ -320,6 +326,8 @@ fn batch_for(dir: &str, perf: &[BatchPerfRow], mem: &[BatchMemRow]) -> Vec Vec Vec &'static str { match self { - Self::Sqlparser => "0.63.0", - Self::PgQuery | Self::PgQuerySummary => "6.2.0", - Self::Qusql | Self::Polyglot => "0.11.0", + Self::Sqlparser => "0.64.0", + Self::PgQuery | Self::PgQuerySummary => "18.1.0", + Self::Qusql => "0.12.0", + Self::Polyglot => "0.14.1", Self::Databend => "0.2.5", - Self::Sqlglot => "0.10.29", + Self::Sqlglot => "0.11.1", Self::Sqlite3 => "0.17.0", - Self::Turso => "0.7.2", + Self::Turso => "0.8.3", } } @@ -409,11 +410,13 @@ impl BenchParser { .map(|_| ()) .map_err(|e| e.to_string()) }), - Self::PgQuery if dialect == Dialect::Postgresql => { - catch_outcome(|| pg_query::parse(sql).map(|_| ()).map_err(|e| e.to_string())) - } + Self::PgQuery if dialect == Dialect::Postgresql => catch_outcome(|| { + pg_query::parse(sql, pg_query::ParserOptions::DEFAULT) + .map(|_| ()) + .map_err(|e| e.to_string()) + }), Self::PgQuerySummary if dialect == Dialect::Postgresql => catch_outcome(|| { - pg_query::summary(sql, -1) + pg_query::summary(sql, pg_query::ParserOptions::DEFAULT, -1) .map(|_| ()) .map_err(|e| e.to_string()) }), @@ -453,8 +456,10 @@ impl BenchParser { Self::Sqlparser => { SqlparserParser::parse_sql(&*sqlparser_dialect(dialect), sql).is_ok() } - Self::PgQuery => pg_query::parse(sql).is_ok(), - Self::PgQuerySummary => pg_query::summary(sql, -1).is_ok(), + Self::PgQuery => pg_query::parse(sql, pg_query::ParserOptions::DEFAULT).is_ok(), + Self::PgQuerySummary => { + pg_query::summary(sql, pg_query::ParserOptions::DEFAULT, -1).is_ok() + } Self::Qusql => { let d = qusql_dialect(dialect).unwrap_or(SQLDialect::PostgreSQL); let opts = ParseOptions::new() @@ -525,10 +530,14 @@ impl BenchParser { SqlparserParser::parse_sql(&*sqlparser_dialect(dialect), sql) .map_or(0, |v| v.len()), ), - Self::PgQuery => (dialect == Dialect::Postgresql) - .then(|| pg_query::parse(sql).map_or(0, |r| r.protobuf.stmts.len())), - Self::PgQuerySummary => (dialect == Dialect::Postgresql) - .then(|| pg_query::summary(sql, -1).map_or(0, |r| r.statement_types.len())), + Self::PgQuery => (dialect == Dialect::Postgresql).then(|| { + pg_query::parse(sql, pg_query::ParserOptions::DEFAULT) + .map_or(0, |r| r.protobuf.stmts.len()) + }), + Self::PgQuerySummary => (dialect == Dialect::Postgresql).then(|| { + pg_query::summary(sql, pg_query::ParserOptions::DEFAULT, -1) + .map_or(0, |r| r.statement_types.len()) + }), Self::Qusql => qusql_dialect(dialect).map(|d| { let opts = ParseOptions::new() .dialect(d) @@ -575,87 +584,60 @@ impl BenchParser { } } - /// Parse `sql` while the `membench` allocator is active and return - /// `(peak, retained)` bytes: the high-water mark of live allocations during - /// the parse, and the bytes still live afterwards (the produced AST plus any - /// scaffolding it keeps alive). `None` for the `libpg_query` bindings, whose - /// real work happens in C and is invisible to the Rust allocator, and for + /// Parse `sql` while the counting allocator is active and return what the + /// parse cost it (see [`mem::MemSample`]): peak and retained bytes, and the + /// allocation pressure. `None` for the `libpg_query` bindings, whose real + /// work happens in C and is invisible to the Rust allocator, and for /// dialects a parser does not model. Intended for accepted statements, called /// single-threaded from the `membench` binary. #[must_use] - #[allow(clippy::too_many_lines)] - pub fn measure_mem(self, sql: &str, dialect: Dialect) -> Option<(usize, usize)> { + pub fn measure_mem(self, sql: &str, dialect: Dialect) -> Option { use std::hint::black_box; - // (peak, retained) relative to the live total just before the parse. - let snap = |before: usize| { - ( - mem::peak().saturating_sub(before), - mem::live().saturating_sub(before), - ) - }; match self { // libpg_query allocates in C, invisible to the Rust allocator. Self::PgQuery | Self::PgQuerySummary => None, Self::Sqlparser => { - let before = mem::live(); - mem::reset_peak(); + let w = mem::Window::start(); let ast = SqlparserParser::parse_sql(&*sqlparser_dialect(dialect), sql); black_box(&ast); - let r = snap(before); - drop(ast); - Some(r) + Some(w.finish()) } Self::Qusql => { let d = qusql_dialect(dialect)?; - let before = mem::live(); - mem::reset_peak(); + let w = mem::Window::start(); let opts = ParseOptions::new() .dialect(d) .arguments(qusql_parse::SQLArguments::Dollar); let mut issues = Issues::new(sql); let ast = parse_statements(sql, &mut issues, &opts); black_box((&ast, &issues)); - let r = snap(before); - drop(ast); - drop(issues); - Some(r) + Some(w.finish()) } Self::Polyglot => { - let before = mem::live(); - mem::reset_peak(); + let w = mem::Window::start(); let ast = polyglot_parse(sql, polyglot_dialect(dialect)); black_box(&ast); - let r = snap(before); - drop(ast); - Some(r) + Some(w.finish()) } Self::Databend => { let d = databend_dialect_of(dialect)?; - let before = mem::live(); - mem::reset_peak(); + let w = mem::Window::start(); let toks = databend_tokenize(sql); let ast = toks.as_ref().ok().map(|t| databend_parse(t, d)); black_box((&toks, &ast)); - let r = snap(before); - drop(ast); - drop(toks); - Some(r) + Some(w.finish()) } Self::Sqlglot => { - let before = mem::live(); - mem::reset_peak(); + let w = mem::Window::start(); let ast = sqlglot_rust::parser::parse_statements(sql, sqlglot_dialect(dialect)); black_box(&ast); - let r = snap(before); - drop(ast); - Some(r) + Some(w.finish()) } Self::Sqlite3 => { if dialect != Dialect::Sqlite { return None; } - let before = mem::live(); - mem::reset_peak(); + let w = mem::Window::start(); let bump = sqlite3_parser::Bump::new(); let mut parser = sqlite3_parser::lexer::sql::Parser::new(&bump, sql.as_bytes()); let mut out = Vec::new(); @@ -663,41 +645,34 @@ impl BenchParser { out.push(cmd); } black_box((&parser, &out)); - let r = snap(before); - drop(out); - drop(parser); - Some(r) + Some(w.finish()) } Self::Turso => { if dialect != Dialect::Sqlite { return None; } - let before = mem::live(); - mem::reset_peak(); + let w = mem::Window::start(); let mut parser = turso_parser::parser::Parser::new(sql.as_bytes()); let mut out = Vec::new(); while let Ok(Some(cmd)) = parser.next_cmd() { out.push(cmd); } black_box((&parser, &out)); - let r = snap(before); - drop(out); - drop(parser); - Some(r) + Some(w.finish()) } } } /// Like [`Self::measure_mem`], but for a whole multi-statement script: it - /// holds every statement's AST live at once, so the `(peak, retained)` it - /// reports reflects batch parsing (a grown `Vec` of statements, all ASTs - /// retained together). `None` when the parser has no batch entry point + /// holds every statement's AST live at once, so the sample it reports + /// reflects batch parsing (a grown `Vec` of statements, all ASTs retained + /// together). `None` when the parser has no batch entry point /// ([`Self::can_batch`], so databend), when its memory is invisible to the /// Rust allocator (the `libpg_query` bindings), or when it does not model /// `dialect`. Called single-threaded from the `membench` binary. Under any - /// other binary the counters are zero and it returns `Some((0, 0))`. + /// other binary the counters are zero and it returns an all-zero sample. #[must_use] - pub fn measure_mem_batch(self, sql: &str, dialect: Dialect) -> Option<(usize, usize)> { + pub fn measure_mem_batch(self, sql: &str, dialect: Dialect) -> Option { if !self.can_batch() { return None; } @@ -720,7 +695,11 @@ impl BenchParser { .then(|| turso_reprint(sql)) .flatten(), Self::PgQuery => (dialect == Dialect::Postgresql) - .then(|| pg_query::parse(sql).ok().and_then(|p| p.deparse().ok())) + .then(|| { + pg_query::parse(sql, pg_query::ParserOptions::DEFAULT) + .ok() + .and_then(|p| p.deparse(pg_query::DeparseOptions::default()).ok()) + }) .flatten(), _ => None, } @@ -792,9 +771,9 @@ pub trait Parser: Sync { fn parse_batch(&self, sql: &str, dialect: Dialect) -> Option; /// Whether this parser has a multi-statement entry point. fn can_batch(&self) -> bool; - /// `(peak, retained)` bytes under the membench allocator. See + /// Allocator cost of one parse under the counting allocator. See /// [`BenchParser::measure_mem`]. - fn measure_mem(&self, sql: &str, dialect: Dialect) -> Option<(usize, usize)>; + fn measure_mem(&self, sql: &str, dialect: Dialect) -> Option; /// Parse and pretty-print, if the parser has a printer for `dialect`. fn reprint(&self, sql: &str, dialect: Dialect) -> Option; /// Whether this parser can pretty-print in `dialect`. @@ -818,8 +797,8 @@ pub trait Parser: Sync { } } - /// Whole-script `(peak, retained)`, gated on a batch entry point. - fn measure_mem_batch(&self, sql: &str, dialect: Dialect) -> Option<(usize, usize)> { + /// Whole-script allocator cost, gated on a batch entry point. + fn measure_mem_batch(&self, sql: &str, dialect: Dialect) -> Option { if self.can_batch() { self.measure_mem(sql, dialect) } else { @@ -870,7 +849,7 @@ impl Parser for BenchParser { fn can_batch(&self) -> bool { (*self).can_batch() } - fn measure_mem(&self, sql: &str, dialect: Dialect) -> Option<(usize, usize)> { + fn measure_mem(&self, sql: &str, dialect: Dialect) -> Option { (*self).measure_mem(sql, dialect) } fn reprint(&self, sql: &str, dialect: Dialect) -> Option { @@ -885,7 +864,7 @@ impl Parser for BenchParser { fn parse_outcome(&self, sql: &str, dialect: Dialect) -> ParseOutcome { (*self).parse_outcome(sql, dialect) } - fn measure_mem_batch(&self, sql: &str, dialect: Dialect) -> Option<(usize, usize)> { + fn measure_mem_batch(&self, sql: &str, dialect: Dialect) -> Option { (*self).measure_mem_batch(sql, dialect) } fn roundtrips(&self, sql: &str, dialect: Dialect) -> Option { diff --git a/src/mem.rs b/src/mem.rs index daf01a0..55dda3d 100644 --- a/src/mem.rs +++ b/src/mem.rs @@ -1,26 +1,83 @@ //! Process-wide allocation counters for the memory benchmark. //! -//! The counters here are plain, safe atomics. The actual `GlobalAlloc` (which -//! needs `unsafe`, forbidden in this crate) lives in the separate `membench` -//! crate and feeds these via [`record_alloc`] / [`record_dealloc`]. Under any -//! other binary the counters simply stay at zero, so [`crate::BenchParser::measure_mem`] -//! reports nothing. +//! [`CountingAlloc`] wraps the system allocator and feeds the counters. The +//! `membench` and `timemachine-mem` binaries install it as their global +//! allocator. Under any other binary the counters stay at zero, so every +//! [`Window`] reports zeros. +use std::alloc::{GlobalAlloc, Layout, System}; use std::sync::atomic::{AtomicUsize, Ordering}; +/// System allocator that records every call into the counters of this module. +/// Install it with `#[global_allocator]` in a binary that measures memory. +pub struct CountingAlloc; + +// SAFETY: every method forwards to `System` with the caller's arguments +// unchanged, so `System`'s guarantees on returned blocks carry over, and the +// added bookkeeping only touches atomics and never allocates or reenters. +unsafe impl GlobalAlloc for CountingAlloc { + unsafe fn alloc(&self, layout: Layout) -> *mut u8 { + // SAFETY: the caller guarantees `layout` has a nonzero size, as + // `GlobalAlloc::alloc` requires. + let ptr = unsafe { System.alloc(layout) }; + if !ptr.is_null() { + record_alloc(layout.size()); + } + ptr + } + + unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) { + // SAFETY: the caller guarantees `ptr` came from this allocator with + // `layout`, and this allocator hands out only `System` blocks. + unsafe { System.dealloc(ptr, layout) }; + record_dealloc(layout.size()); + } + + unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 { + // SAFETY: the caller guarantees `ptr` came from this allocator with + // `layout` (so from `System`), and that `new_size` is nonzero and does + // not overflow `isize` once rounded to `layout.align()`. + let new_ptr = unsafe { System.realloc(ptr, layout, new_size) }; + if !new_ptr.is_null() { + record_realloc(layout.size(), new_size); + } + new_ptr + } +} + /// Live bytes: total requested minus total released. static LIVE: AtomicUsize = AtomicUsize::new(0); /// High-water mark of [`LIVE`] since the last [`reset_peak`]. static PEAK: AtomicUsize = AtomicUsize::new(0); +/// Allocator calls that hand out memory (allocations and reallocations). +static ALLOCS: AtomicUsize = AtomicUsize::new(0); +/// Bytes requested by those calls, each reallocation counting its new size. +static ALLOCATED: AtomicUsize = AtomicUsize::new(0); -/// Record `size` bytes just allocated. Called from the `membench` allocator. -pub fn record_alloc(size: usize) { +/// Record `size` bytes just allocated. +fn record_alloc(size: usize) { + ALLOCS.fetch_add(1, Ordering::Relaxed); + ALLOCATED.fetch_add(size, Ordering::Relaxed); let now = LIVE.fetch_add(size, Ordering::Relaxed) + size; PEAK.fetch_max(now, Ordering::Relaxed); } -/// Record `size` bytes just freed. Called from the `membench` allocator. -pub fn record_dealloc(size: usize) { +/// Record a block resized from `old` to `new` bytes. Counted as one allocator +/// call requesting `new` bytes, the way a heap profiler sees a reallocation. +fn record_realloc(old: usize, new: usize) { + ALLOCS.fetch_add(1, Ordering::Relaxed); + ALLOCATED.fetch_add(new, Ordering::Relaxed); + if new >= old { + let grown = new - old; + let now = LIVE.fetch_add(grown, Ordering::Relaxed) + grown; + PEAK.fetch_max(now, Ordering::Relaxed); + } else { + LIVE.fetch_sub(old - new, Ordering::Relaxed); + } +} + +/// Record `size` bytes just freed. +fn record_dealloc(size: usize) { LIVE.fetch_sub(size, Ordering::Relaxed); } @@ -41,3 +98,81 @@ pub fn peak() -> usize { pub fn reset_peak() { PEAK.store(LIVE.load(Ordering::Relaxed), Ordering::Relaxed); } + +/// What one measured parse cost the allocator. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)] +pub struct MemSample { + /// High-water mark of live bytes above the window's starting total. + pub peak: usize, + /// Bytes still live when the window closed. + pub retained: usize, + /// Allocator calls that handed out memory. + pub allocs: usize, + /// Bytes those calls requested, freed or not. + pub allocated: usize, +} + +/// A measurement window over the process-wide counters. Open it right before +/// the parse and [`finish`](Self::finish) it while the result is still alive, +/// so the retained bytes include the produced AST. +pub struct Window { + live: usize, + allocs: usize, + allocated: usize, +} + +impl Window { + /// Snapshot the counters and reset the peak to the current live total. + #[must_use] + pub fn start() -> Self { + let live = live(); + reset_peak(); + Self { + live, + allocs: ALLOCS.load(Ordering::Relaxed), + allocated: ALLOCATED.load(Ordering::Relaxed), + } + } + + /// Everything allocated since [`start`](Self::start). + #[must_use] + pub fn finish(&self) -> MemSample { + MemSample { + peak: peak().saturating_sub(self.live), + retained: live().saturating_sub(self.live), + allocs: ALLOCS.load(Ordering::Relaxed).saturating_sub(self.allocs), + allocated: ALLOCATED + .load(Ordering::Relaxed) + .saturating_sub(self.allocated), + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + // The counters are process-wide and the test harness is multi-threaded, so + // one test drives every transition to keep other tests' allocations out + // (nothing in this test binary installs the counting allocator). + #[test] + fn window_tracks_peak_retained_and_pressure() { + let w = Window::start(); + record_alloc(100); + record_alloc(50); + record_dealloc(100); + record_realloc(50, 200); + record_realloc(200, 20); + let s = w.finish(); + assert_eq!( + s, + MemSample { + peak: 200, + retained: 20, + allocs: 4, + allocated: 100 + 50 + 200 + 20, + } + ); + record_dealloc(20); + } +} diff --git a/src/report.rs b/src/report.rs index 0065d67..8f71529 100644 --- a/src/report.rs +++ b/src/report.rs @@ -493,7 +493,7 @@ mod tests { fn can_batch(&self) -> bool { false } - fn measure_mem(&self, _sql: &str, _d: Dialect) -> Option<(usize, usize)> { + fn measure_mem(&self, _sql: &str, _d: Dialect) -> Option { None } fn reprint(&self, _sql: &str, _d: Dialect) -> Option { diff --git a/timemachine/Cargo.toml b/timemachine/Cargo.toml index 9e6010d..f2a6ffc 100644 --- a/timemachine/Cargo.toml +++ b/timemachine/Cargo.toml @@ -9,8 +9,7 @@ rust-version.workspace = true # Hosts several semver-incompatible versions of each pure-Rust parser at once via # `package`-rename aliases, so one binary can benchmark a library across releases. -# The memory runner installs a counting global allocator, so unsafe is allowed -# here (unlike the main crate). +# The memory runner installs the main crate's counting global allocator. [dependencies] sql_ast_benchmark = { path = ".." } @@ -81,10 +80,12 @@ sqlparser_v0_60 = { package = "sqlparser", version = "=0.60.0" } sqlparser_v0_61 = { package = "sqlparser", version = "=0.61.0" } sqlparser_v0_62 = { package = "sqlparser", version = "=0.62.0" } sqlparser_v0_63 = { package = "sqlparser", version = "=0.63.0" } +sqlparser_v0_64 = { package = "sqlparser", version = "=0.64.0" } -# sqlglot-rust minors (0.9 and 0.10 are the two semver-incompatible groups). +# sqlglot-rust minors (0.9, 0.10 and 0.11 are the semver-incompatible groups). sqlglot_v0_9 = { package = "sqlglot-rust", version = "=0.9.37" } -sqlglot_v0_10 = { package = "sqlglot-rust", version = "=0.10.29" } +sqlglot_v0_10 = { package = "sqlglot-rust", version = "=0.10.33" } +sqlglot_v0_11 = { package = "sqlglot-rust", version = "=0.11.1" } # polyglot-sql minors (latest patch of each). polyglot_v0_1 = { package = "polyglot-sql", version = "=0.1.15" } @@ -98,6 +99,9 @@ polyglot_v0_8 = { package = "polyglot-sql", version = "=0.8.1" } polyglot_v0_9 = { package = "polyglot-sql", version = "=0.9.2" } polyglot_v0_10 = { package = "polyglot-sql", version = "=0.10.0" } polyglot_v0_11 = { package = "polyglot-sql", version = "=0.11.0" } +polyglot_v0_12 = { package = "polyglot-sql", version = "=0.12.1" } +polyglot_v0_13 = { package = "polyglot-sql", version = "=0.13.5" } +polyglot_v0_14 = { package = "polyglot-sql", version = "=0.14.1" } # databend-common-ast minors (single-statement parser). databend_v0_0 = { package = "databend-common-ast", version = "=0.0.3" } @@ -128,10 +132,12 @@ qusql_v0_8 = { package = "qusql-parse", version = "=0.8.0" } qusql_v0_9 = { package = "qusql-parse", version = "=0.9.0" } qusql_v0_10 = { package = "qusql-parse", version = "=0.10.0" } qusql_v0_11 = { package = "qusql-parse", version = "=0.11.0" } +qusql_v0_12 = { package = "qusql-parse", version = "=0.12.0" } -# turso_parser has two stable releases (0.6, 0.7). +# turso_parser stable minors (0.6, 0.7, 0.8). turso_v0_6 = { package = "turso_parser", version = "=0.6.1" } turso_v0_7 = { package = "turso_parser", version = "=0.7.2" } +turso_v0_8 = { package = "turso_parser", version = "=0.8.3" } [[bin]] name = "timemachine" diff --git a/timemachine/src/bin/timemachine_mem.rs b/timemachine/src/bin/timemachine_mem.rs index 92c0bf5..71d29d6 100644 --- a/timemachine/src/bin/timemachine_mem.rs +++ b/timemachine/src/bin/timemachine_mem.rs @@ -1,47 +1,16 @@ //! Time-machine memory runner. //! -//! Installs a counting global allocator (like `membench`) and measures peak and -//! retained bytes per accepted statement for every registered version, writing a -//! per-family memory sidecar under `target/timemachine/` that the timing runner -//! merges into the final history. Single-threaded by design (the counters are -//! process-wide). - -use std::alloc::{GlobalAlloc, Layout, System}; +//! Installs the counting global allocator (like `membench`) and measures peak, +//! retained, and allocation pressure per accepted statement for every registered +//! version, writing a per-family memory sidecar under `target/timemachine/` that +//! the timing runner merges into the final history. Single-threaded by design +//! (the counters are process-wide). +use sql_ast_benchmark::mem::CountingAlloc; use sql_ast_benchmark::report::WORKER_STACK; -/// System allocator that records each allocation into `sql_ast_benchmark::mem`. -struct Counting; - -// SAFETY: a thin pass-through to the system allocator that only adds atomic -// bookkeeping (no allocation of its own) around each call. -unsafe impl GlobalAlloc for Counting { - unsafe fn alloc(&self, layout: Layout) -> *mut u8 { - let ptr = System.alloc(layout); - if !ptr.is_null() { - sql_ast_benchmark::mem::record_alloc(layout.size()); - } - ptr - } - unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) { - System.dealloc(ptr, layout); - sql_ast_benchmark::mem::record_dealloc(layout.size()); - } - unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 { - let new_ptr = System.realloc(ptr, layout, new_size); - if !new_ptr.is_null() { - if new_size >= layout.size() { - sql_ast_benchmark::mem::record_alloc(new_size - layout.size()); - } else { - sql_ast_benchmark::mem::record_dealloc(layout.size() - new_size); - } - } - new_ptr - } -} - #[global_allocator] -static GLOBAL: Counting = Counting; +static GLOBAL: CountingAlloc = CountingAlloc; fn main() { std::panic::set_hook(Box::new(|_| {})); diff --git a/timemachine/src/families/databend.rs b/timemachine/src/families/databend.rs index 5087c63..5236191 100644 --- a/timemachine/src/families/databend.rs +++ b/timemachine/src/families/databend.rs @@ -81,22 +81,19 @@ macro_rules! databend_version { false } - fn measure_mem(&self, sql: &str, dialect: Dialect) -> Option<(usize, usize)> { + fn measure_mem( + &self, + sql: &str, + dialect: Dialect, + ) -> Option { use sql_ast_benchmark::mem; use std::hint::black_box; let d = Self::dialect(dialect)?; - let before = mem::live(); - mem::reset_peak(); + let w = mem::Window::start(); let toks = $cr::parser::tokenize_sql(sql); let ast = toks.as_ref().ok().map(|t| $cr::parser::parse_sql(t, d)); black_box((&toks, &ast)); - let r = ( - mem::peak().saturating_sub(before), - mem::live().saturating_sub(before), - ); - drop(ast); - drop(toks); - Some(r) + Some(w.finish()) } fn reprint(&self, sql: &str, dialect: Dialect) -> Option { diff --git a/timemachine/src/families/polyglot.rs b/timemachine/src/families/polyglot.rs index 0fae931..9dfc0d1 100644 --- a/timemachine/src/families/polyglot.rs +++ b/timemachine/src/families/polyglot.rs @@ -81,19 +81,17 @@ macro_rules! polyglot_version { true } - fn measure_mem(&self, sql: &str, dialect: Dialect) -> Option<(usize, usize)> { + fn measure_mem( + &self, + sql: &str, + dialect: Dialect, + ) -> Option { use sql_ast_benchmark::mem; use std::hint::black_box; - let before = mem::live(); - mem::reset_peak(); + let w = mem::Window::start(); let ast = $cr::parse(sql, Self::dialect(dialect)); black_box(&ast); - let r = ( - mem::peak().saturating_sub(before), - mem::live().saturating_sub(before), - ); - drop(ast); - Some(r) + Some(w.finish()) } fn reprint(&self, sql: &str, dialect: Dialect) -> Option { @@ -125,3 +123,6 @@ polyglot_version!(PolyglotV0_8, polyglot_v0_8, "0.8.1", "2026-08-06"); polyglot_version!(PolyglotV0_9, polyglot_v0_9, "0.9.2", "2026-08-18"); polyglot_version!(PolyglotV0_10, polyglot_v0_10, "0.10.0", "2026-09-14"); polyglot_version!(PolyglotV0_11, polyglot_v0_11, "0.11.0", "2026-09-16"); +polyglot_version!(PolyglotV0_12, polyglot_v0_12, "0.12.1", "2026-09-21"); +polyglot_version!(PolyglotV0_13, polyglot_v0_13, "0.13.5", "2026-10-31"); +polyglot_version!(PolyglotV0_14, polyglot_v0_14, "0.14.1", "2026-11-08"); diff --git a/timemachine/src/families/qusql.rs b/timemachine/src/families/qusql.rs index 7b6d1e1..f9404ce 100644 --- a/timemachine/src/families/qusql.rs +++ b/timemachine/src/families/qusql.rs @@ -99,23 +99,20 @@ macro_rules! qusql_version { true } - fn measure_mem(&self, sql: &str, dialect: Dialect) -> Option<(usize, usize)> { + fn measure_mem( + &self, + sql: &str, + dialect: Dialect, + ) -> Option { use sql_ast_benchmark::mem; use std::hint::black_box; let d = Self::dialect(dialect)?; - let before = mem::live(); - mem::reset_peak(); + let w = mem::Window::start(); let opts = Self::options(d); let mut issues = $cr::Issues::new(sql); let ast = $cr::parse_statements(sql, &mut issues, &opts); black_box((&ast, &issues)); - let r = ( - mem::peak().saturating_sub(before), - mem::live().saturating_sub(before), - ); - drop(ast); - drop(issues); - Some(r) + Some(w.finish()) } fn reprint(&self, _sql: &str, _dialect: Dialect) -> Option { @@ -140,3 +137,4 @@ qusql_version!(QusqlV0_8, qusql_v0_8, "0.8.0", "2026-05-03"); qusql_version!(QusqlV0_9, qusql_v0_9, "0.9.0", "2026-07-06"); qusql_version!(QusqlV0_10, qusql_v0_10, "0.10.0", "2026-07-07"); qusql_version!(QusqlV0_11, qusql_v0_11, "0.11.0", "2026-07-19"); +qusql_version!(QusqlV0_12, qusql_v0_12, "0.12.0", "2026-10-16"); diff --git a/timemachine/src/families/sqlglot.rs b/timemachine/src/families/sqlglot.rs index 641a44c..53f87f1 100644 --- a/timemachine/src/families/sqlglot.rs +++ b/timemachine/src/families/sqlglot.rs @@ -84,19 +84,17 @@ macro_rules! sqlglot_version { true } - fn measure_mem(&self, sql: &str, dialect: Dialect) -> Option<(usize, usize)> { + fn measure_mem( + &self, + sql: &str, + dialect: Dialect, + ) -> Option { use sql_ast_benchmark::mem; use std::hint::black_box; - let before = mem::live(); - mem::reset_peak(); + let w = mem::Window::start(); let ast = $cr::parser::parse_statements(sql, Self::dialect(dialect)); black_box(&ast); - let r = ( - mem::peak().saturating_sub(before), - mem::live().saturating_sub(before), - ); - drop(ast); - Some(r) + Some(w.finish()) } fn reprint(&self, sql: &str, dialect: Dialect) -> Option { @@ -125,4 +123,5 @@ macro_rules! sqlglot_version { } sqlglot_version!(SqlglotV0_9, sqlglot_v0_9, "0.9.37", "2026-05-28"); -sqlglot_version!(SqlglotV0_10, sqlglot_v0_10, "0.10.29", "2026-08-08"); +sqlglot_version!(SqlglotV0_10, sqlglot_v0_10, "0.10.33", "2026-10-17"); +sqlglot_version!(SqlglotV0_11, sqlglot_v0_11, "0.11.1", "2026-10-28"); diff --git a/timemachine/src/families/sqlite3.rs b/timemachine/src/families/sqlite3.rs index 4d9f1cb..e2d618e 100644 --- a/timemachine/src/families/sqlite3.rs +++ b/timemachine/src/families/sqlite3.rs @@ -114,14 +114,17 @@ macro_rules! sqlite3_version { true } - fn measure_mem(&self, sql: &str, dialect: Dialect) -> Option<(usize, usize)> { + fn measure_mem( + &self, + sql: &str, + dialect: Dialect, + ) -> Option { use sql_ast_benchmark::mem; use std::hint::black_box; if dialect != Dialect::Sqlite { return None; } - let before = mem::live(); - mem::reset_peak(); + let w = mem::Window::start(); let arena = <$arena>::default(); let mut parser = Self::parser(&arena, sql.as_bytes()); let mut out = Vec::new(); @@ -129,13 +132,7 @@ macro_rules! sqlite3_version { out.push(cmd); } black_box((&parser, &out)); - let r = ( - mem::peak().saturating_sub(before), - mem::live().saturating_sub(before), - ); - drop(out); - drop(parser); - Some(r) + Some(w.finish()) } fn reprint(&self, sql: &str, dialect: Dialect) -> Option { diff --git a/timemachine/src/families/sqlparser.rs b/timemachine/src/families/sqlparser.rs index 3cd3fa5..1c15760 100644 --- a/timemachine/src/families/sqlparser.rs +++ b/timemachine/src/families/sqlparser.rs @@ -94,19 +94,13 @@ macro_rules! sqlparser_version { true } - fn measure_mem(&self, sql: &str, dialect: Dialect) -> Option<(usize, usize)> { + fn measure_mem(&self, sql: &str, dialect: Dialect) -> Option { use sql_ast_benchmark::mem; use std::hint::black_box; - let before = mem::live(); - mem::reset_peak(); + let w = mem::Window::start(); let ast = $cr::parser::Parser::parse_sql(&*Self::dialect(dialect), sql); black_box(&ast); - let r = ( - mem::peak().saturating_sub(before), - mem::live().saturating_sub(before), - ); - drop(ast); - Some(r) + Some(w.finish()) } fn reprint(&self, sql: &str, dialect: Dialect) -> Option { @@ -195,6 +189,7 @@ sqlparser_version!(SqlparserV0_60, sqlparser_v0_60, "0.60.0", "2025-12-07"); sqlparser_version!(SqlparserV0_61, sqlparser_v0_61, "0.61.0", "2026-02-10"); sqlparser_version!(SqlparserV0_62, sqlparser_v0_62, "0.62.0", "2026-05-07"); sqlparser_version!(SqlparserV0_63, sqlparser_v0_63, "0.63.0", "2026-09-13"); +sqlparser_version!(SqlparserV0_64, sqlparser_v0_64, "0.64.0", "2026-11-09"); #[cfg(test)] mod tests { diff --git a/timemachine/src/families/turso.rs b/timemachine/src/families/turso.rs index 2211db3..8bbd788 100644 --- a/timemachine/src/families/turso.rs +++ b/timemachine/src/families/turso.rs @@ -90,27 +90,24 @@ macro_rules! turso_version { true } - fn measure_mem(&self, sql: &str, dialect: Dialect) -> Option<(usize, usize)> { + fn measure_mem( + &self, + sql: &str, + dialect: Dialect, + ) -> Option { use sql_ast_benchmark::mem; use std::hint::black_box; if dialect != Dialect::Sqlite { return None; } - let before = mem::live(); - mem::reset_peak(); + let w = mem::Window::start(); let mut parser = $cr::parser::Parser::new(sql.as_bytes()); let mut out = Vec::new(); while let Ok(Some(cmd)) = parser.next_cmd() { out.push(cmd); } black_box((&parser, &out)); - let r = ( - mem::peak().saturating_sub(before), - mem::live().saturating_sub(before), - ); - drop(out); - drop(parser); - Some(r) + Some(w.finish()) } fn reprint(&self, sql: &str, dialect: Dialect) -> Option { @@ -145,3 +142,4 @@ macro_rules! turso_version { turso_version!(TursoV0_6, turso_v0_6, "0.6.1", "2026-05-22"); turso_version!(TursoV0_7, turso_v0_7, "0.7.2", "2026-07-30"); +turso_version!(TursoV0_8, turso_v0_8, "0.8.3", "2026-10-12"); diff --git a/timemachine/src/registry.rs b/timemachine/src/registry.rs index 8fc9ea4..e39de45 100644 --- a/timemachine/src/registry.rs +++ b/timemachine/src/registry.rs @@ -84,8 +84,10 @@ pub fn all() -> Vec> { Box::new(sqlparser::SqlparserV0_61), Box::new(sqlparser::SqlparserV0_62), Box::new(sqlparser::SqlparserV0_63), + Box::new(sqlparser::SqlparserV0_64), Box::new(sqlglot::SqlglotV0_9), Box::new(sqlglot::SqlglotV0_10), + Box::new(sqlglot::SqlglotV0_11), Box::new(polyglot::PolyglotV0_1), Box::new(polyglot::PolyglotV0_2), Box::new(polyglot::PolyglotV0_3), @@ -97,6 +99,9 @@ pub fn all() -> Vec> { Box::new(polyglot::PolyglotV0_9), Box::new(polyglot::PolyglotV0_10), Box::new(polyglot::PolyglotV0_11), + Box::new(polyglot::PolyglotV0_12), + Box::new(polyglot::PolyglotV0_13), + Box::new(polyglot::PolyglotV0_14), Box::new(databend::DatabendV0_0), Box::new(databend::DatabendV0_1), Box::new(databend::DatabendV0_2), @@ -119,8 +124,10 @@ pub fn all() -> Vec> { Box::new(qusql::QusqlV0_9), Box::new(qusql::QusqlV0_10), Box::new(qusql::QusqlV0_11), + Box::new(qusql::QusqlV0_12), Box::new(turso::TursoV0_6), Box::new(turso::TursoV0_7), + Box::new(turso::TursoV0_8), ] } diff --git a/timemachine/src/run.rs b/timemachine/src/run.rs index 6b9b24a..efd0e77 100644 --- a/timemachine/src/run.rs +++ b/timemachine/src/run.rs @@ -294,6 +294,8 @@ fn timing_dialect_run(p: &dyn Parser, d: Dialect, stmts: &[String]) -> (DialectR ns_per_stmt, peak_per_stmt: None, retained_per_stmt: None, + allocs_per_stmt: None, + allocated_per_stmt: None, }) }; @@ -315,7 +317,8 @@ fn timing_dialect_run(p: &dyn Parser, d: Dialect, stmts: &[String]) -> (DialectR ) } -/// Build the memory part of one version's run for one dialect (peak + retained). +/// Build the memory part of one version's run for one dialect (peak, retained, +/// and allocation pressure). fn mem_dialect_run(p: &dyn Parser, d: Dialect, stmts: &[String]) -> Option { let accepted: Vec<&str> = stmts .iter() @@ -327,21 +330,29 @@ fn mem_dialect_run(p: &dyn Parser, d: Dialect, stmts: &[String]) -> Option; 4] = std::array::from_fn(|_| Vec::with_capacity(accepted.len())); for s in &accepted { - if let Some((pk, rt)) = p.measure_mem(s, d) { - peak.push(pk as f64); - retained.push(rt as f64); + if let Some(m) = p.measure_mem(s, d) { + for (xs, v) in series + .iter_mut() + .zip([m.peak, m.retained, m.allocs, m.allocated]) + { + // Precision loss above 2^53 bytes or calls is irrelevant here. + xs.push(v as f64); + } } } - peak.sort_by(|a, b| a.partial_cmp(b).unwrap()); - retained.sort_by(|a, b| a.partial_cmp(b).unwrap()); + for xs in &mut series { + xs.sort_by(f64::total_cmp); + } + let [peak, retained, allocs, allocated] = series; Some(ParserMem { parser: p.id().family.to_string(), n: peak.len(), peak: stats::dist_from(&peak), retained: stats::dist_from(&retained), + allocs: stats::dist_from(&allocs), + allocated: stats::dist_from(&allocated), }) } diff --git a/viz/src/lib.rs b/viz/src/lib.rs index 2bdb0cd..b8be5d7 100644 --- a/viz/src/lib.rs +++ b/viz/src/lib.rs @@ -20,7 +20,8 @@ pub use chart::{ pub use color::{parser_hex, parser_rgb}; pub use marker::{marker_for, Marker}; pub use schema::{ - Bundle, CoverageFile, CoverageMatrix, DepthReport, DepthScan, DialectData, DialectDelta, - DialectRun, FamilyHistory, FeatureCounts, FeatureScan, LintPolicy, MemDist, ParserBatch, - ParserFailures, ParserFeatures, ParserMem, ParserMetrics, ParserPerf, RuleMeta, VersionRun, + BuildCost, BuildScan, Bundle, CoverageFile, CoverageMatrix, DepthReport, DepthScan, + DialectData, DialectDelta, DialectRun, FamilyHistory, FeatureCounts, FeatureScan, LintPolicy, + MemDist, ParserBatch, ParserFailures, ParserFeatures, ParserMem, ParserMetrics, ParserPerf, + RuleMeta, VersionRun, }; diff --git a/viz/src/schema.rs b/viz/src/schema.rs index 1fef3ef..5c4b8f8 100644 --- a/viz/src/schema.rs +++ b/viz/src/schema.rs @@ -118,6 +118,49 @@ pub struct DepthReport { pub ceil: usize, } +/// Build cost of every benchmarked parser release. Produced by +/// `featurescan-build`, which compiles a minimal program per release (read +/// stdin, parse it, print the statement count) from a clean target directory. +#[derive(Serialize, Deserialize, Clone, Debug)] +pub struct BuildScan { + pub note: String, + /// `rustc -V` of the toolchain that built every entry. + pub rustc: String, + /// The `-j` every build ran with, which the wall times depend on. + pub jobs: usize, + /// Stripped size of the same program without any parser, subtracted from + /// every [`BuildCost::binary_bytes`]. + pub baseline_bytes: u64, + pub entries: Vec, +} + +/// Build cost of one parser release. +#[derive(Serialize, Deserialize, Clone, Debug)] +pub struct BuildCost { + /// Display name, matching the parser-page and time-machine family name. + pub parser: String, + pub package: String, + pub version: String, + /// `crates.io`, or the git URL and short commit for a git dependency. + pub source: String, + /// The release the main snapshot benchmarks, as opposed to a time-machine + /// milestone. One parser can have both for the same version number. + pub current: bool, + /// CPU seconds (user plus system) of the clean release build, every + /// dependency and build script included. + pub cpu_secs: f64, + /// Wall-clock seconds of the same build at [`BuildScan::jobs`]. + pub wall_secs: f64, + /// Seconds cargo spent compiling the parser crate itself, without its + /// dependencies. `None` when the timing report could not be read. + pub crate_secs: Option, + /// Crates compiled by the build, the parser and its dependency tree. + pub units: usize, + /// Stripped release binary size minus [`BuildScan::baseline_bytes`]: what + /// the parser adds to a program after dead-code elimination. + pub binary_bytes: u64, +} + /// Top-level results bundle (one committed `bench.json.zst`). #[derive(Serialize, Deserialize, Clone, Debug)] pub struct Bundle { @@ -210,6 +253,13 @@ pub struct ParserBatch { /// Retained bytes per statement over the correctly parsed batches. #[serde(default)] pub retained_per_stmt: Option, + /// Allocator calls per statement over the correctly parsed batches. + #[serde(default)] + pub allocs_per_stmt: Option, + /// Bytes requested from the allocator per statement over the correctly + /// parsed batches. + #[serde(default)] + pub allocated_per_stmt: Option, } /// Per-statement memory distribution for one parser in one dialect. Bytes, @@ -223,6 +273,14 @@ pub struct ParserMem { pub peak: MemDist, /// Bytes still live after the parse: the AST plus the scaffolding it retains. pub retained: MemDist, + /// Allocator calls during the parse (allocations and reallocations), a + /// count rather than bytes, in the same percentile shape. + pub allocs: MemDist, + /// Total bytes requested from the allocator during the parse, freed or not + /// (a reallocation counts its new size). Together with [`Self::allocs`] this + /// is the allocation pressure, which [`Self::peak`] hides when memory is + /// freed and reused. + pub allocated: MemDist, } /// A byte distribution: the same percentile set as [`ParserPerf`], in bytes, diff --git a/web/src/components.rs b/web/src/components.rs index 246219e..6dac475 100644 --- a/web/src/components.rs +++ b/web/src/components.rs @@ -8,9 +8,9 @@ use dioxus_free_icons::icons::fa_brands_icons::{FaGit, FaGithub, FaRust}; use dioxus_free_icons::icons::fa_solid_icons::{ FaArrowLeftLong, FaArrowsRotate, FaBan, FaBomb, FaBox, FaBug, FaBuilding, FaBullseye, FaCalendarDays, FaChartColumn, FaChartLine, FaCircleXmark, FaCode, FaCodeCommit, FaCodeFork, - FaCopy, FaCube, FaDatabase, FaDna, FaDownload, FaFileShield, FaFlaskVial, FaGaugeHigh, FaHeart, - FaHeartPulse, FaLayerGroup, FaMicrochip, FaMobileScreen, FaRankingStar, FaScaleBalanced, - FaServer, FaShieldHalved, FaSitemap, FaStar, FaStopwatch, FaTableCells, FaTag, + FaCopy, FaCube, FaCubes, FaDatabase, FaDna, FaDownload, FaFileShield, FaFlaskVial, FaGaugeHigh, + FaHeart, FaHeartPulse, FaLayerGroup, FaMicrochip, FaMobileScreen, FaRankingStar, + FaScaleBalanced, FaServer, FaShieldHalved, FaSitemap, FaStar, FaStopwatch, FaTableCells, FaTag, FaTriangleExclamation, FaUsers, FaVial, }; use dioxus_free_icons::Icon; @@ -380,6 +380,8 @@ pub fn Overview() -> Element { {cross_family_section()} + {build_cost_section()} + div { class: "section-head", h2 { Icon { width: 18, height: 18, fill: "currentColor".to_string(), class: "h2-ico".to_string(), icon: FaDatabase } @@ -738,7 +740,66 @@ pub fn DialectView(dir: String) -> Element { } } -/// Per-parser detail page: how one parser does across the dialects it supports. +/// Cross-parser build-cost table: one row per parser with a current release in +/// the build scan (CPU time, wall time, crate-only time, crates compiled, +/// binary size added). Renders nothing when the scan has no current entries. +fn build_cost_section() -> Element { + let scan = crate::data::build_scan(); + let columns: Vec = [ + "CPU time", + "wall time", + "crate time", + "crates", + "binary size", + ] + .iter() + .map(ToString::to_string) + .collect(); + let rows: Vec = scan + .entries + .iter() + .filter(|e| e.current) + .map(|e| Row { + key: e.parser.clone(), + head: Head::Parser(e.parser.clone()), + cells: vec![ + Cell::with(format!("{:.1} s", e.cpu_secs), Some(e.cpu_secs)), + Cell::with(format!("{:.1} s", e.wall_secs), Some(e.wall_secs)), + e.crate_secs + .map_or_else(Cell::na, |s| Cell::with(format!("{:.1} s", s), Some(s))), + Cell::with(commas(e.units), Some(e.units as f64)), + Cell::with( + fmt_bytes(e.binary_bytes as f64), + Some(e.binary_bytes as f64), + ), + ], + }) + .collect(); + if rows.is_empty() { + return rsx! {}; + } + rsx! { + div { class: "section-head", + h2 { + Icon { width: 18, height: 18, fill: "currentColor".to_string(), class: "h2-ico".to_string(), icon: FaCubes } + "Build cost" + } + } + div { class: "block", + p { class: "table-cap", + {format!("A clean release build of a minimal stdin-parse program per parser release (read stdin, parse it, print the statement count). CPU time is the build's CPU seconds, user plus system, including every dependency and build script. Wall time is the wall-clock seconds at -j {} on a shared machine, so it moves with machine load. Crate time is the parser crate alone, crates the number compiled, and binary size the stripped size minus the same program without the parser ({}).", scan.jobs, scan.rustc)} + } + SortTable { + caption: "Clean release build cost per parser".to_string(), + corner: "parser".to_string(), + columns, + rows, + footer: None, + } + } + } +} + #[component] pub fn ParserView(name: String) -> Element { let b = bundle(); @@ -901,6 +962,8 @@ pub fn ParserView(name: String) -> Element { {parser_memory_section(b, &parser)} + {parser_build_section(&parser)} + VersionHistory { parser: parser.clone() } {failures_section(b, &parser)} @@ -998,6 +1061,7 @@ fn VersionHistory(parser: String) -> Element { .collect(); let mut time_series = Vec::new(); let mut peak_series = Vec::new(); + let mut alloc_series = Vec::new(); let mut recall_series = Vec::new(); let mut fp_series = Vec::new(); let mut coverage_series = Vec::new(); @@ -1008,6 +1072,7 @@ fn VersionHistory(parser: String) -> Element { let rgb = brand(dir).accent_rgb; let mut time_points = Vec::new(); let mut peak_points = Vec::new(); + let mut alloc_points = Vec::new(); let mut recall_points = Vec::new(); let mut fp_points = Vec::new(); let mut coverage_points = Vec::new(); @@ -1026,6 +1091,7 @@ fn VersionHistory(parser: String) -> Element { } if let Some(m) = dr.memory.as_ref() { peak_points.push((x, m.peak.median, m.peak.p25, m.peak.p75)); + alloc_points.push((x, m.allocs.median, m.allocs.p25, m.allocs.p75)); } if let Some(c) = dr.correctness.as_ref() { let accept = if dr.has_reference { @@ -1060,6 +1126,7 @@ fn VersionHistory(parser: String) -> Element { }; time_series.push(series(time_points)); peak_series.push(series(peak_points)); + alloc_series.push(series(alloc_points)); if !recall_points.is_empty() { recall_series.push(series(recall_points)); } @@ -1093,6 +1160,13 @@ fn VersionHistory(parser: String) -> Element { 460, "bytes / statement", ); + let alloc_trend = viz::trend_lines( + &format!("{parser} allocation pressure"), + &alloc_series, + 760, + 460, + "allocations / statement", + ); let recall_trend = viz::pct_trend_lines( &format!("{parser} accept / recall"), &recall_series, @@ -1271,11 +1345,12 @@ fn VersionHistory(parser: String) -> Element { "Across versions" } p { class: "table-cap", - "How {parser} changed across releases, each version placed at its release date. For time and memory each point is the median over a dialect's accepted statements with an interquartile (p25 to p75) bar on a log scale, so the heavily right-skewed tails do not distort it: lower is faster and leaner. Coverage is the raw count of valid statements accepted, and the quality trends show recall (acceptance rate where there is no reference engine), recall excluding contentious constructs, round-trip stability, the share of invalid statements wrongly accepted (lower is better), and the empirical panic rate. Pick a version to see its full charts, its results, and exactly which statements it gained or lost versus the previous release." + "How {parser} changed across releases, each version placed at its release date. For time, memory, and allocation each point is the median over a dialect's accepted statements with an interquartile (p25 to p75) bar on a log scale, so the heavily right-skewed tails do not distort it: lower is faster, leaner, and calmer. Coverage is the raw count of valid statements accepted, and the quality trends show recall (acceptance rate where there is no reference engine), recall excluding contentious constructs, round-trip stability, the share of invalid statements wrongly accepted (lower is better), and the empirical panic rate. Pick a version to see its full charts, its results, and exactly which statements it gained or lost versus the previous release." } div { class: "charts", {chart_figure(&format!("chart-{pslug}-time-trend"), &time_trend, &format!("Parse-time trend for {parser} across releases, one line per dialect."), "Median parse time by release date, one line per dialect (log scale, interquartile bars).", &format!("{pslug}-time-trend"))} {chart_figure(&format!("chart-{pslug}-mem-trend"), &peak_trend, &format!("Peak-memory trend for {parser} across releases, one line per dialect."), "Median peak memory by release date, one line per dialect (log scale, interquartile bars).", &format!("{pslug}-mem-trend"))} + {chart_figure(&format!("chart-{pslug}-alloc-trend"), &alloc_trend, &format!("Allocation-pressure trend for {parser} across releases, one line per dialect."), "Median allocations per statement by release date, one line per dialect (log scale, interquartile bars).", &format!("{pslug}-alloc-trend"))} if let Some(t) = &coverage_trend { {chart_figure(&format!("chart-{pslug}-coverage-trend"), t, &format!("Coverage trend for {parser} across releases, one line per dialect."), "Count of valid statements accepted by release date, one line per dialect (linear scale). Higher is broader coverage.", &format!("{pslug}-coverage-trend"))} } @@ -1407,6 +1482,63 @@ fn parser_memory_section(b: &viz::Bundle, parser: &str) -> Element { 460, "bytes / statement", ); + // Allocation pressure, the same dialects in the same order. + let alloc_columns = [ + "allocs p50", + "allocs p90", + "allocs mean", + "batch allocs/stmt", + "alloc'd p50", + "alloc'd p90", + "alloc'd mean", + "batch alloc'd/stmt", + ] + .iter() + .map(ToString::to_string) + .collect(); + let alloc_rows: Vec = b + .dialects + .iter() + .filter_map(|d| { + d.memory.iter().find(|m| m.parser == parser).map(|m| Row { + key: d.dir_name.clone(), + head: Head::Dialect { + dir: d.dir_name.clone(), + name: d.display_name.clone(), + }, + cells: vec![ + Cell::count(Some(m.allocs.median)), + Cell::count(Some(m.allocs.p90)), + Cell::count(Some(m.allocs.mean)), + Cell::count(batch_of(d, parser).and_then(|x| x.allocs_per_stmt)), + Cell::bytes(Some(m.allocated.median)), + Cell::bytes(Some(m.allocated.p90)), + Cell::bytes(Some(m.allocated.mean)), + Cell::bytes(batch_of(d, parser).and_then(|x| x.allocated_per_stmt)), + ], + }) + }) + .collect(); + let alloca_lines: Vec = b + .dialects + .iter() + .filter_map(|d| { + d.memory.iter().find(|m| m.parser == parser).map(|m| { + viz::mem_line( + d.display_name.clone(), + brand(&d.dir_name).accent_rgb, + &m.allocs, + ) + }) + }) + .collect(); + let alloca_ecdf = viz::ecdf_lines( + &format!("{parser} allocation pressure"), + &alloca_lines, + 760, + 460, + "allocations / statement", + ); rsx! { section { class: "block", h2 { @@ -1428,6 +1560,127 @@ fn parser_memory_section(b: &viz::Bundle, parser: &str) -> Element { footer: None, } } + section { class: "block", + h2 { + Icon { width: 17, height: 17, fill: "currentColor".to_string(), class: "h2-ico".to_string(), icon: FaGaugeHigh } + "Allocation pressure" + } + p { class: "table-cap", + "Every allocator call and every requested byte during the parse, including memory freed before it ends, so a parser can show a small peak yet heavy churn. \"allocs\" is the count of allocator calls, \"alloc'd\" the bytes requested from the allocator (a reallocation counts its new size). \"allocs mean\" and \"alloc'd mean\" are the per-statement averages, and \"batch allocs/stmt\" and \"batch alloc'd/stmt\" are the same over random 128-statement scripts, averaged over the ones that reparsed correctly, so compare each batch column to the adjacent mean." + } + div { class: "charts", + {chart_figure(&format!("chart-{}-alloca-ecdf", slug(parser)), &alloca_ecdf, &format!("Empirical CDF of {parser} allocation count, one curve per dialect."), "Allocator calls per parse, one curve per dialect. Further left is calmer (log scale).", &format!("{}-allocation-ecdf", slug(parser)))} + } + SortTable { + caption: format!("Per-dialect allocation pressure for {}", parser), + corner: "dialect".to_string(), + columns: alloc_columns, + rows: alloc_rows, + footer: None, + } + } + } +} + +/// The "Build cost" section for a parser: the current release's clean-build +/// cost (CPU time, wall time, crate-only time, crates compiled, binary size +/// added), plus compile-CPU and binary-size trends over the time-machine +/// milestones where they exist. Renders nothing when the build scan has no +/// current entry for the parser. +fn parser_build_section(parser: &str) -> Element { + let scan = crate::data::build_scan(); + let Some(cur) = crate::data::current_build(parser) else { + return rsx! {}; + }; + let pslug = slug(parser); + // Milestone trend points, release dates joined by version string against + // the time-machine history. + let mut cpu_points = Vec::new(); + let mut size_points = Vec::new(); + if let Some(hist) = crate::data::history(parser) { + for e in crate::data::build_history(parser) { + let Some(v) = hist.versions.iter().find(|v| v.version == e.version) else { + continue; + }; + let Some(x) = viz::year_frac(&v.released) else { + continue; + }; + cpu_points.push((x, e.cpu_secs, e.cpu_secs, e.cpu_secs)); + // u64 -> f64 loses nothing below 512 EiB. + let bytes = e.binary_bytes as f64; + size_points.push((x, bytes, bytes, bytes)); + } + } + let has_trends = !cpu_points.is_empty(); + let trend = |title: String, points: Vec<(f64, f64, f64, f64)>, y: &str| { + viz::trend_lines( + &title, + &[viz::TrendSeries { + label: parser.to_string(), + rgb: parser_rgb(parser), + points, + }], + 760, + 460, + y, + ) + }; + let (cpu_trend, size_trend) = if has_trends { + ( + trend( + format!("{parser} compile time"), + cpu_points, + "CPU seconds (clean release build)", + ), + trend( + format!("{parser} binary size"), + size_points, + "bytes added (stripped binary, baseline subtracted)", + ), + ) + } else { + (String::new(), String::new()) + }; + let mut cap = format!( + "The clean release build of a minimal program (read stdin, parse it, print the statement count) of the release this page benchmarks, {} from {}, built with {} at -j {}.", + cur.version, cur.source, scan.rustc, scan.jobs + ); + if !scan.note.is_empty() { + cap.push(' '); + cap.push_str(&scan.note); + } + if has_trends { + cap.push_str(" The trends place each time-machine milestone at its release date."); + } + let cpu = format!("{:.1} s", cur.cpu_secs); + let wall = format!("{:.1} s", cur.wall_secs); + let crate_only = cur.crate_secs.map(|s| format!("{:.1} s", s)); + let units = commas(cur.units); + let added = fmt_bytes(cur.binary_bytes as f64); + + rsx! { + section { class: "block", + h2 { + Icon { width: 17, height: 17, fill: "currentColor".to_string(), class: "h2-ico".to_string(), icon: FaCubes } + "Build cost" + } + p { class: "table-cap", {cap} } + p { class: "hero-stats", + span { class: "stat", strong { {cpu} } " CPU time (every dependency and build script)" } + span { class: "stat", strong { {wall} } " wall time" } + if let Some(cs) = crate_only { + span { class: "stat", strong { {cs} } " crate-only time" } + } + span { class: "stat", strong { {units} } " crates compiled" } + span { class: "stat", strong { {added} } " binary size added" } + } + if has_trends { + div { class: "charts", + {chart_figure(&format!("chart-{pslug}-build-cpu-trend"), &cpu_trend, &format!("Compile-time trend for {parser} across releases."), "CPU seconds of the clean release build by release date (log scale).", &format!("{pslug}-build-cpu-trend"))} + {chart_figure(&format!("chart-{pslug}-build-size-trend"), &size_trend, &format!("Binary-size trend for {parser} across releases."), "Bytes the stripped release binary adds over the same program without the parser, by release date (log scale).", &format!("{pslug}-build-size-trend"))} + } + } + } } } @@ -2444,6 +2697,34 @@ impl Cell { sub: None, } } + /// Count cell from an optional value, for allocator calls rather than + /// bytes: whole counts comma-grouped, fractional means with one decimal + /// ("N/A" if missing). + fn count(v: Option) -> Cell { + Cell { + text: v.map_or_else( + || "N/A".to_string(), + |x| { + if (x - x.round()).abs() < 0.05 { + // Display quantization to the nearest whole call. + commas(x.round() as usize) + } else { + format!("{x:.1}") + } + }, + ), + num: v, + sub: None, + } + } + /// An "N/A" cell with no sort key: the value was never measured. + fn na() -> Cell { + Cell { + text: "N/A".to_string(), + num: None, + sub: None, + } + } /// Cell with explicit text and a numeric sort key. fn with(text: String, num: Option) -> Cell { Cell { @@ -2564,6 +2845,19 @@ fn col_help(name: &str) -> Option<&'static str> { "peak p90" => "90th-percentile peak live memory per statement: nine in ten statements stay under this high-water mark.", "retained p50" => "Median retained memory per statement: the bytes the produced AST (plus the scaffolding it keeps alive) holds after parsing. Half of statements retain less.", "retained p90" => "90th-percentile retained memory per statement: the AST footprint nine in ten statements stay under.", + "allocs p50" => "Median allocator calls while parsing one statement (allocations and reallocations, a count rather than bytes): the churn that half of statements stay under.", + "allocs p90" => "90th-percentile allocator calls per statement: nine in ten statements make fewer allocator calls than this.", + "allocs mean" => "Mean allocator calls per statement: the per-statement average count of allocator calls, freed memory included.", + "batch allocs/stmt" => "Allocator calls per statement inside a multi-statement script, averaged over the batches that parsed correctly. Compare with the adjacent mean to see what bulk parsing amortizes. Blank only when no sampled batch parsed correctly.", + "alloc'd p50" => "Median bytes requested from the allocator while parsing one statement, freed or not (a reallocation counts its new size): the total allocation traffic that half of statements stay under.", + "alloc'd p90" => "90th-percentile bytes requested from the allocator per statement, freed or not: the allocation traffic nine in ten statements stay under.", + "alloc'd mean" => "Mean bytes requested from the allocator per statement, freed or not: the per-statement average allocation traffic.", + "batch alloc'd/stmt" => "Bytes requested from the allocator per statement inside a multi-statement script, averaged over the batches that parsed correctly. Compare with the adjacent mean. Blank only when no sampled batch parsed correctly.", + "CPU time" => "CPU seconds (user plus system) of a clean release build of a minimal parse program, every dependency and build script included.", + "wall time" => "Wall-clock seconds of the same clean release build, run at the scan's -j on a shared machine.", + "crate time" => "Seconds the build spent compiling the parser crate itself, without its dependencies. Blank when the timing report could not be read.", + "crates" => "Crates the build compiled: the parser and its whole dependency tree.", + "binary size" => "Stripped release binary size of the parse program minus the same program without the parser: what the parser adds after dead-code elimination.", "batch ok%" => "Batch parse rate: of 200 random 128-statement scripts built from statements this parser accepts individually, the share it reparsed to the exact statement count. Below 100% means it mishandles a statement boundary (for example swallowing the terminator) in some multi-statement scripts. Higher is better.", "batch ns/stmt" => "Per-statement parse time inside a multi-statement script, averaged over the batches that parsed correctly. Compare with mean ns to see what bulk parsing amortizes. Blank only when no sampled batch parsed correctly.", _ => return None, @@ -2797,6 +3091,49 @@ fn memory_table(d: &DialectData) -> Element { let peak_box = viz::box_lines(&peak_title, &peak_lines, 760, 460, "bytes / statement"); let ret_ecdf = viz::ecdf_lines(&ret_title, &ret_lines, 760, 460, "bytes / statement"); let ret_box = viz::box_lines(&ret_title, &ret_lines, 760, 460, "bytes / statement"); + // Allocation pressure, the same parsers in the same order. + let alloc_columns = [ + "allocs p50", + "allocs p90", + "allocs mean", + "batch allocs/stmt", + "alloc'd p50", + "alloc'd p90", + "alloc'd mean", + "batch alloc'd/stmt", + ] + .iter() + .map(ToString::to_string) + .collect(); + let alloc_rows = mems + .iter() + .map(|m| Row { + key: m.parser.clone(), + head: Head::Parser(m.parser.clone()), + cells: vec![ + Cell::count(Some(m.allocs.median)), + Cell::count(Some(m.allocs.p90)), + Cell::count(Some(m.allocs.mean)), + Cell::count(batch_of(d, &m.parser).and_then(|x| x.allocs_per_stmt)), + Cell::bytes(Some(m.allocated.median)), + Cell::bytes(Some(m.allocated.p90)), + Cell::bytes(Some(m.allocated.mean)), + Cell::bytes(batch_of(d, &m.parser).and_then(|x| x.allocated_per_stmt)), + ], + }) + .collect(); + let alloca_lines: Vec = mems + .iter() + .map(|m| viz::mem_line(m.parser.clone(), parser_rgb(&m.parser), &m.allocs)) + .collect(); + let alloca_title = format!("{} allocation pressure", d.display_name); + let alloca_ecdf = viz::ecdf_lines( + &alloca_title, + &alloca_lines, + 760, + 460, + "allocations / statement", + ); rsx! { section { class: "block", h2 { @@ -2820,6 +3157,25 @@ fn memory_table(d: &DialectData) -> Element { footer: None, } } + section { class: "block", + h2 { + Icon { width: 17, height: 17, fill: "currentColor".to_string(), class: "h2-ico".to_string(), icon: FaGaugeHigh } + "Allocation pressure" + } + p { class: "table-cap", + "Every allocator call and every requested byte during the parse, including memory freed before it ends, so a parser can show a small peak yet heavy churn. \"allocs\" is the count of allocator calls, \"alloc'd\" the bytes requested from the allocator (a reallocation counts its new size). \"allocs mean\" and \"alloc'd mean\" are the per-statement averages, and \"batch allocs/stmt\" and \"batch alloc'd/stmt\" are the same over random 128-statement scripts, averaged over the ones that reparsed correctly, so compare each batch column to the adjacent mean." + } + div { class: "charts", + {chart_figure(&format!("chart-{}-alloca-ecdf", d.dir_name), &alloca_ecdf, &format!("Empirical CDF of allocation count for {}, one curve per parser.", d.display_name), "Allocator calls per parse, one curve per parser. Further left is calmer (log scale).", &format!("{}-allocation-ecdf", d.dir_name))} + } + SortTable { + caption: format!("Per-parser allocation pressure for {}", d.display_name), + corner: "parser".to_string(), + columns: alloc_columns, + rows: alloc_rows, + footer: None, + } + } } } diff --git a/web/src/data.rs b/web/src/data.rs index 920d285..e72f88f 100644 --- a/web/src/data.rs +++ b/web/src/data.rs @@ -7,7 +7,10 @@ //! (the bundle is ~25x smaller compressed). use std::sync::OnceLock; -use viz::{Bundle, DepthReport, DepthScan, FamilyHistory, FeatureScan, ParserFeatures}; +use viz::{ + BuildCost, BuildScan, Bundle, DepthReport, DepthScan, FamilyHistory, FeatureScan, + ParserFeatures, +}; /// The results bundle, zstd-compressed and embedded. static BUNDLE_RAW: &[u8] = include_bytes!("../assets/bench.json.zst"); @@ -22,6 +25,9 @@ static FEATURESCAN_RAW: &str = include_str!("../../featurescan/data/featurescan. /// Recursion-depth probe results, committed uncompressed. static DEPTH_RAW: &str = include_str!("../../featurescan/data/depth.json"); +/// Build cost of every benchmarked parser release, committed uncompressed. +static BUILD_RAW: &str = include_str!("../../featurescan/data/build.json"); + /// Decompress an embedded zstd blob to bytes. fn unzstd(raw: &[u8]) -> Vec { let mut decoder = ruzstd::StreamingDecoder::new(raw).expect("embedded blob is valid zstd"); @@ -82,6 +88,38 @@ pub fn parser_depth(parser: &str) -> Option<&'static DepthReport> { depth_scan().parsers.iter().find(|p| p.parser == parser) } +/// The build-cost scan (method note, toolchain, `-j`, every entry, parsed once). +fn build_scan_raw() -> &'static BuildScan { + static CACHE: OnceLock = OnceLock::new(); + CACHE.get_or_init(|| serde_json::from_str(BUILD_RAW).expect("build.json is valid")) +} + +/// The parsed build-cost scan. +#[must_use] +pub fn build_scan() -> &'static BuildScan { + build_scan_raw() +} + +/// The release the main snapshot benchmarks, for one parser by display name +/// (the `current == true` entry). +#[must_use] +pub fn current_build(parser: &str) -> Option<&'static BuildCost> { + build_scan_raw() + .entries + .iter() + .find(|e| e.parser == parser && e.current) +} + +/// Time-machine milestones for one parser, by display name, in committed order. +#[must_use] +pub fn build_history(parser: &str) -> Vec<&'static BuildCost> { + build_scan_raw() + .entries + .iter() + .filter(|e| e.parser == parser && !e.current) + .collect() +} + /// Aggregate empirical panic totals for one parser across every dialect it runs: /// `(panicked, attempted)`. The per-parser panic rate is `panicked / attempted`. /// Returns `None` if nothing was attempted (e.g. an older snapshot without the @@ -128,8 +166,9 @@ mod tests { assert!(!b.parsers.is_empty()); } - /// The committed feature-scan and depth snapshots must parse into the shared - /// schema, failing the build if `featurescan` output and `viz` drift. + /// The committed feature-scan, depth-scan, and build-cost snapshots must + /// parse into the shared schema, failing the build if `featurescan` output + /// and `viz` drift. #[test] fn committed_featurescan_and_depth_parse() { assert!(!super::featurescan().parsers.is_empty()); @@ -137,5 +176,8 @@ mod tests { // sqlparser-rs is covered by both scans. assert!(super::parser_features("sqlparser-rs").is_some()); assert!(super::parser_depth("sqlparser-rs").is_some()); + // The build-cost snapshot parses with its method note and parallelism. + let scan = super::build_scan(); + assert!(scan.jobs > 0); } } diff --git a/web/src/metadata.rs b/web/src/metadata.rs index ad0c3e2..443132e 100644 --- a/web/src/metadata.rs +++ b/web/src/metadata.rs @@ -9,7 +9,7 @@ use crate::cadence::Cadence; /// The date the figures below were collected (ISO 8601). -pub const SNAPSHOT: &str = "2026-05-31"; +pub const SNAPSHOT: &str = "2026-11-09"; /// Whether a parser ships its own fuzzing harness. #[derive(Clone, Copy, PartialEq)] @@ -254,7 +254,7 @@ pub struct ParserMeta { /// Releases at least this recent count as actively maintained. Roughly twelve /// months before [`SNAPSHOT`]. `"YYYY-MM"` strings order by date. -const MAINTAINED_SINCE: &str = "2025-05"; +const MAINTAINED_SINCE: &str = "2025-11"; /// Whether the crate's latest release is recent enough to look maintained. #[must_use] @@ -367,10 +367,10 @@ pub fn repo_description(url: &str) -> String { pub fn parser_meta(name: &str) -> Option { Some(match name { "sqlparser-rs" => ParserMeta { - stars: 3460, - forks: 774, - commits: 2024, - contributors: 336, + stars: 3480, + forks: 806, + commits: 2219, + contributors: 345, since: 2018, license: "Apache-2.0", fuzz: Fuzz::Yes, @@ -378,13 +378,13 @@ pub fn parser_meta(name: &str) -> Option { benches: true, no_std: true, wasm: true, - downloads: "78.0M", + downloads: "90.2M", cadence: Cadence::Quarterly, repo: "https://github.com/apache/datafusion-sqlparser-rs", crates_io: true, pure_rust: true, unsafe_note: "", - last_release: "2026-09", + last_release: "2026-11", sanitizers: "", cargo_audit: true, cargo_mutants: false, @@ -393,10 +393,10 @@ pub fn parser_meta(name: &str) -> Option { // Both libpg_query bindings: own crate has no harness, but libpg_query // (PostgreSQL's own C parser) is fuzzed upstream. "pg_query.rs" | "pg_query (summary)" => ParserMeta { - stars: 256, - forks: 29, - commits: 108, - contributors: 15, + stars: 259, + forks: 32, + commits: 111, + contributors: 16, since: 2022, license: "MIT", fuzz: Fuzz::Upstream, @@ -404,47 +404,47 @@ pub fn parser_meta(name: &str) -> Option { benches: true, no_std: false, wasm: false, - downloads: "2.4M", + downloads: "4.3M", cadence: Cadence::Quarterly, repo: "https://github.com/pganalyze/pg_query.rs", crates_io: true, pure_rust: false, unsafe_note: "the C FFI bindings to libpg_query", - last_release: "2026-08", + last_release: "2026-10", sanitizers: "leak", cargo_audit: false, cargo_mutants: false, cargo_deny: false, }, "qusql-parse" => ParserMeta { - stars: 20, - forks: 0, - commits: 830, - contributors: 8, + stars: 21, + forks: 1, + commits: 835, + contributors: 3, since: 2025, license: "Apache-2.0", fuzz: Fuzz::No, tests: true, benches: false, - no_std: false, + no_std: true, wasm: true, - downloads: "5.8k", + downloads: "7.7k", cadence: Cadence::Monthly, repo: "https://github.com/antialize/qusql", crates_io: true, pure_rust: true, unsafe_note: "one unchecked UTF-8 conversion in the lexer", - last_release: "2026-07", + last_release: "2026-10", sanitizers: "", cargo_audit: false, cargo_mutants: false, cargo_deny: false, }, "polyglot-sql" => ParserMeta { - stars: 956, - forks: 67, - commits: 187, - contributors: 13, + stars: 1004, + forks: 77, + commits: 233, + contributors: 14, since: 2026, license: "MIT", fuzz: Fuzz::No, @@ -452,31 +452,31 @@ pub fn parser_meta(name: &str) -> Option { benches: true, no_std: false, wasm: true, - downloads: "66.9k", + downloads: "138.9k", cadence: Cadence::Monthly, repo: "https://github.com/tobilg/polyglot", crates_io: true, pure_rust: true, unsafe_note: "", - last_release: "2026-09", + last_release: "2026-11", sanitizers: "", cargo_audit: false, cargo_mutants: false, cargo_deny: false, }, "databend-common-ast" => ParserMeta { - stars: 9442, - forks: 897, - commits: 34676, - contributors: 259, + stars: 9492, + forks: 901, + commits: 34938, + contributors: 263, since: 2020, - license: "custom", + license: "Apache-2.0", fuzz: Fuzz::Yes, tests: true, benches: true, no_std: false, wasm: false, - downloads: "31.8k", + downloads: "36.6k", cadence: Cadence::Quarterly, repo: "https://github.com/datafuselabs/databend/tree/main/src/query/ast", crates_io: true, @@ -489,10 +489,10 @@ pub fn parser_meta(name: &str) -> Option { cargo_deny: false, }, "sqlglot-rust" => ParserMeta { - stars: 32, - forks: 6, - commits: 215, - contributors: 4, + stars: 40, + forks: 7, + commits: 236, + contributors: 5, since: 2026, license: "MIT", fuzz: Fuzz::No, @@ -500,23 +500,23 @@ pub fn parser_meta(name: &str) -> Option { benches: true, no_std: false, wasm: true, - downloads: "4.0k", + downloads: "5.0k", cadence: Cadence::Monthly, repo: "https://github.com/protegrity/sql-glot-rust", crates_io: true, pure_rust: true, - unsafe_note: "its C-ABI export layer", - last_release: "2026-08", + unsafe_note: "its C-ABI export layer and internal raw pointer handling", + last_release: "2026-10", sanitizers: "", cargo_audit: false, cargo_mutants: false, cargo_deny: false, }, "sqlite3-parser" => ParserMeta { - stars: 65, - forks: 17, - commits: 542, - contributors: 7, + stars: 66, + forks: 19, + commits: 551, + contributors: 8, since: 2017, license: "Unlicense", fuzz: Fuzz::No, @@ -524,23 +524,23 @@ pub fn parser_meta(name: &str) -> Option { benches: true, no_std: false, wasm: true, - downloads: "3.3M", + downloads: "4.7M", cadence: Cadence::Quarterly, repo: "https://github.com/gwenn/lemon-rs", crates_io: true, pure_rust: true, unsafe_note: "", last_release: "2026-06", - sanitizers: "", + sanitizers: "miri", cargo_audit: false, cargo_mutants: false, cargo_deny: true, }, "turso_parser" => ParserMeta { - stars: 24303, - forks: 1351, - commits: 20349, - contributors: 330, + stars: 25204, + forks: 1539, + commits: 22960, + contributors: 389, since: 2023, license: "MIT", fuzz: Fuzz::Yes, @@ -548,13 +548,13 @@ pub fn parser_meta(name: &str) -> Option { benches: true, no_std: false, wasm: true, - downloads: "946k", + downloads: "1.6M", cadence: Cadence::Monthly, repo: "https://github.com/tursodatabase/turso", crates_io: true, pure_rust: true, unsafe_note: "unchecked UTF-8 conversions in the lexer and parser", - last_release: "2026-07", + last_release: "2026-10", sanitizers: "", cargo_audit: false, cargo_mutants: false,