From 0809359694ebe94583aecf8e2f334e133af4cd99 Mon Sep 17 00:00:00 2001 From: Rick Radewagen Date: Thu, 12 Nov 2026 20:46:00 +0100 Subject: [PATCH] Submit MalloyBench semantic query engineering series --- weekly/malloybench-by-heart.yml | 9 +++++++++ weekly/malloybench-compiler-feedback.yml | 9 +++++++++ weekly/malloybench-language-and-compiler.yml | 9 +++++++++ 3 files changed, 27 insertions(+) create mode 100644 weekly/malloybench-by-heart.yml create mode 100644 weekly/malloybench-compiler-feedback.yml create mode 100644 weekly/malloybench-language-and-compiler.yml diff --git a/weekly/malloybench-by-heart.yml b/weekly/malloybench-by-heart.yml new file mode 100644 index 0000000..be275c9 --- /dev/null +++ b/weekly/malloybench-by-heart.yml @@ -0,0 +1,9 @@ +article: + title: "MalloyBench: how well do language models know Malloy by heart?" + author: + name: "GetDot (Rick Radewagen)" + linkedin: "" + twitter: "" + link: "https://www.getdot.ai/blog/malloybench-by-heart" + review: "The methodology behind MalloyBench, a benchmark for querying and constructing semantic models. Tasks span familiar, unusual and synthetic datasets. Correctness requires matching reference results on the full data and two samples, including hidden queries for model-building tasks. The article describes public and private task splits, contamination checks, reproducibility and limitations. This is a self-submission by the author, from GetDot; the engineering interest is the evaluation design rather than the model ranking." + tags: "semantic layers, Malloy, data modeling, evaluation, query correctness, reproducibility" diff --git a/weekly/malloybench-compiler-feedback.yml b/weekly/malloybench-compiler-feedback.yml new file mode 100644 index 0000000..924f6ff --- /dev/null +++ b/weekly/malloybench-compiler-feedback.yml @@ -0,0 +1,9 @@ +article: + title: "MalloyBench: language models write Malloy well with its compiler" + author: + name: "GetDot (Rick Radewagen)" + linkedin: "" + twitter: "" + link: "https://www.getdot.ai/blog/malloybench" + review: "An evaluation of compiler feedback for querying a semantic model in Malloy. The study uses 118 tasks over six datasets and checks result correctness against reference queries on the full data and two samples. It explains the compile, execute and submit loop, the remaining semantic failures, and cost and latency tradeoffs. The harness and public half of the tasks are open source. This is a self-submission by the author; the work is from GetDot. Results cover one engine and one run per task and setting." + tags: "semantic layers, query languages, Malloy, DuckDB, data modeling, query correctness" diff --git a/weekly/malloybench-language-and-compiler.yml b/weekly/malloybench-language-and-compiler.yml new file mode 100644 index 0000000..9ab7517 --- /dev/null +++ b/weekly/malloybench-language-and-compiler.yml @@ -0,0 +1,9 @@ +article: + title: "MalloyBench: what Malloy could improve for language models" + author: + name: "GetDot (Rick Radewagen)" + linkedin: "" + twitter: "" + link: "https://www.getdot.ai/blog/malloybench-mistakes" + review: "A failure analysis of queries against Malloy semantic models, covering recurring syntax mistakes, diagnostic quality and the effect of a language reference. Building the benchmark also exposed compiler defects, including silent wrong-result cases. The article links to upstream issues and self-contained DuckDB reproductions, and distinguishes errors a compiler can reject from incorrect results it cannot detect. This is a self-submission by the author, from GetDot." + tags: "query engines, compilers, Malloy, DuckDB, data quality, semantic layers, query correctness"