Skip to content

Submit MalloyBench series on semantic queries and compiler correctness #81

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
zurferr wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Submit MalloyBench series on semantic queries and compiler correctness #81

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions weekly/malloybench-by-heart.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
article:
title: "MalloyBench: how well do language models know Malloy by heart?"
author:
name: "GetDot (Rick Radewagen)"
linkedin: ""
twitter: ""
link: "https://www.getdot.ai/blog/malloybench-by-heart"
review: "The methodology behind MalloyBench, a benchmark for querying and constructing semantic models. Tasks span familiar, unusual and synthetic datasets. Correctness requires matching reference results on the full data and two samples, including hidden queries for model-building tasks. The article describes public and private task splits, contamination checks, reproducibility and limitations. This is a self-submission by the author, from GetDot; the engineering interest is the evaluation design rather than the model ranking."
tags: "semantic layers, Malloy, data modeling, evaluation, query correctness, reproducibility"
9 changes: 9 additions & 0 deletions weekly/malloybench-compiler-feedback.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
article:
title: "MalloyBench: language models write Malloy well with its compiler"
author:
name: "GetDot (Rick Radewagen)"
linkedin: ""
twitter: ""
link: "https://www.getdot.ai/blog/malloybench"
review: "An evaluation of compiler feedback for querying a semantic model in Malloy. The study uses 118 tasks over six datasets and checks result correctness against reference queries on the full data and two samples. It explains the compile, execute and submit loop, the remaining semantic failures, and cost and latency tradeoffs. The harness and public half of the tasks are open source. This is a self-submission by the author; the work is from GetDot. Results cover one engine and one run per task and setting."
tags: "semantic layers, query languages, Malloy, DuckDB, data modeling, query correctness"
9 changes: 9 additions & 0 deletions weekly/malloybench-language-and-compiler.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
article:
title: "MalloyBench: what Malloy could improve for language models"
author:
name: "GetDot (Rick Radewagen)"
linkedin: ""
twitter: ""
link: "https://www.getdot.ai/blog/malloybench-mistakes"
review: "A failure analysis of queries against Malloy semantic models, covering recurring syntax mistakes, diagnostic quality and the effect of a language reference. Building the benchmark also exposed compiler defects, including silent wrong-result cases. The article links to upstream issues and self-contained DuckDB reproductions, and distinguishes errors a compiler can reject from incorrect results it cannot detect. This is a self-submission by the author, from GetDot."
tags: "query engines, compilers, Malloy, DuckDB, data quality, semantic layers, query correctness"