Skip to content

Add weekly article: a benchmark found a bug in an access-control check - #69

Open
ashishsinha1602 wants to merge 1 commit into
Data-Engineering-Weekly:mainfrom
ashishsinha1602:add-access-control-benchmark-article
Open

ashishsinha1602 wants to merge 1 commit into
Data-Engineering-Weekly:mainfrom
ashishsinha1602:add-access-control-benchmark-article

Conversation

@ashishsinha1602

@ashishsinha1602 ashishsinha1602 commented Oct 18, 2026 •

Copy link
Copy Markdown

schemagate in numbers

  • BIRD: Data Engineering Weekly Article submission #1 on the official BIRD Mini-Dev leaderboard in SQLite, PostgreSQL and MySQL (81.0 / 72.6 / 72.4 EX). BIRD hidden test set, scored by the BIRD team: 74.34 EX (simple 81.03 / moderate 72.25 / challenging 56.14; leaderboard row pending), dev 84.03 EX.
  • Tokens: ~91% fewer schema tokens on 50+ table databases (median, Spider 2.0-lite; BENCHMARKS.md).
  • Enterprise: MIT BEAVER warehouses 31-39% EX vs 11.4% for the paper's best setup; LiveSQLBench-Large 34.79% (both self-reported).
  • LangChain / LangGraph: SchemagateRetriever is listed in LangChain's official retriever integrations (langchain-schemagate) and works in LangChain and LangGraph agents; also a LlamaIndex retriever and the official MCP Registry. 34.5k PyPI downloads, Apache-2.0.

Suggesting an article for the weekly: an engineering write-up on testing LLM text-to-SQL access control against Spider-ACL (Findings of EACL 2026), whose labels come from real PostgreSQL GRANTs. Running against labels the author did not write exposed a fail-closed bug that had quietly refused valid queries, and the post covers the fix, the full 19,624-record result and its limits.

Disclosure: I wrote the article and maintain the open-source library it tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant