Repository navigation
the MCP server published a thousandfold error and 35k tokens of catalogue - #2764
Merged
Merged
Conversation
…ogue
Straight to main because the first of these is wrong data on a public
endpoint, and the endpoint is what agents cite us from. The legal pages that
shipped in the same PR to dev are deliberately left behind: they belong to the
app-directory submission, not here.
The values were wrong. Latency benches declaring unit "s" store milliseconds
by the fmtUnit convention, and the endpoint published the stored number beside
the declared unit: {"value":443.494,"unit":"s"} for a head lag the site and
/api/stat both render as 0.44 s. An agent citing value and unit together was
off by a thousand, under provider names, on three benches including number
001. valueInDeclaredUnit already existed for exactly this reason, its
docstring names this same bench, and /api/stat already called it; the MCP
server never did. Both the tool path and the resource path now go through it,
nine conversions for nine raw reads, and rankings carry the stored `ms`
alongside a `value` in the declared unit so either read is right.
list_benchmarks returned the catalogue. All 229 live benches in one response,
142,555 characters, about 35,638 tokens, for a question as broad as "what do
you measure". It is now compact, capped and filterable: 25 rows of nine short
fields, about 2,400 tokens, 94% less, and it says how many it held back.
The tools were named for the REST surface they mirrored, not for the questions
people ask, and a client routes on the description text. search_benchmarks
takes the user's words and ranks the catalogue against them, including by the
provider names a bench measures, so "is Helius faster than Triton" finds the
Solana bench without anyone knowing its slug. compare_providers returns a
provider it does not measure as `missing` rather than as a loser, because
those are different answers. recommend_provider returns the caveat with the
ranking, since "read p99 not p50 for a bot" is usually worth more.
query_prom is gone, with its allowlist and validator. Behind the allowlist it
was safe enough for a developer tool, but no end user asks for a PromQL
passthrough, it was attack surface with no reader, and every extra tool
dilutes the routing the useful ones depend on.
Shaping lives in src/lib/mcp-tools.ts so it is testable without a server: 13
tests cover the unit conversion in both directions, provider-name search, the
cap, the longest-phrase use-case match, and the missing-vs-loser rule.
Read-only annotations and structuredContent on every tool.
Flotapponnier
added a commit
that referenced
this pull request
Oct 1, 2026
#2765) Both found by calling production after #2764 shipped, not by the tests, which exercised the shaping and never the filter path. recommend_provider returned an empty benchmark list whenever a chain was given. It passed the chain straight to getBenchmark, but a chain RPC bench IS a chain: solana-rpc declares no chain dimension, so the filter made it unresolvable and every match was dropped. "A Solana trading bot" came back with the guidance and nothing to apply it to. That is the common case rather than an edge, since every RPC bench is per-chain. compare_providers could not reach the keyed cohort. Chain RPC benches rank public endpoints and API-key providers apart, the default is public, and the tool took no tier. Asking it to compare Alchemy and QuickNode on Solana answered that neither is measured, about two providers sitting on the keyed tab. resolveBenchmark now applies only the filters a bench declares and drops the rest, so passing a chain is harmless where it is meaningless. compare_providers takes a tier, reports which cohort it answered on, and when a provider is missing it looks on the other cohorts before saying so: "not measured" when the answer is one tab away is the wrong answer.
Flotapponnier
added a commit
that referenced
this pull request
Oct 17, 2026
…3019) Preparing the dev to main merge, the conflict triage turned up one file group where dev is not ahead but behind, and behind on a leak. `src/lib/mcp-tools.ts` and the MCP route exist on both branches and were written independently, so they conflict as add/add. main's copies carry five commits dev never received: a thousandfold error and a 35k-token catalogue (#2764), the search endpoint handing back an API key it had been asked to search for (#2766), two tools that answered nothing (#2765), a third annotation the submission scan reads (#2768), and a chain name outranking the subject of the question (#2770). Measured rather than inferred from the titles: `redact` appears 7 times in main's mcp-tools.ts and 3 times in its route, and 0 times in either of dev's. Resolving those conflicts in dev's favour, which a bulk resolution would do, reintroduces the leak. Checked before overwriting: the dev-to-main diff is additive. main adds category-aware scoring, a minimum score that cuts prefix-only noise, and redactSecrets(); the only lines it removes are the ones it replaces. dev holds nothing in these files that main lacks. The privacy page is the same shape without the security edge: main's is 219 lines against dev's 140 and is alone in documenting retention and the Vercel dependency. dev's predates it, from when the page existed only because the app directory requires a policy URL. Doing this now, in its own change, means the merge no longer has a resolution that can silently restore a credential leak. Gates: tsc clean, eslint clean, 567 tests (up from 561: main's test file carries six more).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Straight to
mainbecause the first of these is wrong data on a public endpoint, and that endpoint is what agents cite us from. The legal pages that shipped alongside this in #2759 are deliberately left ondev: they belong to the app-directory submission, not here.Same change already merged to
devas #2759; this carries only the MCP files.The values were wrong
valueunit/api/stat/aggregator-head-lagslist_benchmarkssLatency benches declaring
unit: "s"store milliseconds (thefmtUnitconvention informat.ts:44). The site converts,/api/statconverts, the MCP server never did. An agent citingvalue+unitpublished 443 seconds of head lag instead of 0.44.Three benches affected:
aggregator-head-lag(bench 001),l1-finality,pm-resolution-delay.valueInDeclaredUnitalready existed for exactly this reason and its docstring names this same benchmark — it was written when the bug was found in the REST route and never carried across. Both the tool path and the resource path now use it: nine conversions for nine raw reads. Rankings carry the storedmsalongside avaluein the declared unit, so either read is right.list_benchmarks returned the whole catalogue
229 live benches, 142,555 characters, ~35,638 tokens in one response, for a question as broad as "what do you measure". Measured against the live endpoint.
Now compact, capped and filterable: 25 rows of nine short fields, ~2,400 tokens, 94% less, and it reports how many it held back.
The tools were named after the REST surface
A client routes on the description text, so
list_benchmarkswas never going to match "which Solana RPC should I use".search_benchmarkstakes the user's words and ranks the catalogue against them, including by the provider names a bench measures — "is Helius faster than Triton" finds the Solana bench without anyone knowing its slugcompare_providersreturns a provider it does not measure inmissingrather than as a loser; those are different answers and conflating them is how a comparison tool starts lyingrecommend_providerreturns the caveat with the ranking ("read p99 not p50 for a bot"), usually worth more than the ranking itselfquery_promis removed, with its allowlist and validator. Behind the allowlist it was safe enough for a developer tool, but no end user asks for a PromQL passthrough, it was attack surface with no reader, and every extra tool dilutes the routing the useful ones depend on.Verification
tsc --noEmitclean,pnpm validatepasses,pnpm lint0 errorsmain's own base, not justdev'sRead-only annotations and
structuredContenton every tool.