Repository navigation
the two new tools answered nothing on the question they were built for - #2765
Merged
Merged
Conversation
Both found by calling production after #2764 shipped, not by the tests, which exercised the shaping and never the filter path. recommend_provider returned an empty benchmark list whenever a chain was given. It passed the chain straight to getBenchmark, but a chain RPC bench IS a chain: solana-rpc declares no chain dimension, so the filter made it unresolvable and every match was dropped. "A Solana trading bot" came back with the guidance and nothing to apply it to. That is the common case rather than an edge, since every RPC bench is per-chain. compare_providers could not reach the keyed cohort. Chain RPC benches rank public endpoints and API-key providers apart, the default is public, and the tool took no tier. Asking it to compare Alchemy and QuickNode on Solana answered that neither is measured, about two providers sitting on the keyed tab. resolveBenchmark now applies only the filters a bench declares and drops the rest, so passing a chain is harmless where it is meaningless. compare_providers takes a tier, reports which cohort it answered on, and when a provider is missing it looks on the other cohorts before saying so: "not measured" when the answer is one tab away is the wrong answer.
Flotapponnier
added a commit
that referenced
this pull request
Oct 17, 2026
…3019) Preparing the dev to main merge, the conflict triage turned up one file group where dev is not ahead but behind, and behind on a leak. `src/lib/mcp-tools.ts` and the MCP route exist on both branches and were written independently, so they conflict as add/add. main's copies carry five commits dev never received: a thousandfold error and a 35k-token catalogue (#2764), the search endpoint handing back an API key it had been asked to search for (#2766), two tools that answered nothing (#2765), a third annotation the submission scan reads (#2768), and a chain name outranking the subject of the question (#2770). Measured rather than inferred from the titles: `redact` appears 7 times in main's mcp-tools.ts and 3 times in its route, and 0 times in either of dev's. Resolving those conflicts in dev's favour, which a bulk resolution would do, reintroduces the leak. Checked before overwriting: the dev-to-main diff is additive. main adds category-aware scoring, a minimum score that cuts prefix-only noise, and redactSecrets(); the only lines it removes are the ones it replaces. dev holds nothing in these files that main lacks. The privacy page is the same shape without the security edge: main's is 219 lines against dev's 140 and is alone in documenting retention and the Vercel dependency. dev's predates it, from when the page existed only because the app directory requires a policy URL. Doing this now, in its own change, means the merge no longer has a resolution that can silently restore a credential leak. Gates: tsc clean, eslint clean, 567 tests (up from 561: main's test file carries six more).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #2764. Both bugs found by calling production, not by the tests, which exercised the shaping and never the filter path.
recommend_provider returned nothing whenever a chain was given
It passed the chain straight to
getBenchmark, but a chain RPC bench IS a chain:solana-rpcdeclares nochaindimension, sogetBenchmark("solana-rpc", { chain: "solana" })resolves to nothing and every match was dropped. Verified directly — that call returnsunknown_slugon production.The search was fine;
search_benchmarks({ query: "solana solana trading bot", category: "RPCs" })returns 3 matches. They were all discarded one step later.This is the common case, not an edge: every RPC bench is per-chain, and the chain is the most natural thing for a user to say.
compare_providers could not reach the keyed cohort
Chain RPC benches rank two cohorts apart:
solana-rpcget_benchmarktakes atier;compare_providersdid not. So "compare Alchemy and QuickNode on Solana" answered that neither is measured — about two providers sitting on the other tab.The fix
resolveBenchmarkapplies only the filters a bench declares and drops the rest, so passing a chain is harmless where it is meaningless.compare_providersnow takes atier, reports whichcohortit answered on, and when a provider is missing it checks the other cohorts before saying so:{ "missing": ["alchemy"], "measuredOnAnotherCohort": { "alchemy": "keyed" }, "note": "Not on this cohort, but measured on: keyed. Call again with that tier." }"Not measured" when the answer is one tab away is the wrong answer, and the whole point of the missing-vs-loser rule was to stop the tool saying wrong things about providers.
Verified:
tscclean, 406 tests pass,pnpm validatepasses, lint 0 errors.