Repository navigation
the public Prometheus host had no credential and an enabled admin API - #2786
Merged
Merged
Conversation
prom.openchainbench.com answers the internet. It is a bare reverse_proxy
with no auth, and Prometheus runs behind it with --web.enable-admin-api,
so a POST from anywhere could delete series. Measured from a laptop:
/api/v1/status/config returned 200 and the delete_series route answered
405 to a GET, which is Prometheus saying "wrong verb", not "go away".
The destructive routes are now blocked at the edge, which breaks nothing
because the access log shows the only paths in use are /api/v1/query and
/api/v1/query_range. This commit closes the rest: an Authorization header
on the one fetch every Prometheus request passes through.
It has to be a header. Node's fetch refuses a URL that embeds credentials
("Request cannot be constructed from a URL that includes credentials"),
so user:pass@ in PROMETHEUS_URL would not weaken authentication, it would
break every query. Tested before writing this.
Absent PROMETHEUS_BASIC_AUTH the helper adds nothing, and that is the
point: the rollout only works in one order, because Vercel binds env vars
at deployment rather than at request time.
1. this commit ships, inert
2. PROMETHEUS_BASIC_AUTH set in production
3. redeploy, so the value is actually bound
4. basic_auth turned on in Caddy
Reversed, every bench 401s until step 3 lands. A test pins step 1: an
unset or empty variable must produce no header at all.
Not sent over plain http, so the worker's in-network hop to ocb-prom:9090
never carries a credential it has no use for.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Flotapponnier
force-pushed
the
fix/prom-public-host-auth
branch
from
October 2, 2026 17:18
0f3eebf to
b56c7f2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Found while auditing why the VPS was dropping ssh connections.
What is open today
prom.openchainbench.comis a barereverse_proxy prometheus:9090with noauthentication, and Prometheus runs behind it with
--web.enable-admin-api.From a laptop, outside the network:
GET /api/v1/status/configGET /api/v1/query?query=upPOST /api/v1/admin/tsdb/delete_seriesThat 405 is Prometheus saying "wrong verb", not "go away". The route is
reachable, so a POST from anywhere could delete series.
Already closed at the edge, before this PR
The destructive routes now answer 403 in Caddy, wrapped in
routeso they areevaluated before the proxy. This breaks nothing: the access log over 40,000
requests shows the only paths in use are
/api/v1/query(25,223) and/api/v1/query_range(14,772).What this PR adds
An
Authorizationheader onfetchEnvelope, the single fetch every Prometheusrequest in the site passes through. Every other
fetchin the files thatmention
PROMETHEUS_URLturned out to target Slack, llama.fi or Upstash, soone place covers all of it.
It has to be a header rather than credentials in the URL. Node's fetch refuses
those outright, which I checked before writing the code:
So
user:pass@inPROMETHEUS_URLwould not weaken authentication, it wouldbreak every query on the site.
The rollout only works in one order
Vercel binds environment variables at deployment, not at request time, so:
PROMETHEUS_BASIC_AUTHis set in productionbasic_authgoes on in CaddyReversed, every bench 401s until step 3 lands.
prometheus-auth.test.tspinsstep 1: an unset or empty variable must produce no header at all.
The credential is never sent over plain http, so the worker's in-network hop to
ocb-prom:9090does not carry one it has no use for.Checks
bun test src/441 pass, 0 fail (7 new)pnpm exec tsc --noEmitcleanpnpm validate241 specs validContext
The audit that found this also cut the Prometheus query CPU in half, by
pointing the materialize worker at
ocb-prom:9090instead of routing its425 req/s back through the public edge in TLS.
assertPublicHostalreadyallow-listed that hostname, with a comment calling the public hop "exactly the
failure mode that broke under load"; only the container's env var had never
been switched over.
🤖 Generated with Claude Code