Skip to content

the public Prometheus host had no credential and an enabled admin API - #2786

Merged
Flotapponnier merged 1 commit into
mainfrom
fix/prom-public-host-auth
Oct 3, 2026
Merged

Flotapponnier merged 1 commit into
mainfrom
fix/prom-public-host-auth

Conversation

@Flotapponnier

Copy link
Copy Markdown
Collaborator

Found while auditing why the VPS was dropping ssh connections.

What is open today

prom.openchainbench.com is a bare reverse_proxy prometheus:9090 with no
authentication, and Prometheus runs behind it with --web.enable-admin-api.
From a laptop, outside the network:

request answer
GET /api/v1/status/config 200, the whole config
GET /api/v1/query?query=up 200
POST /api/v1/admin/tsdb/delete_series 405

That 405 is Prometheus saying "wrong verb", not "go away". The route is
reachable, so a POST from anywhere could delete series.

Already closed at the edge, before this PR

The destructive routes now answer 403 in Caddy, wrapped in route so they are
evaluated before the proxy. This breaks nothing: the access log over 40,000
requests shows the only paths in use are /api/v1/query (25,223) and
/api/v1/query_range (14,772).

POST /api/v1/admin/tsdb/delete_series    -> 403
POST /api/v1/admin/tsdb/clean_tombstones -> 403
POST /-/reload                           -> 403
GET  /api/v1/query?query=up              -> 200   (unchanged)

What this PR adds

An Authorization header on fetchEnvelope, the single fetch every Prometheus
request in the site passes through. Every other fetch in the files that
mention PROMETHEUS_URL turned out to target Slack, llama.fi or Upstash, so
one place covers all of it.

It has to be a header rather than credentials in the URL. Node's fetch refuses
those outright, which I checked before writing the code:

with URL credentials -> failed: Request cannot be constructed from a URL
                        that includes credentials

So user:pass@ in PROMETHEUS_URL would not weaken authentication, it would
break every query on the site.

The rollout only works in one order

Vercel binds environment variables at deployment, not at request time, so:

  1. this PR merges, inert, because the variable is unset
  2. PROMETHEUS_BASIC_AUTH is set in production
  3. redeploy, so the value is actually bound
  4. basic_auth goes on in Caddy

Reversed, every bench 401s until step 3 lands. prometheus-auth.test.ts pins
step 1: an unset or empty variable must produce no header at all.

The credential is never sent over plain http, so the worker's in-network hop to
ocb-prom:9090 does not carry one it has no use for.

Checks

  • bun test src/ 441 pass, 0 fail (7 new)
  • pnpm exec tsc --noEmit clean
  • pnpm validate 241 specs valid

Context

The audit that found this also cut the Prometheus query CPU in half, by
pointing the materialize worker at ocb-prom:9090 instead of routing its
425 req/s back through the public edge in TLS. assertPublicHost already
allow-listed that hostname, with a comment calling the public hop "exactly the
failure mode that broke under load"; only the container's env var had never
been switched over.

🤖 Generated with Claude Code

prom.openchainbench.com answers the internet. It is a bare reverse_proxy
with no auth, and Prometheus runs behind it with --web.enable-admin-api,
so a POST from anywhere could delete series. Measured from a laptop:
/api/v1/status/config returned 200 and the delete_series route answered
405 to a GET, which is Prometheus saying "wrong verb", not "go away".

The destructive routes are now blocked at the edge, which breaks nothing
because the access log shows the only paths in use are /api/v1/query and
/api/v1/query_range. This commit closes the rest: an Authorization header
on the one fetch every Prometheus request passes through.

It has to be a header. Node's fetch refuses a URL that embeds credentials
("Request cannot be constructed from a URL that includes credentials"),
so user:pass@ in PROMETHEUS_URL would not weaken authentication, it would
break every query. Tested before writing this.

Absent PROMETHEUS_BASIC_AUTH the helper adds nothing, and that is the
point: the rollout only works in one order, because Vercel binds env vars
at deployment rather than at request time.

  1. this commit ships, inert
  2. PROMETHEUS_BASIC_AUTH set in production
  3. redeploy, so the value is actually bound
  4. basic_auth turned on in Caddy

Reversed, every bench 401s until step 3 lands. A test pins step 1: an
unset or empty variable must produce no header at all.

Not sent over plain http, so the worker's in-network hop to ocb-prom:9090
never carries a credential it has no use for.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Flotapponnier
Flotapponnier force-pushed the fix/prom-public-host-auth branch from 0f3eebf to b56c7f2 Compare October 2, 2026 17:18
@Flotapponnier
Flotapponnier merged commit f699cbd into main Oct 3, 2026
2 checks passed
@Flotapponnier
Flotapponnier deleted the fix/prom-public-host-auth branch October 3, 2026 22:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant