Skip to content

prod read hub snapshots from a store the worker never writes (main) - #2946

Merged
Flotapponnier merged 1 commit into
mainfrom
hotfix/cohort-snapshot-reads-worker-store
Oct 11, 2026
Merged

Flotapponnier merged 1 commit into
mainfrom
hotfix/cohort-snapshot-reads-worker-store

Conversation

@Flotapponnier

Copy link
Copy Markdown
Collaborator

Cherry-pick of #2945 (git cherry-pick -x 139a168c), applied cleanly. Only this fix.

Cohort snapshots now read from COHORT_KV_REST_API_URL/TOKEN (the VPS Redis the worker writes, via https://kv.openchainbench.com) instead of Upstash, which only had stale staging writebacks. Env vars already set on production + preview. Verified on staging: runtime reads from iad1 get 200 from SRH; the only 401s are the CI build, where vercel pull writes [SENSITIVE] (same as the existing Upstash token).

Rollback: delete the two COHORT_KV_* env vars and redeploy.

The VPS worker writes cohort snapshots (rpc-hub, perps, pm-hub,
search-featured...) to the VPS Redis behind SRH. Since the 2026-09-28
team migration the site's KV_REST_API_* point at Upstash, which only
gets staging's occasional writebacks: rpc-hub-v3 was 70 min old there
against a 10 min read ceiling, 5 min old in SRH. Every miss rebuilt the
RPC hub from ~100 variant blobs, on every /products render.

cohort-snapshot now prefers a COHORT_KV_REST_API_URL/TOKEN pair
(https://kv.openchainbench.com, the SRH route Caddy already exposes).
KV_REST_API_* stay on Upstash because snapshot.ts and compare-cache use
path-style /get and /set routes SRH does not serve. Credentials are now
picked as pairs so a URL never travels with another store's token.

(cherry picked from commit 139a168)
@Flotapponnier
Flotapponnier merged commit 4add64c into main Oct 11, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant