Repository navigation
prod read hub snapshots from a store the worker never writes - #2945
Merged
Merged
Conversation
The VPS worker writes cohort snapshots (rpc-hub, perps, pm-hub, search-featured...) to the VPS Redis behind SRH. Since the 2026-09-28 team migration the site's KV_REST_API_* point at Upstash, which only gets staging's occasional writebacks: rpc-hub-v3 was 70 min old there against a 10 min read ceiling, 5 min old in SRH. Every miss rebuilt the RPC hub from ~100 variant blobs, on every /products render. cohort-snapshot now prefers a COHORT_KV_REST_API_URL/TOKEN pair (https://kv.openchainbench.com, the SRH route Caddy already exposes). KV_REST_API_* stay on Upstash because snapshot.ts and compare-cache use path-style /get and /set routes SRH does not serve. Credentials are now picked as pairs so a URL never travels with another store's token.
Flotapponnier
added a commit
that referenced
this pull request
Oct 11, 2026
…2946) The VPS worker writes cohort snapshots (rpc-hub, perps, pm-hub, search-featured...) to the VPS Redis behind SRH. Since the 2026-09-28 team migration the site's KV_REST_API_* point at Upstash, which only gets staging's occasional writebacks: rpc-hub-v3 was 70 min old there against a 10 min read ceiling, 5 min old in SRH. Every miss rebuilt the RPC hub from ~100 variant blobs, on every /products render. cohort-snapshot now prefers a COHORT_KV_REST_API_URL/TOKEN pair (https://kv.openchainbench.com, the SRH route Caddy already exposes). KV_REST_API_* stay on Upstash because snapshot.ts and compare-cache use path-style /get and /set routes SRH does not serve. Credentials are now picked as pairs so a URL never travels with another store's token. (cherry picked from commit 139a168)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The VPS worker writes cohort snapshots (rpc-hub, perps, pm-hub, search-featured, hl-*) to the VPS Redis behind SRH. Since the 2026-09-28 team migration the site's
KV_REST_API_*point at Upstash, which only receives staging's occasional writebacks.Measured 2026-10-10:
rpc-hub-v3was 70 min old in Upstash against the reader's 10 min ceiling, 5 min old in SRH. Every miss falls through to a fresh build, so prod rebuilt the RPC hub from ~100 variant blobs on every/products/[slug]render (~200k kv fetches/day on that route alone, each an Observability Event), and hub data was rebuilt live instead of read from the worker.Change:
cohort-snapshotprefers a newCOHORT_KV_REST_API_URL/TOKENpair, set on production + preview tohttps://kv.openchainbench.com(the SRH route Caddy already exposes; POST command API verified 200, snapshots fresh).KV_REST_API_*stay on Upstash on purpose:snapshot.tsandcompare-cache.tsuse path-style/get/setroutes that SRH answers with "Endpoint not found". Credentials are now resolved as pairs, so a half-set pair never mixes one store's URL with another's token (test added).Rollback: delete the two
COHORT_KV_*env vars and redeploy; the code falls back toKV_REST_API_*.