Skip to content

Run beacon's raw-WS indexer in production and retire the legacy firehose #2

Open

Description

@melvincarvalho

What

beacon serves did:nostr documents from Mongo, but the data is still kept fresh by the legacy nostr-beacon indexers (firehose, followshose) running under pm2 on nostr.social. firehose is crash-prone (hundreds of pm2 restarts). To own the full pipeline, run beacon's own clean-room indexer (src/indexer.js — raw WebSocket, no nostr-tools) in production and retire firehose.

Scope

  • Deploy src/indexer.js under pm2 (e.g. beacon-indexer), subscribing to kinds 0/3/10002 across the relay set, latest-wins upsert into beacon / follows / relay_lists.
  • Validate it keeps the collections fresh (new profiles/follows show up), then retire firehose.

Open question: follows schema

The new indexer writes the raw kind-3 event ({ tags: [["p", hex], ...] }). The existing prod follows collection is the derived shape followshose produces: { pubkey, follows: [hex, ...], count, created_at }. The resolver (diddoc.js) already reads both, but writing two shapes into one collection is messy — decide the canonical going-forward shape (and whether the new indexer should emit the derived shape for consistency).

Watch-outs

  • followshose may also populate the relay-health relays collection (uptime stats) that the new indexer does not produce — confirm before retiring it, or keep that piece.
  • The indexer writes to the live prod Mongo (same DB the live resolver reads) — stage carefully.
  • Preserve latest-wins dedupe; don't regress already-indexed newer events.

Acceptance

  • beacon's indexer runs in prod, keeping beacon / follows / relay_lists fresh.
  • firehose retired (and followshose retired or its relay-health role preserved).
  • No resolver regression; the chosen follows schema is documented.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    diddid:nostr / DID methodenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions