[REQUEST] log and visualize key performance parameters #144

Activity

1. Giszmo

   added

   enhancementNew feature or requestNew feature or request

   on Jan 18, 2023
2. Giszmo

   assigned

   cameri

   on Jan 18, 2023
3. bleetube commented on Feb 1, 2023

   

   bleetube

   on Feb 1, 2023

   Last edited by bleetube

   More actions

   I'll also add $100 (of sats) to this pledge. (see below) For an MVP I would be satisfied just with basic metrics for relayed nostr events.

   @Giszmo Most of the stuff you're asking for has little or nothing to do with nostream itself and already handled in other ways, or sounds relatively complex to implement.

   • concurrent websockets open - that's a proxy concern, not nostream
   • concurrent queries watched - medium complexity (minding connection timeouts)
   • websockets opened/closed - I'd guess that's the downstream proxy's concern (e.g nginx), not nostream
   • time from connect to first EOSE - connections are handled by nginx so this is
   • time from [e:[] query to EOSE - high complexity (summary metric)
   • events served - agreed
   • Standard system load parameters: CPU, Load, Memory, Disk I/O, Disk Usage, Bandwidth - just use prometheus + node_exporter or a comparable stack

   Creating a grafana dashboard is easy enough, I can do that side of it. Just need a metrics endpoint. I've done those too, but not in TS.

   👍React with 👍1Reacted by Anton Livaja
4. jmarbach commented on Feb 8, 2023

   

   jmarbach

   on Feb 8, 2023

   Last edited by jmarbach

   More actions

   @Giszmo @bleetube I've shared your request with the Grafana staff to raise awareness on building a metrics collector. No promises.

   Meanwhile, if you just need a metrics endpoint, you can forward whatever Prometheus metrics you'd like to long term storage in Grafana Cloud's managed Mimir service using the Influx proxy:

   https://grafana.com/docs/grafana-cloud/data-configuration/metrics/metrics-influxdb/push-from-telegraf/
5. bleetube commented on Feb 18, 2023

   

   bleetube

   on Feb 18, 2023

   Last edited by bleetube

   More actions

   Actually I realized I can do the MVP part I described without touching nodejs. I can just write some python to talk directly to postgres. I have started working on it this weekend in a separate repo:

   https://github.com/bleetube/nostream_exporter (work in progress)

   To start it is exporting one metric, the total count of events in the events table. I have select queries to add in for these metrics as well:

   • top events by kind
   • top talker users by pubkey all time
   • top talker users by pubkey recently
   • count of paid users

   I'll implement those and add in more as time permits. I'll also put together a grafana dashboard to chart them out. And if I think of any good metrics to alert on using alertmanager, I'll add those to the repo as well. Might be nice to send myself an alert if a user is spamming the relay, for instance.

   👍React with 👍1Reacted by Leo Wandersleb
6. kushagra0902 commented on Apr 9, 2026

   

   kushagra0902

   on Apr 9, 2026

   Last edited by kushagra0902

   Contributor

   More actions

   Architecture Comparison

   1. Polling architecture

   • Model: run KPI queries on a fixed interval (for example, every 5s), then push updates.
   • Read load profile: constant, predictable, independent of write traffic.
   • Write load profile: no extra write-side overhead.
   • Freshness latency: bounded by an interval; average latency is roughly half the interval plus query time.
   • Burst behaviour: stable load under write bursts, but can miss sub-interval spikes and appear stale.
   • Idle behaviour: still queries at full interval cadence even when no data changes.

   2. Event-driven architecture
      Model: database triggers emit NOTIFY events on relevant table changes; service LISTENs and recomputes only affected KPI groups probably using a windowed method to avoid recalculation.
      Read load profile: proportional to real changes, near-zero when idle.
      Write load profile: additional trigger and notification overhead on write paths.
      Freshness latency: near real-time (typically notification + recompute + websocket dispatch).
      Burst behaviour: can spike if every write causes full recompute; requires coalescing and KPI invalidation map.
      Idle behaviour: minimal work while no writes occur.

   Steps

   Phase 1: Observability Service Skeleton (PR 1)

   • Goal: Standalone TypeScript process with stable API and websocket contracts.

   • Deliverables:

   • New observability entrypoint under src.

   • Endpoints: GET /health, GET /api/v1/kpis/, websocket /api/v1/kpis/stream.

   • Typed contracts for snapshot payload and websocket messages.

   • Verification and Testing

   • Service boots locally.

   • Health and snapshot routes respond correctly.

   • Websocket handshake succeeds.

   Phase 2: Event-Driven KPI Pipeline (PR 2)

   • Goal: Bring MVP KPIs under observation.
   • Deliverables:
   • KPI collectors for events by kind, plus other, paid users, sats paid, top talkers all-time, top talkers recent.
   • Phase 2 depends on architecture decision

   Phase 3: Dashboard UI (PR 3)

   • Depends on Phase 2 API contract.
   • Goal: deliver a public-first, real-time dashboard consuming snapshot plus websocket updates.
   • Deliverables:
   • Dashboard page served by observability service.
   • KPI cards, event-kind distribution, top-talkers panels, and recent activity feed.
   • Client flow: initial snapshot fetch, websocket subscription, incremental update apply.
   • Reconnect handling, stale state indicator, and heartbeat timeout UX.
   • Responsive behaviour for desktop and mobile.

   Verification Matrix

   1. Correctness

   • KPI values match direct SQL for each metric.
   • Snapshot sequence strictly increases, and clients receive ordered updates.

   2. Freshness

   • Measure write-to-dashboard latency p50 and p95 under representative write rates.
   • Compare the polling baseline estimate to validate the improvement.

   Questions:

   • Polling or real-time-based architecture( imo, polling based architecture will be more stable in heavy traffics)
   • Use of external services to make pipeline and processing( like flinch, although I feel this will complicate unnecessarily)
7. cameri commented on Apr 9, 2026

   

   cameri

   on Apr 9, 2026

   Owner

   More actions

   what are you thoughts on using OTEL instead for pushing metrics?
8. kushagra0902 commented on Apr 9, 2026

   

   kushagra0902

   on Apr 9, 2026

   Contributor

   More actions

   what are you thoughts on using OTEL instead for pushing metrics?

   Yes will be a good architecture. Just a small follow-up, is our main intent data observability or real-time performance tracking, as I think a complete OTEL stack of collector and all the necessary modules will increase the overhead.
   If it is just data observability, then I think we can go with a normal PSQL-based architecture

   Please correct me if I am wrong at any point.
9. chappie-daemon

   assigned

   kushagra0902

   and unassigned

   cameri

   on Apr 9, 2026
10. cameri commented on Apr 9, 2026

    

    cameri

    on Apr 9, 2026

    Owner

    More actions

    @kushagra0902 let's go with PSQL based architecture

    👍React with 👍1Reacted by Kushagra Kinra
11. Ferryx349 commented on Aug 18, 2026

    

    Ferryx349

    on Aug 18, 2026

    Collaborator

    More actions

    Hi, I believe this issue is now addressed. The logging and visualization work for nostream’s admin/observability stack is being implemented over the last few months.

    What #144 asked for and what shipped

    1. Log key performance indicators
    Relay metrics are collected via OpenTelemetry and exposed to Prometheus, including:

    • WebSocket connection counts
    • Events accepted/rejected (rate and totals)
    • Process CPU and memory per worker

    2. Expose KPIs through a usable interface
    Operators can view live metrics in two ways:

    • Admin console (/admin/) → Metrics tab with live SSE snapshots from /admin/metrics
    • Grafana → nostream-overview dashboard (embedded in admin UI and available via docker-compose stack)

    3. KPIs from the original checklist

    Requested KPI	Where it’s visible
    Concurrent WebSockets	Active connections, WebSocket connections (15m)
    Events served	Events / sec, Accepted total
    System load (CPU/memory)	CPU load %, Memory by worker, CPU by worker
    Rejection visibility	Rejected / sec, Rejected total

    Implementation PRs

    Metrics foundation

    • feat(observability): add OpenTelemetry metrics bootstrap with Prometheus sidecar #653 :- OpenTelemetry metrics bootstrap, Prometheus sidecar
    • feat(observability): instrument event and websocket handlers with OTEL #661 :- Instrument event, WebSocket handlers with OTEL metrics

    Admin API and live metrics

    • feat: add admin backend foundation (login, session, health) #641 — Admin backend foundation (auth, session, health)
    • feat(admin): add Prometheus-backed /admin/metrics SSE endpoint #662 — Prometheus-backed /admin/metrics SSE endpoint

    Visualization UI

    • feat(admin): add observability dashboard UI with Grafana embeds #666 :- Observability dashboard UI with Grafana embeds

    Admin console extensions

    • feat(admin): extract shared settings-config module for CLI and admin UI #672 :- Shared settings-config module (CLI + admin UI)
    • feat(admin): add authenticated settings API endpoints #690 :- Authenticated settings API endpoints
    • feat(admin): add settings editor tab to admin dashboard #706 :- Settings editor tab in admin dashboard

    How to try it

    1. Enable admin in settings and set ADMIN_PASSWORD
    2. Run the observability stack (docker compose up) for Prometheus/Grafana
    3. Open /admin/ → Metrics tab for live KPIs, or use Grafana nostream-overview

    ---

    If this covers the scope of #144, I’d appreciate closing this issue. Happy to adjust anything if something is still missing from the original checklist.

    ❤️React with ❤️1Reacted by Ricardo Cabral
12. cameri commented on Aug 19, 2026

    

    cameri

    on Aug 19, 2026

    Owner

    More actions

    Thank you @Ferryx349 !
13. cameri

    assigned

    Ferryx349

    and unassigned

    kushagra0902

    on Aug 19, 2026
14. cameri

    closed this as completedon Aug 19, 2026