Repository navigation
[FEATURE] Import .jsonl file with events into the events table #406
Copy link
Copy link
Closed
Labels
enhancementNew feature or requestNew feature or requestgood first issueGood for newcomersGood for newcomers
Description
Is your feature request related to a problem? Please describe.
Currently, there is no built-in way to seed a Nostream relay with existing data or migrate events from another relay. While operators can manually interact with the database, a standardized way to import NIP-01 compliant .jsonl files is missing. This makes relay migrations, disaster recovery, and data sharing cumbersome.
Describe the solution you'd like
I would like a CLI utility to import Nostr events from a .jsonl file into the Nostream events table.
Functional Requirements:
- Validation: Every event must be validated before insertion (checking signature,
idhash, and basic NIP-01 structure) to ensure database integrity. - Deduplication: The importer should gracefully handle "Duplicate Key" errors. If an event already exists in the database, it should be skipped without failing the entire batch.
- Batch Processing: To optimize performance, events should be inserted in batches (e.g., 500–1000 events per transaction) rather than one by one.
CLI Interface Example:
# Basic import
npm run import -- ./events.jsonlDescribe alternatives you've considered
- Manual SQL Inserts: Too prone to error and fails to validate Nostr signatures.
- Using a Nostr Client to "Replay" Events: Extremely slow and puts unnecessary load on the relay's WebSocket interface and rate limiters.
Additional context
Technical Considerations:
- Backpressure & Memory: Use Node.js streams (
readlineinterface or a custom transform stream) to process the file line-by-line. This ensures that even a 100GB file won't cause an Out-Of-Memory (OOM) error. - Performance vs. Availability: For very large imports, the utility should ideally provide a way to throttle the import speed or run in the background to ensure the relay remains responsive to active users.
- Worker Threads: Validation (especially signature verification) is CPU-intensive. Consider using a worker pool to validate signatures in parallel before sending batches to the database.
- Ensure any rate limiting is bypassed during import operations: interact directly with the database layer (or service layer) rather than hitting the WebSocket endpoint.
Expected Outcomes:
- A new
npm run importcommand. - Support for
.jsonlformat. - Progress Tracking: A progress bar or log showing
[Processed: 50k | Inserted: 45k | Skipped: 5k | Errors: 0]. - Documentation updated in
README.md.
Out of Scope:
- Broadcasting events to connected clients
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or requestgood first issueGood for newcomersGood for newcomers
Activity
Anshumancanrock commentedon Apr 9, 2026
@cameri could you please assign this to me ?
chappie-daemon commentedon Apr 9, 2026
Hi @Anshumancanrock! This issue is now assigned to you. Feel free to get started — good luck! 🚀
chappie-daemon commentedon Apr 10, 2026
Thanks @Anshumancanrock for the excellent implementation! 🙌 The feature is now merged and available on main.