Repository navigation
feat(plugins): add external-to-internal data replication plugin (#72) - #360
Open
andreikharytonenka-lab wants to merge 1 commit into
Open
andreikharytonenka-lab wants to merge 1 commit into
andreikharytonenka-lab wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
/claim #72
Closes #72
Summary
Implements a pull-based External-to-Internal Data Replication Plugin (
DataSyncPlugin&DatabaseSyncSourceadapters) that replicates schema and data from an external database (PostgreSQL / Supabase, MySQL, Turso, Cloudflare D1, StarbaseDB, Cloudflare Hyperdrive, or Outerbase API) into StarbaseDB's internal Cloudflare Durable Object SQLite database so StarbaseDB can act as an ultra-low-latency edge replica.Architecture & Key Design Decisions
1. Decoupled
DataSyncPlugin&DatabaseSyncSourceAbstractionFollowing
@Brayden's architecture guidance,DataSyncPluginis decoupled from dialect-specific introspection and accepts aDatabaseSyncSourceimplementation (PostgresSyncSource,MySQLSyncSource,SQLiteSyncSource), while also supporting auto-selection fromdataSource.external.dialect:2. True External vs. Internal Query Routing
DatabaseSyncSource.executeExternalQuery): Executesinformation_schema/pragma_table_infoschema discovery, connection checks, and incremental batch pulls against the external data source via StarbaseDB's external query pipeline (executeExternalQuery/@outerbase/sdk/ Outerbase API / Cloudflare Hyperdrivepostgresclient).dataSource.rpc.executeQuery): Creates replicated SQLite tables, upserts sanitized rows (INSERT OR REPLACE INTO), and updates sync watermarks in the internal Durable Object SQLite database.@outerbase/sdkobject-array responses (Record<string, any>[]) and tabular responses ({ rows: any[][], columns: string[] }), and sanitizes values (Date-> ISO-8601 string,boolean->1 | 0,JSON/Array-> serialized JSON string,bigint-> safe integer/string,undefined->null) prior to writing to Durable ObjectSqlStorage.3. Replicated Table Naming & Outerbase Studio Compatibility
tmp_prefix exclusively for internal plugin metadata (tmp_data_sync_metadata) so users can distinguish Starbase system metadata from their own tables.publicschema tables (e.g.'users','person','public.products') are created in SQLite root asusers,person,products— without atmp_prefix and without literal dots in SQLite table names (which break Outerbase Studio's table viewer).'analytics.events') are created in SQLite asanalytics_events(or an explicittarget_tableoverride).beforeQueryHook:public.prefixes (SELECT * FROM public.users/SELECT * FROM "public"."users"->SELECT * FROM users) and maps custom schema prefixes (SELECT * FROM analytics.events->SELECT * FROM analytics_events) when querying the internal SQLite data source.sqlite_master,pragma_table_info,PRAGMA).4. Flexible Incremental Cursors & Tables Without
created_atoridcursor_column,timestamp_column,id_column,primary_key,batch_size).created_atorid: When a table is configured by string name (such astables: ['users', 'person']wherepersononly hasfirst_nameandlast_name),validateTableStructureandsyncTableinspect the discovered columns and gracefully fall back to full-table batch replication (SELECT * FROM public.person LIMIT 1000) instead of failing validation.5. Request Lifecycle & Monitoring Endpoints
tmp_data_sync_metadataand triggers initial/interval-based sync inside the Hono request middleware oncec.get('dataSource')is bound (usingexecutionContext.waitUntilfor non-blocking background interval syncs).GET /sync-status&GET /sync/status— Current sync configuration and per-table watermarks/errorsPOST /sync&POST /sync/trigger— Trigger on-demand sync for all tables or a specific{ "table": "users" }GET /sync-data&GET /sync/data— Preview replicated rows and metadataVerification & Testing
plugins/postgres-sync/index.test.ts(19 tests) andplugins/data-sync/index.test.ts(27 tests):PostgresSyncSource,MySQLSyncSource, andSQLiteSyncSourcecreated_at/id, customcursor_column, and tables withoutcreated_atorid(person)beforeQuerySQL schema rewriting and string literal preservationnpx prettier --check plugins/data-sync plugins/postgres-sync dist/plugins.ts).