Skip to content

feat: add postgres replication plugin for external data sync#133

Open
Lumantis wants to merge 1 commit intoouterbase:mainfrom
Lumantis:bounty/72-postgres-replication-plugin
Open

feat: add postgres replication plugin for external data sync#133
Lumantis wants to merge 1 commit intoouterbase:mainfrom
Lumantis:bounty/72-postgres-replication-plugin

Conversation

@Lumantis
Copy link
Copy Markdown

Add PostgreSQL Replication Plugin for External Data Sync

Summary

Adds a new postgres-replication plugin that enables one-way data synchronization from an external PostgreSQL database into StarbaseDB's local SQLite storage. This is useful for read-heavy workloads where data originates in PostgreSQL but needs to be served from the edge with low latency.

What it does

  • Full sync: Reads all rows from a configured PostgreSQL source table and upserts them into a local SQLite target table.
  • Incremental sync: Uses a cursor (based on primary key) to fetch only rows added since the last sync, reducing transfer volume.
  • Config management via REST API: Admin-only endpoints under /replication to add, list, pause, resume, and delete replication configs.
  • Sync logs: Every sync operation is logged with status, row count, and error details for observability.
  • Event callbacks: Other plugins can subscribe to sync events via onEvent(), following the same pattern as the CDC plugin.

API Endpoints

Method Path Description
GET /replication List all replication configs
POST /replication Add a new replication config
POST /replication/:id/sync Trigger manual sync (full or incremental)
PATCH /replication/:id Pause or resume a config
DELETE /replication/:id Remove a config
GET /replication/logs View recent sync logs

Files changed

  • plugins/postgres-replication/index.ts -- Main plugin implementation
  • plugins/postgres-replication/index.test.ts -- Unit tests (vitest)
  • plugins/postgres-replication/meta.json -- Plugin metadata
  • dist/plugins.ts -- Added export for the new plugin

Usage example

import { PostgresReplicationPlugin } from 'starbasedb/dist/plugins'

const replicationPlugin = new PostgresReplicationPlugin()

// Subscribe to sync events
replicationPlugin.onEvent((event) => {
    console.log(`Synced ${event.rowsSynced} rows from ${event.sourceTable}`)
})

Design decisions

  • Follows the existing plugin architecture (extends StarbasePlugin, uses register() for routes, onEvent() for callbacks).
  • Uses INSERT OR REPLACE for upserts to handle both new and updated rows.
  • Requires role: 'admin' for all endpoints, consistent with the CDC plugin.
  • Stores config and logs in tmp_ prefixed tables following the project convention (see cron plugin).
  • Incremental sync falls back to full sync when no cursor exists yet.

Testing

All tests use vitest with mocked dataSource.rpc.executeQuery. Coverage includes config CRUD, full sync, incremental sync, error handling, event callbacks, and edge cases (empty tables, missing configs, wrong dialect).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant