Skip to content

Commit ebfbf9d

Browse files
marcobambinimarcobambini
committed
Added PG documentation
1 parent ac334ae commit ebfbf9d

File tree

6 files changed

+1040
-0
lines changed

6 files changed

+1040
-0
lines changed

‎docs/postgresql/CLIENT.md‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# SQLite Sync
2+
3+
**SQLite Sync** is a multi-platform extension that brings a true **local-first experience** to your applications with minimal effort. It extends standard SQLite tables with built-in support for offline work and automatic synchronization, allowing multiple devices to operate independently — even without a network connection — while seamlessly staying in sync.
4+
5+
With SQLite Sync, developers can build **distributed, collaborative applications** while continuing to rely on the **simplicity, reliability, and performance of SQLite**.
6+
7+
Under the hood, SQLite Sync uses advanced **CRDT (Conflict-free Replicated Data Type)** algorithms and data structures designed specifically for **collaborative, distributed systems**:
8+
9+
- Devices can update data independently, even without a network connection.
10+
- When they reconnect, all changes are **merged automatically and without conflicts**.
11+
- **No data loss. No overwrites. No manual conflict resolution.**
12+
13+
---
14+
15+
## IMPORTANT
16+
17+
- Make sure to use version **0.9.96 or newer**
18+
(verify with `SELECT cloudsync_version();`)
19+
20+
- Until v0.9.96 is released upstream, always use the development fork:
21+
https://github.com/sqliteai/sqlite-sync-dev
22+
and **not** the original repository:
23+
https://github.com/sqliteai/sqlite-sync
24+
25+
- Updated example apps are available at:
26+
https://github.com/sqliteai/sqlite-sync-dev/tree/main/examples
27+
- sport-tracker-app (WASM), see SPORT_APP_README_SUPABASE.md
28+
- to-do-app (React)
29+
- React-Native (Expo): https://github.com/sqliteai/sqlite-sync-react-native
30+
- Remaining demos will be updated in the next days
31+
32+
---
33+
34+
## Conversion Between SQLite and PostgreSQL Tables
35+
36+
- In this version, make sure to **manually create** the same tables in the PostgreSQL database as used in the SQLite client.
37+
- Follow the Database Schema Recommendations:
38+
https://github.com/sqliteai/sqlite-sync-dev?tab=readme-ov-file#database-schema-recommendations
39+
40+
---
41+
42+
## Pre-built Binaries
43+
44+
Download the appropriate pre-built binary for your platform from the official [Releases](https://github.com/sqliteai/sqlite-sync-dev/releases) page:
45+
46+
- Linux: x86 and ARM
47+
- macOS: x86 and ARM
48+
- Windows: x86
49+
- Android
50+
- iOS
51+
52+
53+
54+
## Loading the Extension
55+
56+
```
57+
-- In SQLite CLI
58+
.load ./cloudsync
59+
60+
-- In SQL
61+
SELECT load_extension('./cloudsync');
62+
```
63+
64+
65+
66+
## WASM Version
67+
68+
```
69+
npm i sqlite-wasm@dev
70+
```
71+
72+
Then follow the instructions available from https://www.npmjs.com/package/@sqliteai/sqlite-wasm
73+
74+
75+
76+
## Swift Package
77+
78+
You can [add this repository as a package dependency to your Swift project](https://developer.apple.com/documentation/xcode/adding-package-dependencies-to-your-app#Add-a-package-dependency). After adding the package, you'll need to set up SQLite with extension loading by following steps 4 and 5 of [this guide](https://github.com/sqliteai/sqlite-extensions-guide/blob/main/platforms/ios.md#4-set-up-sqlite-with-extension-loading).
79+
80+
81+
82+
## Android Package
83+
84+
Add the [following](https://central.sonatype.com/artifact/ai.sqlite/sync.dev) to your Gradle dependencies:
85+
86+
```
87+
implementation 'ai.sqlite:sync.dev:0.9.92'
88+
```
89+
90+
91+
92+
## Expo
93+
94+
Install the Expo package:
95+
96+
```
97+
npm install @sqliteai/sqlite-sync-expo-dev
98+
```
99+
100+
Then follow the instructions from:
101+
102+
https://www.npmjs.com/package/@sqliteai/sqlite-sync-expo-dev
103+
104+
105+
106+
## React/Node
107+
108+
```js
109+
npm i better-sqlite3
110+
npm i @sqliteai/sqlite-sync-dev
111+
112+
echo "import { getExtensionPath } from '@sqliteai/sqlite-sync-dev';
113+
import Database from 'better-sqlite3';
114+
115+
const db = new Database(':memory:');
116+
db.loadExtension(getExtensionPath());
117+
118+
// Ready to use
119+
const version = db.prepare('SELECT cloudsync_version()').pluck().get();
120+
console.log('Sync extension version:', version);" >> index.js
121+
122+
node index.js
123+
```
124+
125+
---
126+
127+
## Naming Clarification
128+
129+
- **sqlite-sync** → Client-side SQLite extension
130+
- **cloudsync** → Synchronization server microservice
131+
- **postgres-sync** → PostgreSQL extension
132+
133+
The sqlite-sync extension is loaded in SQLite under the extension name:
134+
`cloudsync`

‎docs/postgresql/CLOUDSYNC.md‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
# Demo Deployment
2+
3+
For the current demo, a single cloudsync node is deployed in **Europe** on Fly.io.
4+
5+
If testing from other regions, latency will reflect this single-node deployment.
6+
A production deployment would use **geographically distributed nodes with regional routing** for global coverage.
7+
8+
---
9+
10+
### Fly.io
11+
12+
Project Name: **cloudsync-staging**
13+
Fly.io App: https://fly.io/apps/cloudsync-staging
14+
CloudSync Server URL: https://cloudsync-staging.fly.dev/
15+
Logs: https://fly.io/apps/cloudsync-staging/monitoring
16+
17+
> Note: This is a **demo-only environment**, not intended for production use.
18+
19+
---
20+
21+
## Environment Variables
22+
23+
Edit in the Fly.io **Secrets** section:
24+
https://fly.io/apps/cloudsync-staging/secrets
25+
26+
After editing, the machine restarts automatically.
27+
28+
The server is currently configured to point to a demo Supabase project (https://supabase.com/dashboard/project/ajgnsrqbwmnhytqyesyr).
29+
30+
Environment variables:
31+
32+
- `CLOUDSYNC_JOBS_DATABASE_CONNECTION_STRING` — database for jobs table
33+
- `CLOUDSYNC_ARTIFACT_STORE_CONNECTION_STRING` — database for sync artifacts
34+
- `CLOUDSYNC_METRICS_DB_CONNECTION_STRING` — database for metrics
35+
- `CLOUDSYNC_SERVICE_PROJECT_ID` — project ID for service user
36+
- `CLOUDSYNC_SERVICE_DATABASE_CONNECTION_STRING` — service user DB connection
37+
38+
---
39+
40+
## Tables
41+
42+
- **cloudsync_jobs** — queue of asynchronous jobs
43+
- `check`: generate blob of client changes
44+
- `apply`: apply client changes
45+
- `notify`: send Expo push notifications
46+
47+
- **cloudsync_artifacts** — blobs ready for client download
48+
- **cloudsync_metrics** — collected metrics
49+
- **cloudsync_push_tokens** — Expo push tokens
50+
51+
---
52+
53+
## Metrics
54+
55+
CloudSync integrates a simple metrics collector (authenticated via user JWT).
56+
57+
To visualize metrics, import `grafana-dashboard.json` into Grafana and configure your PostgreSQL database as a data source.
58+
59+
Alternatively, call the API directly:
60+
61+
```bash
62+
curl --request GET --url 'https://cloudsync-staging.fly.dev/v2/cloudsync/metrics?from=<ISO_START>&to=<ISO_END>&projectId=<PROJECT>&database=<DB>&siteId=<SITE>&action=check' --header 'Authorization: Bearer <JWT>'
63+
```
64+
65+
Filters:
66+
67+
- `from`
68+
- `to`
69+
- `projectId`
70+
- `database`
71+
- `siteId`
72+
- `action` (`check` / `apply`)

‎docs/postgresql/README.md‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
# Architecture Overview
2+
3+
The **SQLite AI offline-sync solution** consists of three main components:
4+
* **sqlite-sync**: Native client-side SQLite extension
5+
* **cloud-sync**: Synchronization microservice
6+
* **postgres-sync**: Native PostgreSQL extension
7+
8+
Together, these components provide a complete, production-grade **offline-first synchronization stack** for SQLite and PostgreSQL.
9+
10+
# sqlite-sync
11+
12+
**sqlite-sync** is a native SQLite extension that must be installed and loaded on all client devices.
13+
We provide prebuilt binaries for:
14+
* Desktop and mobile platforms
15+
* WebAssembly (WASM)
16+
* Popular frameworks including React, Expo, npm, and more
17+
18+
**Note:** The latest version (v0.9.96) is not yet available in the official sqlite-sync repository.
 Please use our development fork instead:~[https://github.com/sqliteai/sqlite-sync-dev](https://github.com/sqliteai/sqlite-sync-dev)~
19+
20+
### Architecture Refactoring
21+
The extension has been refactored to support both **SQLite** and **PostgreSQL** backends.
22+
* All database-specific native calls have been isolated in database.h
23+
* Each database engine implements its own engine-dependent layer
24+
* The core **CRDT logic** is fully shared across engines
25+
26+
This modular design improves **portability**, **maintainability**, and **cross-database consistency**.
27+
### Testing & Reliability
28+
* Shared CRDT and SQLite components include extensive unit tests
29+
* Code coverage exceeds **90%**
30+
* PostgreSQL-specific code has its own dedicated test suite
31+
32+
Key Features
33+
* Deep integration with SQLite — the default database for Edge applications
34+
* Built-in network layer exposed as ordinary SQLite functions
35+
* Cross-platform, language-agnostic payload format
36+
* Works seamlessly in any framework or programming language
37+
38+
Unlike other offline-sync solutions, **sqlite-sync embeds networking directly inside SQLite**, eliminating external sync SDKs.
39+
40+
### Supported CRDTs
41+
Currently implemented CRDT algorithms:
42+
* **Last-Write-Wins (LWW)**
43+
* **Grow-Only Set (G-Set)**
44+
45+
Additional CRDTs can be implemented if needed, though LWW covers most real-world use cases.
46+
47+
48+
49+
# cloud-sync
50+
51+
**cloudsync** is a lightweight, stateless microservice responsible for synchronizing clients with central servers.
52+
### Responsibilities
53+
* Synchronizes clients with:
54+
* **SQLiteCloud servers**
55+
* **PostgreSQL servers**
56+
* Manages upload and download of CRDT payloads
57+
* Stores payloads via **AWS S3**
58+
* Collects operational metrics (connected devices, sync volume, traffic, etc.)
59+
* Exposes a complete **REST API**
60+
61+
⠀
62+
63+
Technology Stack
64+
65+
* Written in **Go**
66+
* Built on the high-performance **Gin Web Framework**
67+
* Fully **multitenant**
68+
* Connects to multiple DBMS backends
69+
* Stateless architecture enables horizontal scaling simply by adding nodes
70+
* Serialized job queue ensures **no job loss**, even after restarts
71+
72+
⠀
73+
74+
Observability
75+
76+
* Metrics dashboard available in grafana-dashboard.json
77+
78+
* Additional logs available via the Fly.io monitoring dashboard
79+
80+
81+
82+
Demo Deployment
83+
For the current demo, a single cloudsync node is deployed in **Europe** on Fly.io.
84+
If testing from other regions, latency will reflect this single-node deployment.
 A production deployment would use **geographically distributed nodes with regional routing** for global coverage.
85+
86+
87+
88+
# postgres-sync
89+
90+
**postgres-sync** is a native PostgreSQL extension derived from sqlite-sync.
91+
### Features
92+
* Implements the same CRDT algorithms available in sqlite-sync
93+
* Applies CRDT logic to:
94+
* Changes coming from synchronized clients
95+
* Changes made directly in PostgreSQL (CLI, Drizzle, dashboards, etc.)
96+
97+
This ensures **full bidirectional consistency**, regardless of where changes originate.
98+
99+
### Schema Handling
100+
SQLite does not support schemas, while PostgreSQL does.
To bridge this difference, postgres-sync introduces a mechanism to:
101+
* Associate each synchronized table with a specific PostgreSQL schema
102+
* Allow different schemas per table
103+
This preserves PostgreSQL-native organization while maintaining SQLite compatibility.
104+
105+
106+
107+
# Current Limitations
108+
109+
The PostgreSQL integration is actively evolving. Current limitations include:
110+
* **User Impersonation**
: The microservice currently applies server changes using the Supabase Admin user. 
In the next version, changes will be applied under the identity associated with the client’s JWT.
111+
* **Table Creation**
: Tables must currently be created manually in PostgreSQL before synchronization.
 We are implementing automatic translation of SQLite CREATE TABLE statements to PostgreSQL syntax.
112+
* **Row-Level Security**: RLS is fully implemented for SQLiteCloud servers.
PostgreSQL RLS integration is in progress and will be included in the final release.
113+
* **Beta Status**
: While extensively tested, the PostgreSQL sync stack should currently be considered **beta software**. Please report any issues, we are committed to resolving them quickly.

‎docs/postgresql/SPORT_APP_README_SUPABASE.md‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# Sport Tracker app with SQLite Sync 🚵
2+
3+
A Vite/React demonstration app showcasing [**SQLite Sync (Dev)**](https://github.com/sqliteai/sqlite-sync-dev) implementation for **offline-first** data synchronization across multiple devices. This example illustrates how to integrate SQLite AI's sync capabilities into modern web applications with proper authentication via [Access Token](https://docs.sqlitecloud.io/docs/access-tokens) and [Row-Level Security (RLS)](https://docs.sqlitecloud.io/docs/rls).
4+
5+
> This app uses the packed WASM version of SQLite with the [SQLite Sync extension enabled](https://www.npmjs.com/package/@sqliteai/sqlite-wasm).
6+
7+
## Setup Instructions
8+
9+
### 1. Prerequisites
10+
- Node.js 20.x or \>=22.12.0
11+
12+
### 2. Database Setup
13+
1. Create database
14+
2. Execute the schema with [sport-tracker-schema-postgres.sql](sport-tracker-schema-postgres.sql).
15+
3. Enable CloudSync for all tables on the remote database with:
16+
```sql
17+
CREATE EXTENSION IF NOT EXISTS cloudsync;
18+
SELECT cloudsync_init('users_sport');
19+
SELECT cloudsync_init('workouts');
20+
SELECT cloudsync_init('activities');
21+
```
22+
23+
### 3. Environment Configuration
24+
25+
Rename the `.env.example` into `.env` and fill with your values.
26+
27+
- `VITE_SQLITECLOUD_CONNECTION_STRING`: the url to the CloudSync server: https://cloudsync-staging.fly.dev/<database>
28+
- `VITE_SQLITECLOUD_DATABASE`: remote database name.
29+
- `VITE_SQLITECLOUD_API_KEY`: a valid user's JWT token. Refresh it when it expires.
30+
- `VITE_SQLITECLOUD_API_URL`: Supabase project API URL.
31+
32+
### 4. Installation & Run
33+
34+
```bash
35+
npm install
36+
npm run dev
37+
```
38+
39+
### Demo
40+
41+
Continue reading on the official [README](https://github.com/sqliteai/sqlite-sync-dev/blob/main/examples/sport-tracker-app/README.md#demo-use-case-multi-user-sync-scenario).

0 commit comments

Comments
 (0)