Repository navigation
Store on PostgreSQL instead of SQLite - #19
andreimerlescu wants to merge 2 commits into
Conversation
Signed-off-by: Andrei Merlescu <andrei+github@merlescu.net>
|
Thank you, Andrei. We'll take this in: we're keeping SQLite as the default for local use and making PostgreSQL an optional backend chosen by GRAPHICLE_DATABASE_URL, building on your commit. We'll also ship the read-lock and WAL fixes for SQLite users. Could you tell us about the production host (33 readers on 6 cores)? We'd like to measure the fix against that setup. |
The host in question was an OVH VPS-3 host which has 12GiB of RAM and 6 vCPUs that each are 2.4GHz (Haswell, no TSX). The 33 readers were a baseline to measure concurrent performance and reliability in the backend swap. Good call on defaulting to SQLite. |
docs/configuration.md says how the store engine is chosen, which role Graphicle needs on PostgreSQL (its own schema, no superuser), how TLS and password files are used, and what deletion receipts promise on each engine. The deploy files go back to the SQLite defaults of 0.1.13; deploy/postgresql.graphicle.conf stays as an optional example. The changelog credits Andrei Merlescu for the PostgreSQL store (#19). Signed-off-by: VikramLex <vikramlex@karlexai.com>
Every crate and both SDKs are at 0.1.14, with a CHANGELOG entry for the work since 0.1.13: PostgreSQL 16 or newer as an optional store when GRAPHICLE_DATABASE_URL is set (Andrei Merlescu's work, #19, made optional and hardened), and SQLite stores that keep their write-ahead log bounded and serve reads while an indexer holds the write lock. SQLite stays the default, unchanged for users who do not set the variable. server.json is the MCP Registry entry of 0.1.13, the latest listed release. Signed-off-by: VikramLex <vikramlex@karlexai.com>
|
I'll get these integrations addressed. |
|
Thank you, Andrei, and thanks for the host details. Your PostgreSQL store shipped in 0.1.14 as a0fb021, with your authorship and sign-off kept, so there is nothing more to change on this branch. We kept SQLite as the default and PostgreSQL as an option when |
What this changes
Moves the store from SQLite (
reviews.sqlite3) to PostgreSQL 16+, one schemaper data folder, so the WAL can no longer grow without bound under constant
reads (the 8.3 GB WAL of the 2026-11-03 outage); existing stores move over
once with the new
graphicle import-sqlite. Commands, flags, the HTTP/MCPAPIs and their output are unchanged; it also adds a load harness
(
bench/store-load/) and stops read requests from taking the write lock.How you tested it
On a 2-vCPU, 8 GB machine against a local PostgreSQL 16, with
GRAPHICLE_DATABASE_URL="host=/tmp dbname=graphicle user=graphicle":cargo fmt --check: clean.cargo clippy --all-targets -- -D warnings: clean.cargo test(crates/graphicle): 450 unit, 1embedding_cache,27
extension, 528itpass; 2 fail:deletion::connector_deletion_returns_a_verified_receipt_covering_derived_stateexpects
secure_delete: true. PostgreSQL cannot overwrite deleted rows,so the receipt now says
false(documented in docs/configuration.md andthe CHANGELOG). The test is left as it is, for review.
fix_command::the_test_command_gets_no_model_keyfails only when run asroot; it passes as an ordinary user.
cargo deny check(licenses, bans, sources, advisories): ok.Load runs with
bench/store-load/run.py, 10 MCP readers while a separategraphicle sync PATH --project ID --jsonindexes a 1,123-file Kubernetessubset, on local and throttled disks. Commands and full tables are in
bench/store-load/RESULTS.md. During indexing:Indexing alone takes 22.9 s, against 16.3 s on SQLite. The production
host's 33 readers on 6 cores have not been measured after the fixes.
Checklist
git commit -s), as theDeveloper Certificate of Origin asks
(CONTRIBUTING.md).
Cargo.lockchanged, I rancargo deny checkin that crate andrefreshed its licence page, as CONTRIBUTING.md describes.
credited in the crate's
NOTICE.md.evidence in
docs/benchmarks.md.