Repository navigation
Description
Problem
The Workbench shows an Amazon S3 card under Remote data sources, and the card only links to a GitHub search for label:connector:s3. There is no way to bind a bucket prefix and compile those objects into the knowledge base.
This is an ingest source. The knowledge base stays a directory on disk. It is not a request to store the wiki itself in S3. That storage-engine idea is #61 (S3-compatible read/write of the knowledge base, including MinIO) and stays separate.
Approach
openkb source copies selected objects into raw/ and compiles them with the existing add_single_file path. Query and chat keep reading the compiled wiki.
A binding is one entry in .openkb/config.yaml:
connectors:
- name: papers
provider: s3
region: us-east-1
bucket: acme-docs
prefix: papers/ # empty prefix is the bucket rootCredentials are OPENKB_S3_ACCESS_KEY_ID and OPENKB_S3_SECRET_ACCESS_KEY. Lookup order is the CLI order: process environment, then <kb>/.env, then ~/.config/openkb/.env. Sign-in and sync do not call resolve_credential_bundle (that helper uses the opposite order). boto3 is an exact-pinned s3 extra. pip install openkb and pip install "openkb[web]" do not gain it.
openkb source add s3 --name NAME --region REGION --bucket BUCKET [--prefix PREFIX]
openkb source list
openkb source remove NAME
openkb source sync [NAME] [--non-interactive]
Sync downloads extensions already accepted by openkb add (.pdf, .md, .markdown, .docx, .pptx, .xlsx, .xls, .html, .htm, .txt, .csv). The registry identity of an object is s3://{bucket}/{key}. The same key and the same bytes are skipped. The same key and new bytes recompile that wiki page. A new key is a new document. Removing a binding, or deleting an object in the bucket, leaves compiled pages and files already in raw/ in place.
Last sync status is written to .openkb/connector-state.json with atomic_write_json, echoing provider and location. Exclusive kb_lock on .openkb/ingest.lock is held only around that write and around the .env update, for at most 30 seconds. Downloads and add_single_file run outside the lock. add_single_file is called on a thread that does not already hold the lock.
This issue also lands the shared list/download interface, the credential resolver, the state file, and the source command. Google Drive (#275) and OneDrive (#276) are separate issues and separate pull requests that implement the same interface.
Please add the label connector:s3. The Workbench card searches is:issue label:connector:s3.
Pull request rules
Taken from what has actually been required before a merge (CI, and review on the REST API / mutation pull requests):
- The pull request closes this issue. No product code before the approach here is accepted.
- The API calls the same functions as the CLI. Lock-holding work runs in a worker thread (
run_in_threadpool).kb_ingest_lockreentrancy is per-thread, so that work must not run on the event-loop thread. - Wiki bytes go through
add_single_file, which already journals the write. Sync does not writewiki/itself. - New modules stay under 800 lines.
cli.pyandapi.pyonly register the command and the router. - New dependencies are exact pins.
uv lockis committed. The SDK is imported lazily souv sync --locked --extra dev --extra apistill collects tests. If a test needs the extra,.github/workflows/ci.ymlis updated in the same pull request. - Tests cover the behavior and would fail without it: credential precedence across the three layers, hash skip, in-place recompile, per-file failure, lock not held across
add_single_file, and a fake S3 list/download. No live AWS calls. ruff check,ruff format --check,mypy openkb, andpytestare green. The Workbench change includes screenshots of the S3 card (vote state is gone for S3 only; Drive and OneDrive stay vote links) and keeps English and Chinese strings. The builtweb/bundle is not committed.- No unrelated edits (config defaults, BOM, formatting of untouched files).
Out of scope
MinIO, R2, and custom endpoints. A folder browser. Deleting wiki pages when an object disappears. A background daemon. Reading S3 at query time.
Activity