Skip to content

Remote data source: ingest an Amazon S3 prefix into the local knowledge base #274

Open

Description

@kalyan-fireweave

Problem

The Workbench shows an Amazon S3 card under Remote data sources, and the card only links to a GitHub search for label:connector:s3. There is no way to bind a bucket prefix and compile those objects into the knowledge base.

This is an ingest source. The knowledge base stays a directory on disk. It is not a request to store the wiki itself in S3. That storage-engine idea is #61 (S3-compatible read/write of the knowledge base, including MinIO) and stays separate.

Approach

openkb source copies selected objects into raw/ and compiles them with the existing add_single_file path. Query and chat keep reading the compiled wiki.

A binding is one entry in .openkb/config.yaml:

connectors:
  - name: papers
    provider: s3
    region: us-east-1
    bucket: acme-docs
    prefix: papers/   # empty prefix is the bucket root

Credentials are OPENKB_S3_ACCESS_KEY_ID and OPENKB_S3_SECRET_ACCESS_KEY. Lookup order is the CLI order: process environment, then <kb>/.env, then ~/.config/openkb/.env. Sign-in and sync do not call resolve_credential_bundle (that helper uses the opposite order). boto3 is an exact-pinned s3 extra. pip install openkb and pip install "openkb[web]" do not gain it.

openkb source add s3 --name NAME --region REGION --bucket BUCKET [--prefix PREFIX]
openkb source list
openkb source remove NAME
openkb source sync [NAME] [--non-interactive]

Sync downloads extensions already accepted by openkb add (.pdf, .md, .markdown, .docx, .pptx, .xlsx, .xls, .html, .htm, .txt, .csv). The registry identity of an object is s3://{bucket}/{key}. The same key and the same bytes are skipped. The same key and new bytes recompile that wiki page. A new key is a new document. Removing a binding, or deleting an object in the bucket, leaves compiled pages and files already in raw/ in place.

Last sync status is written to .openkb/connector-state.json with atomic_write_json, echoing provider and location. Exclusive kb_lock on .openkb/ingest.lock is held only around that write and around the .env update, for at most 30 seconds. Downloads and add_single_file run outside the lock. add_single_file is called on a thread that does not already hold the lock.

This issue also lands the shared list/download interface, the credential resolver, the state file, and the source command. Google Drive (#275) and OneDrive (#276) are separate issues and separate pull requests that implement the same interface.

Please add the label connector:s3. The Workbench card searches is:issue label:connector:s3.

Pull request rules

Taken from what has actually been required before a merge (CI, and review on the REST API / mutation pull requests):

  • The pull request closes this issue. No product code before the approach here is accepted.
  • The API calls the same functions as the CLI. Lock-holding work runs in a worker thread (run_in_threadpool). kb_ingest_lock reentrancy is per-thread, so that work must not run on the event-loop thread.
  • Wiki bytes go through add_single_file, which already journals the write. Sync does not write wiki/ itself.
  • New modules stay under 800 lines. cli.py and api.py only register the command and the router.
  • New dependencies are exact pins. uv lock is committed. The SDK is imported lazily so uv sync --locked --extra dev --extra api still collects tests. If a test needs the extra, .github/workflows/ci.yml is updated in the same pull request.
  • Tests cover the behavior and would fail without it: credential precedence across the three layers, hash skip, in-place recompile, per-file failure, lock not held across add_single_file, and a fake S3 list/download. No live AWS calls.
  • ruff check, ruff format --check, mypy openkb, and pytest are green. The Workbench change includes screenshots of the S3 card (vote state is gone for S3 only; Drive and OneDrive stay vote links) and keeps English and Chinese strings. The built web/ bundle is not committed.
  • No unrelated edits (config defaults, BOM, formatting of untouched files).

Out of scope

MinIO, R2, and custom endpoints. A folder browser. Deleting wiki pages when an object disappears. A background daemon. Reading S3 at query time.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions