diff --git a/README.md b/README.md index d5ce7e1..e66d0f6 100644 --- a/README.md +++ b/README.md @@ -19,6 +19,7 @@ that help readers decide what to do next without hunting. | Evaluate quantum scaling | [research/QUANTUM_SCALING_STRATEGY.md](research/QUANTUM_SCALING_STRATEGY.md) | [technical/GRADIENT_OPTIMIZATION_POSTMORTEM.md](technical/GRADIENT_OPTIMIZATION_POSTMORTEM.md) | | Audit implementation decisions | [technical/IMPLEMENTATION_NOTES.md](technical/IMPLEMENTATION_NOTES.md) | [technical/BENCHMARK.md](technical/BENCHMARK.md) | | Deploy or host the platform | [LIGHTSAIL-DEPLOYMENT.md](LIGHTSAIL-DEPLOYMENT.md) | [START_HERE.md](START_HERE.md) | +| Find documents moved from the application repository | [repository/README.md](repository/README.md) | Original-path migration index | | Review future direction | [FUTURE_ROADMAP.md](FUTURE_ROADMAP.md) | [future-roadmap/README.md](future-roadmap/README.md) | ## What is inside @@ -30,6 +31,8 @@ that help readers decide what to do next without hunting. - [design.md](design.md) - design rationale, cost model, and tradeoffs. - [requirements.md](requirements.md) - functional and non-functional requirements. - [AGENT_PROGRESS.md](AGENT_PROGRESS.md) - implementation and assistant progress notes. +- [repository/README.md](repository/README.md) - application repository documents, retaining their original folder structure. +- [superpowers/PRESENTATION_README.md](superpowers/PRESENTATION_README.md) - demo scripts, presentation material, and links to related design specs. ### Research @@ -87,6 +90,7 @@ linking between documents in this repository. | |-- QAOA_OPTIMIZATION_RESEARCH.md | `-- BENCHMARK.md |-- future-roadmap/ +|-- repository/ `-- superpowers/ ``` diff --git a/START_HERE.md b/START_HERE.md index 93d51ce..90184a2 100644 --- a/START_HERE.md +++ b/START_HERE.md @@ -37,7 +37,7 @@ Why we pivoted from speed tricks to scaling demonstration. Crossover predictions Honest analysis: parameter-shift gradients were 2–3× slower. Root cause: 8× evaluation overhead. ### I want to deploy the platform -**→** [`../DEPLOYMENT-MANUAL.md`](../DEPLOYMENT-MANUAL.md) then [`LIGHTSAIL-DEPLOYMENT.md`](LIGHTSAIL-DEPLOYMENT.md) +**→** [`repository/DEPLOYMENT-MANUAL.md`](repository/DEPLOYMENT-MANUAL.md) then [`LIGHTSAIL-DEPLOYMENT.md`](LIGHTSAIL-DEPLOYMENT.md) ### I want to understand finance theory and the quantum use case **→** [`FINANCIAL_MODELING_FOUNDATIONS.md`](FINANCIAL_MODELING_FOUNDATIONS.md) @@ -133,7 +133,7 @@ docs/ | Understand quantum advantage | `research/QUANTUM_SCALING_STRATEGY.md` → `research/MATHEMATICAL_APPENDIX.md` §F | | Understand system internals | `ARCHITECTURE.md` → `design.md` | | Make product/advisor decisions | `research/QUANTUM_SCALING_STRATEGY.md` → `research/ALTERNATIVE_QUANTUM_FINANCE_PROBLEMS.md` | -| Deploy to production | `../DEPLOYMENT-MANUAL.md` → `LIGHTSAIL-DEPLOYMENT.md` | +| Deploy to production | `repository/DEPLOYMENT-MANUAL.md` → `LIGHTSAIL-DEPLOYMENT.md` | | Take paid x402 requests on Algorand | `algorand-x402/algorand-x402-runbook.md` → `algorand-x402/algorand-x402-design.md` | | Run or host Toju agent storage | `toju/toju-render-testnet-guide.md` → `toju/toju-devcon-demo.md` | diff --git a/guides/audit-cleanup-migration.md b/guides/audit-cleanup-migration.md new file mode 100644 index 0000000..07664e4 --- /dev/null +++ b/guides/audit-cleanup-migration.md @@ -0,0 +1,163 @@ +# Audit cleanup: migration and release notes + +This change set makes setup consistent across source checkouts, Python packages, +and Docker, and moves private storage operations behind authenticated server +routes. Existing installations need configuration and client updates before +upgrading. No database or account migration runs automatically. + +Use the [environment guide](environment.md) for complete settings and the +[validation record](cleanup-validation.md) for checks performed and their limits. +`BREAKING` below means a confirmed change to existing behavior. Possible +regressions and unverified integrations are listed separately. + +## Confirmed breaking changes + +### BREAKING: storage credentials and permanent deletion + +Rename these frontend settings and keep the new values on the server: + +| Old variable | Replacement | +| --- | --- | +| `NEXT_PUBLIC_PINATA_JWT` | `PINATA_JWT` | +| `NEXT_PUBLIC_LIGHTHOUSE_KEY` | `LIGHTHOUSE_API_KEY` | + +Pinata also accepts the server-only pair `PINATA_API_KEY` and `PINATA_API_SECRET`. +Remove the public names and rebuild the frontend. Rotate credentials that were +already shipped in browser bundles; renaming does not revoke them. Backend +research uploads need their own Pinata settings; `WEB3_STORAGE_API_KEY` is unused. + +Vault uploads now require a signed-in session and run through server routes. +Permanent deletion requests (`hardDelete: true`) return HTTP 409 for an owned pin. +Normal removal changes only that user's vault tracking record. Use the provider +dashboard for permanent deletion after checking other users' dependencies on the +content. Existing `nft.storage` records remain supported as a Pinata alias. + +### BREAKING: unified backend MongoDB routing + +Agent, marketplace, proposal, and notification services now honor +`QB2_MONGODB_TARGET` and its selected `QB2_MONGODB_LOCAL_URI` or +`QB2_MONGODB_REMOTE_URI`, like the core API. They no longer always read the local +URI or independently default to `qds`. The shared default database is +`quantum_backend_v2`. Set `QB2_MONGODB_DATABASE` explicitly to your existing `qds` +or custom name before upgrading. If services previously used different servers +or databases, reconcile that split deliberately; changing settings does not move +records, wallets, or accounts. + +Frontend auth remains separate: preserve `MONGODB_URI` and the effective auth +database. `MONGODB_DB_NAME` takes precedence over the newly supported legacy +fallback `MONGODB_DATABASE`; the default remains `quantum_platform`. A previously +ignored fallback value can now select a different database if the canonical name +is absent, so set the intended name explicitly. + +### BREAKING: authenticated run, chat, and pharma access + +Frontend run, legacy agent-chat, and pharma proxies now send the signed-in user's +identity. Legacy backend chat and pharma routes enforce authentication and +ownership; chat no longer uses a shared `default_user`. Anonymous callers and +callers reading another user's records must update their integration. Legacy +chat WebSockets also require authentication and session ownership. + +Set the same server-only `AGENT_AUTH_BRIDGE_SECRET` of at least 32 characters on +both servers, and deploy matching frontend/backend code. Preserve the separate +`BETTER_AUTH_SECRET`. Old shared or unowned records are not automatically assigned +to a user. Keep `QB2_AUTH_REQUIRED=true` on shared deployments; the local bypass +and trial flag are not migration solutions. + +### BREAKING: dependency extras and direct resource paths + +The `dev` extra is removed. Use `uv sync --locked`; development dependencies are +installed through the default dependency group. Use `--no-dev` for production. +Chemistry dependencies now require `--extra pharma`; without them, molecular +submissions return HTTP 503 with installation guidance. Keep the extra on later +`uv run`/`uv sync` commands, or launch the installed environment with `make run`. +Docker chemistry builds require `--build-arg INSTALL_PHARMA=true`. + +Python 3.12 or 3.13 is required; this was already the package requirement, even +though older README text advertised 3.11. Use the updated lockfile instead of +mixing pip changes into the uv environment. Broad prerelease allowance was +removed; required prerelease packages are pinned explicitly. + +Prompt files moved from `backend/prompts/*.txt` to +`backend/src/quantum_backend_v2/prompts/`. Update scripts that read the old paths; +Python consumers can use `importlib.resources.files("quantum_backend_v2")`. +The unused `models.agent` module and browser pinning-provider implementations +were removed, so direct imports of those internal modules need review. Wheels +now include the required sibling packages, prompt assets, and Toju worker sources. +Toju still needs Node 22 and `npm ci` in its worker directory; Docker includes them. + +### BREAKING: deployment and maintenance behavior + +- Native startup consistently reads `backend/.env`; installed wheels read `.env` + in the launch directory. Exported variables win. Compose instead requires root + `.env` and passes it to the backend through `env_file`. Preserve effective values + when consolidating frontend settings into `.env.local`. +- Compose fixes the backend's internal bind to `0.0.0.0:8080`, publishes peer port + 4011 from the backend, and fixes persistent runtime paths. Earlier env overrides + of those fields no longer apply. Set `API_DOMAIN` explicitly for deployment; + its default is now local. Frontend and MongoDB are still hosted separately. +- Kubo no longer starts with plain `docker compose up`; select `--profile storage` + or `--profile monitoring`. The default embedded worker count drops from six to + one. Set `QB2_LIBP2P_DEV_SERVICE_PEER_COUNT` if your setup needs the previous count. +- Backend and Caddy readiness checks now require MongoDB. A running Python process + alone no longer makes the deployment healthy. +- `make run-clean` refuses destructive cleanup. Use `make run`; back up and review + any deliberate peerstore or log removal separately. Startup also refuses busy + ports instead of killing their owners. +- Restart recovery marks interrupted jobs attributed to this coordinator as failed + and reconciles their reservations/credit holds. It does not resume computation + or change unowned legacy jobs. Use a unique, stable `QB2_PEER_ID` per coordinator. + +### BREAKING: documentation paths and automatic guidance discovery + +Forty-one Markdown documents moved into the `documentations` submodule; see the +[old-to-new path index](../repository/README.md). Initialize it with +`git submodule update --init documentations` after cloning or updating the parent +checkout. Update scripts and links that use the old paths. Frontend `AGENTS.md` +and `CLAUDE.md` are no longer discovered beside source files: load their +[new locations](../repository/README.md#frontend-guidance) explicitly in tools +that previously found them automatically. + +## Potential regressions and validation limits + +- Dependency and packaging changes may expose platform-specific chemistry or + external-consumer issues. Docker execution, native chemistry, real browser + IndexedDB, cloud storage, and wallet/payment integrations were not validated + end to end during this cleanup. +- Provider upload and MongoDB tracking writes are separate operations. If the + upload succeeds and the database write fails, provider-dashboard cleanup may be + needed. Pharma state remains process-local and is lost on restart. +- Source lint errors and the landing architecture test failure were fixed during + release checks. Existing lint warnings and optional/live test skips remain; see + the [validation record](cleanup-validation.md) for results and coverage limits. + +A separate deployed sign-in outage was traced to database configuration and +network access. Its configuration-only redeployment did not include this change +set. That incident is not evidence of a regression from these changes, nor proof +that merging them repairs another deployment's database access. Auth error +handling was tested separately; full production account creation and sign-in +were not completed during the recorded checks. + +## Rollout and rollback + +1. Record the application commit, documentation submodule commit, deployed + artifacts, and effective configuration privately. Back up MongoDB and persistent + runtime volumes. Preserve database names, auth secrets, wallet encryption keys, + and coordinator IDs; do not replace existing env files with blank examples. +2. Apply the configuration changes above, install locked dependencies with the + extras you use, and rebuild the frontend. Test a staging deployment with + disposable data: readiness, sign-in, owned run/chat/pharma access, rejection of + unauthorized access, and each enabled storage or payment integration. +3. Drain active work before restarting. Deploy matching frontend and backend + versions, confirm the selected databases and expected records, then reopen + submissions. Check readiness, job failures, and credit reconciliation. +4. Commit documentation in its submodule before updating the parent pointer. + Describe confirmed breaks in commit/PR `BREAKING CHANGE:` notes, with migration + steps; list probable regressions and untested paths separately. This note does + not assign a release version or create a tag. + +If rollback is needed, pause new writes and restore the matching application, +dependency lockfiles, deployment configuration, and documentation pointer. Review +jobs and credit changes made since rollout before restoring any database backup; +a code rollback does not undo them. Do not restore exposed provider credentials or +reopen public routes to make an older build work. If the older version requires +those behaviors, keep the affected feature unavailable until a safe fix is ready. diff --git a/guides/backend-features.md b/guides/backend-features.md new file mode 100644 index 0000000..53f242d --- /dev/null +++ b/guides/backend-features.md @@ -0,0 +1,133 @@ +# Backend feature reference + +For installation and environment ownership, see [Environment setup](environment.md). + +# Backend V2 + +`backend` is the next backend foundation for the decentralized quantum +service network. It starts with a thin FastAPI platform edge, explicit +configuration, typed models, and a protocol-first package layout that will +grow into the full migration plan described in `backend-migration.md`. + +## Provenance + +See [../repository/backend/PROVENANCE.md](../repository/backend/PROVENANCE.md) for the versioned canonical hashing and Merkle +proof format, backward-compatible bundle fields, and Ethereum anchoring handoff. + +## Tokenized-stock portfolios + +`GET /api/v1/finance/tokenized-assets` lists the Robinhood Stock Tokens whose Chainlink +price history can be analysed, with the sampling rules, data source and disclaimer. +`POST /api/v1/finance/tokenized-portfolio` takes `symbols`, `start_date`, `end_date` and +the usual optimizer settings, validates them, and queues a normal finance job (#18): + +```json +{"symbols": ["AAPL", "MSFT", "SPY"], "start_date": "2026-06-23", "end_date": "2026-10-28"} +``` + +The job reads validated daily token prices from Robinhood Chain mainnet (read-only, see +[Robinhood Chain market data](../repository/backend/CHAIN_REGISTRY.md#robinhood-chain-market-data)) and runs the +existing portfolio optimizer on them. `GET /api/v1/finance/{job_id}` then adds: + +- `market_data`: source, chain, block, aligned date range and per-asset token/feed + addresses and first/last round IDs. +- `allocation_weights`: the optimizer's equal weights over each method's selected assets. +- `disclaimer`: analysis only, not investment advice; nothing is signed or traded. +- `provenance`: the existing finance bundle, with every daily close as a Chainlink + oracle reference, so the exact inputs can be re-read by round ID. + +Unsupported assets, future end dates, ranges without 60 weekdays and a `budget` the +optimizer would reject get a 422 before a job exists. Missing, invalid or insufficient +history (fewer than 60 aligned trading days) fails the job with a fixed message instead +of substituting prices. At most two history reads run at once per process, because the +default public RPC is rate-limited; `QB2_ROBINHOOD_MAINNET_RPC_URL` optionally +overrides it. Rounds are immutable, so caching them per feed is a possible follow-up. + +## Kubo RPC client + +`quantum_backend_v2.content.kubo.KuboClient` targets the pinned Kubo **v0.43.1** +service from #59/#60. Use it on the backend's **asyncio** loop, not inside the +libp2p Trio thread. It is optional: this change adds no startup probe, automatic +uploads, gateway reads or quantum-job integration. + +Configuration flows through `AppSettings.from_env()` into `settings.kubo`: + +| Variable | Default / purpose | +|---|---| +| `KUBO_RPC_URL` | `http://kubo:5001`; HTTP(S) root or `/api/v0` base | +| `KUBO_RPC_SECRET` | Private raw token shared with Kubo; required only when constructing the client | +| `KUBO_RPC_TIMEOUT_SECONDS` | `30`; finite I/O and total-operation timeout, maximum `300` | +| `KUBO_RPC_MAX_RESPONSE_BYTES` | `4194304`; decoded response budget, maximum 64 MiB | + +The default hostname resolves inside Compose's `content_network`. Outside Docker, +use a protected connection; **do not publish RPC publicly**. URLs containing +credentials, query parameters or fragments are rejected. Redirects and environment +proxies are disabled; tokens are masked in settings and omitted from client errors. + +```python +from quantum_backend_v2.config import KuboSettings +from quantum_backend_v2.content.kubo import KuboClient, KuboError + + +async def maybe_store(payload: bytes, config: KuboSettings) -> str | None: + try: + async with KuboClient(config) as client: + added = await client.add(payload) + return added.cid + except KuboError: + # Keep the original execution result; storage is optional. + return None +``` + +For repeated operations, reuse one client outside the loop. `async with`/`aclose()` +closes its HTTP session; binary input files remain caller-owned, but multipart +encoding may seek to their beginning and advance their cursor. Inputs may be +bytes or regular binary files; do not pass blocking pipes/custom blocking readers. + +| Method | Typed result | +|---|---| +| `add(data, pin=True)` | CIDv1 and reported DAG size (not necessarily original file length) | +| `pin_add(cid, recursive=True)` / `pin_rm(cid, recursive=True)` | Acknowledged pin CIDs; unpinning is not immediate deletion | +| `pin_ls(cid=None)` | CID-to-pin metadata map; empty listings are valid; an indirect pin reports its pinned root in `through` | +| `dag_import(data, pin_roots=True)` | CAR block statistics and successfully pinned roots | +| `block_stat(cid)` | Reported CID and block size, not independent byte verification | + +`KuboRejectedError` covers invalid client configuration/arguments, rejected HTTP +requests (including disallowed redirects) and explicit RPC/pin errors. Kubo sends +most command errors, such as an invalid CID or `pin_rm`/`pin_ls` on an unpinned +CID, as HTTP 500 with a JSON error body; a well-formed one with code 0 (normal), +1 (client) or 4 (forbidden) is rejected. +`KuboUnavailableError` covers timeouts/transport failures, HTTP 408/429/502-504, +any other HTTP 500 (implementation errors, rate limits, Kubo-internal context +timeouts, non-Kubo bodies) and invalid/incomplete responses. The total-operation +timeout includes upload time, so raise it for large inputs. Errors expose only +operation/category/status, never remote error text. Cancellation propagates. +**No automatic retries:** a failed or +timed-out mutation may already have stored blocks or changed pins; reconcile its +outcome rather than blindly replaying it. CAR import is not transactional, and +`pin_roots=False` returns no pinned roots, not an inventory of CAR header roots. + +Kubo can put a late error in an HTTP trailer after status 200. The default +HTTP/1.1 transport uses public `httpcore`/`h11` interfaces to detect it before +returning a result; plain HTTPX otherwise drops these trailers. There is no +private monkeypatch or replacement TLS/parser stack. Use a direct endpoint or a +proxy that preserves trailers. Responses are read completely within the budget; +compressed responses are rejected, and raw server bodies/headers should not be logged. +See the pinned [RPC commands](https://github.com/ipfs/kubo/tree/v0.43.1/core/commands) +and [late-error emitter](https://github.com/ipfs/go-ipfs-cmds/blob/v0.16.1/http/responseemitter.go). + +```bash +uv run --frozen pytest tests/unit/test_kubo*.py -q +``` + +MockTransport covers all six methods; local wire-level tests exercise actual +chunked HTTP trailers, fragmentation, pooling and privacy without external services. +An optional `tests/live/test_kubo_rpc.py` check is skipped unless +`QB2_KUBO_LIVE_TEST=1`. Run it **only against a disposable private node**, with its +URL/secret already provisioned in the environment; it creates harmless unique +fixtures and removes only their pins, never runs GC or deletes unrelated content. + +```bash +# Only after provisioning the private disposable test URL/token in the environment: +QB2_KUBO_LIVE_TEST=1 uv run --frozen pytest tests/live/test_kubo_rpc.py -m live_kubo -q +``` diff --git a/guides/cleanup-validation.md b/guides/cleanup-validation.md new file mode 100644 index 0000000..ded0a4e --- /dev/null +++ b/guides/cleanup-validation.md @@ -0,0 +1,55 @@ +# Setup cleanup validation — 2026-11-01 + +Release changes are prepared on separate application and documentation branches. The unrelated Playwright snapshot deletion and MONAD research document are excluded. Existing landing work was completed only where needed to fix its failing architecture test. + +## Changes + +- Minimal component env examples and a deployment-only root example; one dotenv path for native CLI and app factory. Root Compose passes backend configuration through its environment file. +- Existing frontend env files consolidated into `.env.local` after a Next.js-loader equivalence check. Provider credentials renamed to server-only settings. The canonical auth database preserves the previous effective `quantum_platform` database; the older ignored label is retained privately as a comment. +- Mongo routing aligned across services; client cleanup and optional Bedrock initialization corrected. +- Signed identity added to run, legacy chat, and pharma proxies; legacy backend endpoints enforce authentication and ownership. +- Provider uploads moved to authenticated server routes. Local tracking removal is supported; permanent provider deletion requires the provider dashboard because shared credentials cannot safely establish exclusive ownership. +- Helia starts on demand, retains its browser-only boundary, and never deletes IndexedDB automatically after a timeout. +- Finance computation offloaded from the API event loop. Restart cleanup targets the configured coordinator's attributed jobs and releases older credit holds even if new-job charging is disabled. +- Pharma dependencies are optional. Python distributions include imported sibling packages, prompt assets, and Toju worker sources; Docker includes Node 22 and worker dependencies. +- Corrected Pinata research uploads, OTP delivery errors, API error display/logging, stale tests, and unsafe live-test defaults. Removed the unimported duplicate agent schema and duplicate Pinata adapter implementation. +- Moved 41 Markdown files into the documentation submodule. Only files named README.md remain outside it in project source; dependency environments and the separate py-libp2p reference checkout retain their own documentation; see the [migration index](../repository/README.md). + +## Verification + +| Check | Result | +| --- | --- | +| Full backend `pytest -q` in a temporary source copy without private env files, live test opt-ins unset | **1,198 passed, 114 skipped**, 139 deprecation/library warnings | +| Frontend test files in isolated Bun processes, automatic env loading disabled | **196 passed, 0 failed**; each file runs in a fresh process to isolate module mocks | +| Frontend TypeScript | Passed | +| ESLint on all changed frontend source/test files | Passed | +| Full frontend ESLint | Passed with 0 errors and 17 existing source warnings. The upstream `public/molstar.js` bundle is excluded explicitly; source lint rules remain enabled | +| Production Next.js build in a temporary copy with synthetic configuration | Passed; no private env files or real provider credentials used | +| Toju Node tests | 4 passed | +| Python sdist → wheel, package/assets inspection, wheel-only import with chemistry imports blocked | Passed | +| `uv lock --check` | Passed | +| Launcher syntax, Compose YAML/boundary checks, 98 local documentation links, Markdown location audit, application and authored-guide whitespace checks | Passed | +| Independent backend/frontend review | Reported issues fixed: missing pharma proxy identity, cost-disabled credit recovery, and unsafe shared-provider deletion | + +Release checks also fixed component lifecycle errors, async client pages in the vault, and the unfinished six-stage architecture sequence. The landing flow now includes result collection and avoids claiming automatic peer payouts or universal IPFS pinning. The dotenv fixture test explicitly isolates its file values from developer environment overrides. + +## README and auth follow-up + +- Retained the README's project, research, demo, and integration material, with navigation, a current setup path, and expandable Toju/Kubo operations. Missing provider credentials are documented as setup gaps; optional integrations remain available in code. +- Both auth forms recover their loading state after failures. Email checks validate response shape, server failures return safe JSON, failed MongoDB connections can retry, and provider delivery failures remain visible to the caller. +- Focused availability, Mongo configuration/recovery, and OTP delivery tests pass. Browser checks against an isolated production build confirmed signup and sign-in show an outage message and re-enable Continue with MongoDB unavailable; no email was sent in those checks. +- A separate configuration-only production redeployment restored OTP delivery, confirmed by the recipient. It did not deploy these source changes. No OTP was redeemed and no account was created by the agent; full account creation/sign-in is unverified. + +## Limits and operational notes + +- Docker is not installed in the execution environment; no Docker image or Compose service was started. YAML checks are not deployment verification. +- Live MongoDB ping, an account-existence count, public auth endpoints, and one authorized OTP email were checked during the deployment follow-up; receipt was confirmed by the user. Cloud storage, wallets/payments, real browser IndexedDB, and native chemistry execution were not tested. Live/optional tests are intentionally skipped. +- Previously browser-exposed provider credentials still need rotation at their providers. Code changes and private variable renaming do not revoke previously published credentials. +- Use a unique, stable `QB2_PEER_ID` per running coordinator. Legacy jobs without coordinator attribution are left untouched; interrupted computation is failed rather than resumed. Pharma state remains process-local. +- Provider upload and Mongo audit writes are separate operations. A successful upload followed by a failed database write may require provider-dashboard cleanup. +- Frontend `AGENTS.md` and `CLAUDE.md` moved at the user's request. Automatic discovery at their old paths stops; the frontend README links their new locations. +- Publish documentation first, then update the parent Git pointer. Feature branches and PRs preserve contributor review; no release tag, database migration, credential rotation, or production source deployment is included. Hosted-backend deployment remains blocked by unavailable AWS access. + +## Markdown cleanup follow-up + +An ignore-aware scan had missed 15 local demo documents, design specs, and plans under `docs/superpowers/`. They now live in the existing `superpowers/` directory of this documentation submodule, alongside their related context and design documents. Those 15 files were checked against their original SHA-256 hashes. The migration index includes all 41 moved files. Four earlier moved guides received link corrections; historical Markdown retains its existing whitespace and intentional hard breaks. diff --git a/guides/content-node.md b/guides/content-node.md new file mode 100644 index 0000000..07a0f8d --- /dev/null +++ b/guides/content-node.md @@ -0,0 +1,186 @@ +# Optional content node + +Native compute does not require Kubo. Deployment configuration belongs in the root `.env`; see [Environment setup](environment.md). Enable `--profile storage` for Kubo; `--profile monitoring` also enables it. + +## Platform content node (Kubo) + +The Compose stack includes **Kubo v0.43.1**, pinned by release tag and multi-platform +image digest. **Before starting or upgrading**, set `KUBO_RPC_SECRET` privately in +`.env`: a random 32–128 character token containing only letters, digits, `_` or `-` +(for example, generate one with `openssl rand -hex 32`). Use the raw token, without +`bearer:`. Protect `.env` with mode `0600`; never commit or paste its contents. +Missing/invalid secrets prevent the Kubo daemon from starting, not backend compute. +With current Docker Engine/Compose, start only the content node: + +```bash +docker compose --profile storage up -d --wait kubo +docker compose exec kubo sh /opt/qmesh/kubo-rpc.sh diag healthy +docker compose stop kubo +``` + +No wallet/provider credentials or backend startup are required. Compose may warn about +unset variables for the existing backend, even when starting only Kubo. If port +8080 is in use, run `KUBO_GATEWAY_PORT=18080 docker compose --profile storage up -d --wait kubo` +(or set that variable in `.env`). The gateway remains loopback-only. + +| Interface | Exposure | +|---|---| +| Swarm `4001/tcp` and `4001/udp` | Published for IPFS peers, separate from quantum-worker libp2p | +| Gateway `8080/tcp` | `127.0.0.1:8080` only; read locally stored content at `/ipfs/`; no writes | +| RPC `5001/tcp` | **Not published** or proxied by Caddy; authenticated backend access at `http://kubo:5001` on `content_network` | + +The named `kubo_data` volume persists `/data/ipfs` across container recreation. +Stopping/removing a container does not remove it. Volume deletion permanently +removes this node's data and identity; do not use `docker compose down --volumes` +on a deployment unless you intend to delete **all** that project's named volumes. +The backend has no Kubo health dependency: content failures must not block compute. + +`IPFS_PROFILE=server` applies **only when a new repository is initialized**. It +turns off mDNS and NAT port mapping and filters private swarm addresses. The +image sets `Addresses.API=/ip4/0.0.0.0/tcp/5001` and +`Addresses.Gateway=/ip4/0.0.0.0/tcp/8080` **inside the container**. Host exposure is +controlled by the mappings above. The server profile does not retrofit existing +repositories; back up the volume and review migrations before changing versions. +The security hook below, unlike the profile, runs on **every container start**. +See the pinned [release](https://github.com/ipfs/kubo/releases/tag/v0.43.1), +[profile/configuration reference](https://github.com/ipfs/kubo/blob/v0.43.1/docs/config.md#server-profile), +[image entrypoint](https://github.com/ipfs/kubo/blob/v0.43.1/bin/container_daemon), +and [healthcheck](https://github.com/ipfs/kubo/blob/v0.43.1/Dockerfile). + +### RPC authorization and managed controls + +Compose mounts `deploy/kubo-init.sh` read-only into the official image's init +hooks. It reapplies native `API.Authorizations` before the daemon starts, including +on existing volumes. Missing hook files fail the mount rather than silently +starting an unprotected node. The backend and Kubo share `KUBO_RPC_SECRET`; +Caddy does not receive it or join their dedicated `content_network`. + +Clients send `Authorization: Bearer `. The backend scope permits only +`id`, `diag/healthy`, `add`, `cat`, `pin/add`, `pin/rm`, `pin/ls`, `dag/import`, and +`block/stat`. Configuration, shutdown and other admin commands are denied even +with that token. Native `/version` also requires authentication, but is implicitly +allowed for valid tokens regardless of `AllowedPaths`. The healthcheck reads its +token inside the container and suppresses command output. + +The hook also manages: + +- Local-only gateway fetching (`Gateway.NoFetch=true`), no DNSLink and no exposed + routing API. Kubo removed `Gateway.Writable` in v0.20: do not set that obsolete + key; the supported gateway is read-only. +- Basic swarm connection-manager watermarks **32/96** (trimming targets, not a + hard connection count), resource management enabled with a **1024** networking + file-descriptor budget, and unused relay/AutoNAT server duties disabled. +- Repository config permissions `0600`. `Swarm.ResourceMgr.MaxMemory` is left at + its default: it is not a process-memory cap. Production memory/disk sizing and + quotas need workload-specific tuning, separately from these swarm controls. + +These managed keys, including the entire `API.Authorizations` map, are overwritten +on restart; do not rely on manual edits to them. To rotate the token, update the +private `.env` then run `docker compose --profile storage up -d --force-recreate kubo backend`. +This briefly interrupts clients but retains content/pins/identity; the old token +is revoked. Do not delete the volume or rerun `ipfs init` to rotate credentials. +Docker administrators and repo readers remain trusted: the token is present in +container environments and native config, and expanded Docker/Compose config or +process inspection can reveal it. Never expose it through browser variables, +logs or public proxies. See the pinned [native authorization implementation](https://github.com/ipfs/kubo/blob/v0.43.1/core/corehttp/commands.go#L176) +and [resource-management guidance](https://github.com/ipfs/kubo/blob/v0.43.1/docs/production/low-memory.md). + +### Storage targets, automatic GC and private monitoring + +`deploy/kubo-storage.sh` runs after the security hook on **every start**, including +existing volumes. It manages `Datastore.StorageMax` (default **10GB**), +`Datastore.StorageGCWatermark` (**90%**) and `Datastore.GCPeriod` (default **1h**). +Compose enables `daemon --enable-gc`, preserving the image's migration/version flags. +Tune `KUBO_STORAGE_MAX` and `KUBO_GC_PERIOD` privately in `.env`, then recreate Kubo. +Accepted formats are a positive integer of up to six digits followed by `MB`/`GB`/`TB` +or `s`/`m`/`h`, respectively; fractions, spaces and leading zeroes are rejected before +configuration is changed. Empty values use defaults. These keys overwrite manual +repo edits at restart. **Upgrading enables automatic GC on existing volumes**: +pin content you need and back up the repository before recreating the node. + +**StorageMax is a soft GC target, not a disk quota.** Every GC interval Kubo checks +whether raw-block storage exceeds the watermark; it only collects unpinned, +unreferenced blocks. Recursively pinned DAGs and live MFS roots are retained even +above StorageMax, so pins can fill the disk. Metadata and datastore overhead require +additional headroom; deleting blocks need not immediately shrink every datastore. +Unpinning makes content eligible for collection, not immediate or global deletion. +The backend credential still cannot run `repo/gc` or change configuration; automatic +GC is internal to the daemon. See the pinned +[storage settings](https://github.com/ipfs/kubo/blob/v0.43.1/docs/config.md#datastorestoragemax) +and [GC implementation](https://github.com/ipfs/kubo/blob/v0.43.1/core/corerepo/gc.go). + +Enable optional monitoring without starting backend/Caddy: + +```bash +docker compose --profile monitoring up -d --wait kubo prometheus +docker compose exec prometheus wget -qO- 'http://127.0.0.1:9090/api/v1/targets' +docker compose exec kubo sh -c 'df -h /data/ipfs; du -sh /data/ipfs' +# Stop only the monitoring service, keeping node/application/data intact: +docker compose --profile monitoring stop prometheus +``` + +When tearing down a monitoring-enabled project, include `--profile monitoring` +with `down` so it also selects the monitor; omit `--volumes` to retain data. + +The digest-pinned Prometheus service scrapes Kubo's native +`http://kubo:5001/debug/metrics/prometheus` every 30s. This native debug endpoint is +outside command-RPC authorization: scraping needs **no backend token**. Prometheus +receives no RPC secret and joins only the trusted `content_network`; neither its UI +nor Kubo's API/metrics has a published host port or Caddy route. Docker/network +administrators remain trusted. Inspect via `exec`, not a public proxy. Compute has +no dependency on monitoring. Once started, the scraper continues when Kubo is down. +Prometheus history uses its own named volume with 7-day/512MB retention targets; +WAL/compaction overhead means these are not hard filesystem limits either. + +Basic operator alert notes (no Alertmanager/notification delivery is installed): + +- `up{job="kubo"} == 0` for 2 minutes: inspect node health/logs and disk space. +- `sum(rate(ipfs_fsrepo_datastore_put_errors_total{job="kubo"}[5m])) > 0`: + investigate datastore write failures, including a full disk. +- Monitor the **filesystem hosting kubo_data** separately: warn at 80% used, + critical at 90%, and size StorageMax well below available capacity. The pinned + native exporter does **not** provide an `ipfs_repo_size` or filesystem-free-space + gauge. `df`/`du` above are manual checks, not automated Prometheus disk alerts; + wire existing host monitoring or a filesystem exporter before production. With + Docker Desktop, monitor the VM allocation and outer host disk as well. +- A full disk is not fixed by repeatedly running GC when pins dominate. Stop new + publications, expand storage or intentionally review/unpin expendable artifacts; + retain MongoDB job results. Never delete the deployment volume as a cleanup step. + +Checks require Python 3.10+ (via uv) and Docker Compose with `up --wait` support: + +```bash +uv run --no-project python -m unittest discover -s scripts -p 'test_kubo*.py' -v +uv run --no-project python scripts/smoke_kubo.py +# Also start a disposable Prometheus and prove scrape success + node-outage detection: +uv run --no-project python scripts/smoke_kubo.py --monitoring +``` + +The first command checks deployment configuration without starting containers. +The optional live smoke test extracts only Kubo into a uniquely named temporary +project, ignores project `.env` values, uses random host ports and runs **offline**. +It generates temporary test-only RPC credentials and checks health/profile, +protected RPC denial (anonymous, wrong token, forbidden admin commands), successful +authorized add/cat, loopback read-only gateway, matching bytes/CIDs, persistence, +real token rotation/revocation, secret-free node/health logs and fail-closed startup +without a secret. It inspects only its disposable config in memory, never prints +it, and attempts to remove only its own containers/network/volume, including on failure. +If cleanup also fails, it reports the disposable project label without replacing +the original error. Command failures identify the operation without dumping +possibly sensitive output. It needs no backend dependencies or private data; +its fixtures are harmless public text and a deterministic 2MiB binary DAG. The +storage test uses a **disposable 1MB target/1s GC interval**: it proves an oversized +pinned DAG remains readable, then unpins it and waits for automatic reclamation +while a separate pin survives. Production defaults are not reduced. `--monitoring` +also checks `up{job="kubo"}=1`, an unpublished monitoring UI, and `up=0` after node +shutdown. It never runs GC against an existing deployment. + +The normal service is **online**: IPFS is public content-addressed networking, +not confidential storage. Never add private data expecting the CID to be secret. +Native authentication adds defense beyond unpublished RPC; it does not make +Docker administration or public IPFS content confidential. The typed async +[Python RPC client](backend-features.md#kubo-rpc-client) supports content writes, +pin management and CAR imports with bounded requests and safe errors (#61). +Optional circuit-run provenance publishing and CID-verified gateway reads are +documented in the backend. Storage-adapter integration and production hosting +remain separate work. diff --git a/guides/environment.md b/guides/environment.md new file mode 100644 index 0000000..eff0ab7 --- /dev/null +++ b/guides/environment.md @@ -0,0 +1,136 @@ +# Environment and setup + +Start with the [repository README](https://github.com/Winter-Soren/distributed-quantum-services/blob/main/README.md). Native local compute needs Python 3.12 or 3.13, uv, and MongoDB. The browser console additionally needs Bun, two locally generated secrets, and a Resend sender for email sign-in. No blockchain, LLM, cloud storage, Stripe, AWS, or PostgreSQL account is required for local circuit and CSV finance computation. + +## Files and precedence + +| File | Owner and purpose | How it is loaded | +| --- | --- | --- | +| `backend/.env` | Native Python API, workers, agents | Copy `backend/.env.example`. CLI and app factory load this exact file, independent of the launch directory. Exported process variables win. Legacy services and typed settings see the same values. | +| `frontend/.env.local` | Next.js server and public browser configuration | Copy `frontend/.env.example`. Next.js precedence is process environment, `.env..local`, `.env.local`, `.env.`, then `.env`. `.env.local` is skipped in test mode. | +| Root `.env` | Optional Docker backend/Caddy/Kubo deployment | Copy root `.env.example` only when deploying with Compose. The backend container receives this file through `env_file`; Compose fixes its internal host/port and persistent paths. It does not load `backend/.env`. Compose interpolation variables can come from the invoking shell. For other container overrides use `docker compose run -e NAME=value backend`. | + +The retired `services/toju-demo` prototype is not part of startup; any old env file there is not read. + +Do not copy all three files onto each other. Native development and deployment can legitimately use different databases, hosts, and secrets. Frontend authentication has its own database, separate from backend job data. Avoid adding a second frontend `.env`; existing files remain supported with the precedence above. When consolidating an existing installation, transfer the effective values to `.env.local` before removing the old file. Do not overwrite credentials with blank examples. + +Installed Python wheels look for `.env` in the launch directory. Explicit environment mappings passed to `load_settings(env=...)` are isolated from local files; tests use this path. No loader searches parent directories for a deployment file. + +`NEXT_PUBLIC_` values are embedded into the browser build. They must never contain provider credentials, signing keys, or the auth bridge. Change them before rebuilding the frontend. Server credentials need only a server restart. + +## Minimal native setup + +1. Start MongoDB locally, or create an Atlas database user and add your development IP to the network access list. Obtain the driver URI using [Atlas Connect → Drivers](https://www.mongodb.com/docs/atlas/driver-connection/). URI passwords with reserved characters must be encoded. +2. Copy the two component examples if their destination files do not exist. Install backend with `cd backend && uv sync --locked`; frontend with `cd frontend && bun install --frozen-lockfile`. +3. The backend example is ready for loopback-only API exploration. Run `cd backend && make run`, then visit `http://127.0.0.1:8081/docs`. `/api/v1/health` reports process state; `/api/v1/ready` checks MongoDB. One embedded worker listens on 4021; the coordinator listens on 4011. A busy port produces an error; startup never kills the process holding it. +4. For console sign-in, generate two independent values with `openssl rand -hex 32`. Use one as `BETTER_AUTH_SECRET` in the frontend. Use the other as `AGENT_AUTH_BRIDGE_SECRET` in **both** components. Add Resend settings below, then run `cd frontend && bun run dev` and open `http://localhost:3000`. +5. When sharing or deploying the API, set `QB2_AUTH_REQUIRED=true`. The local `false` setting grants a development administrator identity and is appropriate only on loopback. `NEXT_PUBLIC_TRIAL_DISABLED=true` skips the trial gate; it does not skip sign-in. + +| Variable | Where | Value/source and purpose | +| --- | --- | --- | +| `QB2_MONGODB_TARGET` | Backend | `local` selects `QB2_MONGODB_LOCAL_URI`; `remote` selects `QB2_MONGODB_REMOTE_URI`. All backend services honor this selection. | +| `QB2_MONGODB_LOCAL_URI` / `QB2_MONGODB_REMOTE_URI` | Backend | Local MongoDB connection or Atlas Drivers URI. Treat a URI with credentials as a secret. | +| `QB2_MONGODB_DATABASE` | Backend | Existing backend database name; fresh setup uses `quantum_backend_v2`. Preserve an existing `qds` or custom name to retain access to its records. No data is moved automatically. | +| `MONGODB_URI` | Frontend server | Connection URI for Better Auth and vault metadata. It can use the same Mongo server as the backend. | +| `MONGODB_DB_NAME` | Frontend server | Auth database, default `quantum_platform`. Legacy `MONGODB_DATABASE` remains a fallback; canonical name wins if both are present. | +| `BETTER_AUTH_SECRET` | Frontend server | Independent random secret from the command above. Keep stable between restarts. | +| `BETTER_AUTH_URL` | Frontend server | `http://localhost:3000` locally; exact deployed frontend origin in production. | +| `AGENT_AUTH_BRIDGE_SECRET` | Both servers | One shared random secret, at least 32 characters. Signs the current user's identity on server-to-server calls. Never public. | +| `RESEND_API_KEY` | Frontend server | Create a sending key in [Resend API Keys](https://resend.com/docs/dashboard/api-keys/introduction). Prefer a key scoped to the sending domain. | +| `OTP_EMAIL_FROM` | Frontend server | A sender address on a [verified Resend domain](https://resend.com/docs/dashboard/domains/introduction). Provider restrictions still apply to test senders. Failed sends are reported to the client. | +| `NEXT_PUBLIC_BACKEND_URL` | Frontend build | Public API origin, locally `http://127.0.0.1:8081`. It contains no credentials. | +| `QB2_API_HOST`, `QB2_API_PORT` | Native backend | Local bind address/port. CLI `--host`/`--port` override them. Compose fixes the internal port to 8080 for Caddy. | + +## Optional integrations: add only what you use + +Add these variables to the owning server's file, or to its hosted secret settings. There is no need to populate unused entries with placeholder credentials. Feature-specific guides contain request examples; the tables below identify how to obtain configuration. + +### AI and wallet features + +| Feature | Variables | Where to get them | +| --- | --- | --- | +| Agent chat (select one provider) | Backend `AGENT_LLM_PROVIDER=groq`, `GROQ_API_KEY`, optional `GROQ_MODEL` | Create a key at [Groq Console](https://console.groq.com/keys); choose an available model. | +| OpenRouter alternative | Backend `AGENT_LLM_PROVIDER=openrouter`, `OPENROUTER_API_KEY`, `OPENROUTER_MODEL` | [OpenRouter keys](https://openrouter.ai/settings/keys). This application accepts free model IDs and enforces zero-price limits; it does not fall back to a paid provider. | +| NEAR alternative | Backend `AGENT_LLM_PROVIDER=near`, `NEARAI_API_KEY`, `NEAR_MODEL` | [NEAR AI Cloud](https://cloud.near.ai/); choose a model supporting attestation and configure your account's spending limit. | +| Legacy Claude agent / proposal analysis | Backend `AWS_REGION`, AWS credential chain (including `AWS_SESSION_TOKEN` for temporary credentials), `AWS_BEDROCK_DEFAULT_MODEL` for legacy agent, `BEDROCK_MODEL_ID` for proposal analysis | Use an AWS role/profile or credentials with permission to invoke the selected Bedrock model. Proposal analysis uses Bedrock lazily. `AWS_BEDROCK_ENABLED=false` selects direct Anthropic only for the legacy agent; its `ANTHROPIC_API_KEY` comes from [Anthropic Console](https://console.anthropic.com/). This flag does not change proposal analysis. | +| CDP wallets | Backend `CDP_API_KEY_ID`, `CDP_API_KEY_SECRET`, `CDP_WALLET_SECRET`, optional `NETWORK=base-sepolia` | Follow the [CDP API key wallet setup](https://docs.cdp.coinbase.com/wallets/quickstart/api-key-auth) to create API credentials and a separate wallet secret. These are server signing credentials. Use the application's supported test network and faucet funds. | +| Wallet/session encryption | Backend `WALLET_ENCRYPTION_KEY` | Generate a persistent encryption secret according to the [Starknet guide](../repository/backend/STARKNET_SESSION_GUIDE.md). Do not change an existing key without migrating encrypted wallet/session data; key rotation is not a setup cleanup step. | +| Backend notification emails | Backend `RESEND_API_KEY`, `RESEND_FROM_EMAIL` | Same Resend setup as sign-in, but configured independently on the backend. `OTP_EMAIL_FROM` is frontend-only. | + +Model defaults live in code and can become unavailable upstream. Set a model override only when needed. Main agent chat and legacy/proposal analysis are different integrations; neither requires configuring every provider. + +### Storage + +| Feature | Variables | Where to get them | +| --- | --- | --- | +| Pinata vault upload | Frontend server `PINATA_JWT`, **or** `PINATA_API_KEY` plus `PINATA_API_SECRET` | Create a scoped key in [Pinata API Keys](https://docs.pinata.cloud/account-management/api-keys). Enable the pin, list, and unpin operations you use. No `NEXT_PUBLIC_PINATA_JWT` is used. | +| Pinata research output | Same names on the backend | Backend uploads research results through Pinata. The old `WEB3_STORAGE_API_KEY` setting is unused. | +| Lighthouse vault upload | Frontend server `LIGHTHOUSE_API_KEY` | Create an API key in the [Lighthouse dashboard](https://files.lighthouse.storage/). No `NEXT_PUBLIC_LIGHTHOUSE_KEY` is used. | +| Browser IPFS | None | Helia initializes when a consumer needs it. An IndexedDB timeout preserves local content; close other vault tabs and retry. | +| Private Kubo publication | Root deployment `KUBO_RPC_SECRET`, optional `KUBO_STORAGE_MAX`, `KUBO_GC_PERIOD`, `KUBO_GATEWAY_PORT` | Generate the RPC secret with `openssl rand -hex 32`. Run Compose with `--profile storage`; `--profile monitoring` includes Kubo and Prometheus. [Content-node guide](content-node.md). | +| Kubo outside Compose | Backend `KUBO_RPC_URL`, `KUBO_RPC_SECRET` | Operator-managed private RPC endpoint and its token. Never publish administration port 5001 to browsers. | +| Gateway access | Backend `IPFS_GATEWAY_URL`, optional `IPFS_PUBLIC_GATEWAY_URL` | Read gateway origin; the latter is a browser-accessible origin for provenance links. Optional limits: `KUBO_RPC_TIMEOUT_SECONDS`, `KUBO_RPC_MAX_RESPONSE_BYTES`, `IPFS_GATEWAY_TIMEOUT_SECONDS`, `IPFS_GATEWAY_MAX_RESPONSE_BYTES`. Defaults are bounded. | +| Toju paid exports | Backend `TOJU_API_URL`, `TOJU_PAYMENT_RECIPIENT`, optional `TOJU_NODE_BINARY` | Obtain endpoint and recipient from your full Toju server operator. Install Node 22 and run `npm ci` in `backend/toju-client`. [Toju guide](toju-storage.md). The backend worker is not the Toju server. | + +Vault provider requests require a signed-in session. Existing `nft.storage` metadata remains supported as a Pinata alias. Removing a pin from the vault updates only your own tracking record. Permanent provider deletion must be performed in the provider dashboard: a shared provider account cannot safely prove that no other user or concurrent upload relies on those bytes. + +### Payments and chains + +| Feature | Variables | Where to get them | +| --- | --- | --- | +| Stripe subscription | Frontend server `STRIPE_SECRET_KEY`, `STRIPE_PRICE_ID`, `STRIPE_WEBHOOK_SECRET` | Stripe Dashboard test API key, recurring product Price ID, and webhook signing secret (or local `stripe listen` secret). All three enable billing. [Stripe keys](https://docs.stripe.com/keys). | +| Credit packs | Frontend `STRIPE_CREDIT_PACKS=price_id:credits`, `BILLING_INTERNAL_SECRET`; backend `QB2_BILLING_INTERNAL_SECRET`, `QB2_CIRCUIT_JOB_CREDIT_COST` | One-time Stripe Price IDs and credit amounts; generate one separate shared secret of at least 32 characters. Keep cost `0` unless enabling charging. Optional `STRIPE_STABLECOIN_PAYMENTS=true` requires Stripe account support. | +| Algorand x402 | Frontend `X402_ENABLED=true`, `X402_NETWORK=testnet`, `X402_AVM_ADDRESS`, `X402_FACILITATOR_URL`, `X402_QUANTUM_RUN_PRICE`, `X402_GATEWAY_SECRET`; backend `QB2_X402_GATEWAY_SECRET` | Your receiver wallet address, facilitator URL, and one separate generated shared secret. Follow the [x402 runbook](../algorand-x402/algorand-x402-runbook.md). In a Next.js env file write `X402_QUANTUM_RUN_PRICE=\$0.01` so dotenv expansion preserves the dollar sign. Never add a payer private key to application env. | +| RPC overrides | Backend `QB2_BASE_SEPOLIA_RPC_URL`, `QB2_ETHEREUM_SEPOLIA_RPC_URL`, `QB2_ROBINHOOD_TESTNET_RPC_URL`, `QB2_ROBINHOOD_MAINNET_RPC_URL`, `QB2_STARKNET_SEPOLIA_RPC_URL` | Optional provider RPC URLs; defaults are public endpoints in the chain registries. URLs containing credentials are secrets. `QB2_CHAIN_REGISTRY_PATH` is an optional operator-owned TOML override. | +| ENS | Backend `ENS_NETWORK`, `ENS_RPC_URL`, `ENS_CACHE_TTL`, `ENS_RESOLVE_TIMEOUT` | Optional Ethereum L1 endpoint/network overrides and bounded cache/timeout settings. [ENS guide](../repository/backend/docs/ens-resolution.md). | +| Starknet session spending | Backend `STARKNET_POLICY_CONTRACT`, `STARKNET_SESSION_KEY_SPEND`, `STARKNET_RELAYER_ADDRESS`, `STARKNET_RELAYER_PRIVATE_KEY`, optional `AVNU_PAYMASTER_API_KEY` | Values from your deployed testnet contract, funded relayer, and AVNU account. See [Starknet setup](../repository/backend/STARKNET_SESSION_GUIDE.md); deploying/funding is a separate action. | +| Provenance anchoring | Backend `QB2_ANCHOR_NETWORK`, `QB2_ANCHOR_SENDER`, optional `QB2_ANCHOR_PAST_SENDERS`, `QB2_ANCHOR_PRIVATE_KEY` | Dedicated low-balance anchor account on a supported network. Optional scheduling/limits: `QB2_ANCHOR_INTERVAL_SECONDS`, `QB2_ANCHOR_MAX_BATCH`, `QB2_ANCHOR_MAX_FEE_GWEI`. [Provenance guide](../repository/backend/PROVENANCE.md). | + +### Local workers and optional chemistry dependencies + +The core install includes circuits, finance, and network operation. Chemistry is optional: `cd backend && uv sync --locked --extra pharma`. Keep `--extra pharma` on later `uv run`/`uv sync` commands if using uv to launch chemistry; `make run` uses the installed virtualenv directly. Without this extra, molecular submissions return an actionable 503. Docker builds enable it with `--build-arg INSTALL_PHARMA=true`. + +`QB2_LIBP2P_DEV_SERVICE_PEER_COUNT=1` is a small local fabric. Additional workers use `QB2_LIBP2P_DEV_SERVICE_BASE_PORT` (default 4021). Remote peers use `QB2_LIBP2P_BOOTSTRAP_PEERS` and optional `QB2_LIBP2P_ADVERTISE_MULTIADDRS`; peers must share `QB2_LIBP2P_RENDEZVOUS_NAMESPACE`. Leave heartbeat/TTL, logging, Mongo timeout, and other tuning settings at their code defaults unless diagnosing a specific need. + +Keep `QB2_PEER_ID` unique and stable for each coordinator using a database. Startup marks only that coordinator's newly attributed interrupted jobs as failed, with safe credit/reservation reconciliation; it does not resume computation or alter unowned legacy jobs. Different simultaneously running coordinators must not share an ID. Pharma jobs remain process-local and do not survive restart. Peer logs and peerstores are durable local state; `make run-clean` now refuses deletion. + +## Upgrading an existing checkout + +- Preserve the chosen backend database explicitly. Services previously defaulting to `qds` now use the same selected database as core jobs; inspect which existing database holds your data before changing settings. +- Rename frontend `NEXT_PUBLIC_PINATA_JWT` to `PINATA_JWT` and `NEXT_PUBLIC_LIGHTHOUSE_KEY` to `LIGHTHOUSE_API_KEY`, remove the public names, and rebuild. Previously shipped browser credentials need rotation through their providers; editing code cannot revoke existing bundles or keys. +- Prefer `MONGODB_DB_NAME`; its older spelling still works. Keep frontend auth data separate from the backend database unless intentionally migrating it. +- Remove stale PostgreSQL/Neon instructions from your local notes. `QB2_POSTGRES_*`, `QUANTUM_BACKEND_API_KEY`, `WEB3_STORAGE_API_KEY`, `PLATFORM_WALLET_SEED`, and `RESEND_TO_EMAIL` are not current application configuration. Do not delete credentials used by another project merely because this application no longer reads them. +- Root Compose deployment now passes optional backend variables through `env_file`, so billing, chain, and provider settings need no extra YAML forwarding. Internal API port is 8080; native API defaults to 8081. Compose hosts the backend, not the frontend or MongoDB. +- Use `uv sync --locked` instead of `--extra dev`; the development dependency group is installed by default. Production installs use `--no-dev`. The lockfile includes only required prerelease packages rather than globally permitting every prerelease. +- Markdown guides moved to [repository documentation](../repository/README.md). That index includes previous paths. Changes in this documentation submodule must be committed there before updating the parent repository's submodule pointer. + +## Troubleshooting + +### Deployed email sign-in + +Email sign-in first checks the frontend's MongoDB, then asks Resend to deliver a code. If `/api/auth/check-email` fails, no email is requested. Check that request and `/api/auth/get-session` in the browser's Network panel before investigating inbox delivery. + +1. Open the frontend project's runtime logs and find the failed auth request. The Python backend's health endpoint does not test the frontend auth database. +2. In the frontend host's **Production** environment, verify `MONGODB_URI` points to a reachable database, not localhost. Preserve the existing auth database with `MONGODB_DB_NAME`; changing its name will hide existing accounts. For Atlas, check the database user/password, encoded password characters, cluster availability, and network access for the deployed server. See [Atlas connection troubleshooting](https://www.mongodb.com/docs/atlas/troubleshoot-connection/). +3. Set `BETTER_AUTH_SECRET`, and make `BETTER_AUTH_URL` match the frontend origin users actually visit, including `www` if applicable. Local `.env.local` settings do not configure Vercel. Environment changes apply to a new deployment; [redeploy after updating them](https://vercel.com/docs/environment-variables). +4. Once the database check succeeds, verify the frontend's `RESEND_API_KEY` and `OTP_EMAIL_FROM`. Confirm the sender domain is verified and the key can send from it. Inspect Resend's delivery status if the OTP request succeeds but the inbox remains empty; API acceptance alone does not prove inbox delivery. + +The forms show an error and allow retry when auth is unavailable. Failed initial MongoDB connections are discarded so a later request can reconnect. Never share connection strings, provider keys, session cookies, or OTPs when reporting a failure. + +`querySrv ENOTFOUND` means the Atlas hostname cannot be resolved; check whether the cluster was paused, removed, or replaced and obtain its current Drivers URI. Restore the existing cluster where possible. A new empty database will not restore existing accounts. + +In a hosting dashboard, paste only the URI as the variable value, without quotes or a `MONGODB_URI=` prefix. `MongoParseError: Invalid scheme` means the value is malformed before a connection is attempted. If the URI works locally but the deployment cannot connect, check Atlas's IP access list: allowing your computer's IP does not allow the hosted server. Vercel uses dynamic outbound IPs by default; [Static IPs](https://vercel.com/kb/guide/how-to-allowlist-deployment-ip-address) require Pro or Enterprise. A `0.0.0.0/0` rule permits connections from any IPv4 address and requires an explicit network-security decision; database authentication remains necessary. + +### Other setup issues + +| Symptom | Check | +| --- | --- | +| API fails at startup or `/ready` fails | Mongo is running; selected target's URI is set; Atlas IP access and credentials are correct. Do not print a credential-bearing URI in diagnostics. | +| Address already in use | Stop your own previous process or change the configured port. The launcher deliberately does not kill port owners. | +| Console cannot sign in | Frontend MongoDB is reachable; Resend key/sender work; use the configured frontend origin consistently. | +| Authenticated API request fails | Both server bridge secrets match and are at least 32 characters; restart both processes after editing. Trial bypass does not authenticate a user. | +| Data appears missing after upgrade | Compare explicit database name and selected target with the previous installation before creating fresh data. No migration runs automatically. | +| Vault upload unavailable | Set a server-only credential for the chosen provider and restart Next.js. Backend research uploads need their own backend Pinata settings. | +| Wheel works except Toju | Install Node 22 and worker dependencies in the packaged `quantum_backend_v2/toju-client` directory; the Docker image already includes them. | + +Live Mongo tests require an explicitly supplied loopback `QB2_MONGODB_TEST_URI` and use uniquely named disposable databases; they do not default to your local application database. Unit checks use fake services. A passing unit suite does not validate a real sender, external wallet, deployment, or payment. Validate optional integrations individually when enabling them. diff --git a/guides/toju-storage.md b/guides/toju-storage.md new file mode 100644 index 0000000..e8ad765 --- /dev/null +++ b/guides/toju-storage.md @@ -0,0 +1,208 @@ +# Toju storage integration + +For initial setup and credentials, use [Environment setup](environment.md). + +## Toju agent storage — testnet integration + +Agents can draft a report in chat and propose storing it on public IPFS through +the **full Toju server**. Ask naturally: **“Let's store this proposal so we can +access it later.”** The agent prepares an immutable approval card; only the +signed-in owner can approve publication and its capped testnet payment. + +The card stays beside its original response. Later messages continue below it, +and saved conversations survive reloads. Use **Agents → Agent Chat** in the +navigation bar to try the flow. + +### Links and deployment boundaries + +| Resource | Location | +| --- | --- | +| Integration branch | `feat/toju-agent-storage` in both [QMeshPy](https://github.com/QMeshPy/distributed-quantum/tree/feat/toju-agent-storage) and [Winter-Soren](https://github.com/Winter-Soren/distributed-quantum-services/tree/feat/toju-agent-storage) | +| Toju API readiness | [Render health](https://toju-devcon-testnet.onrender.com/health) | +| Toju API liveness | [Render liveness](https://toju-devcon-testnet.onrender.com/health/live) | +| Full Toju server source | [Winter-Soren/afara, devcon/testnet](https://github.com/Winter-Soren/afara/tree/devcon/testnet), forked from the official Toju server | +| Hosting instructions | [Full Render / Neon / QStash guide](https://github.com/QMeshPy/documentations/blob/main/toju/toju-render-testnet-guide.md) | +| Video demo | [Watch the Toju agent storage demo on Google Drive](https://drive.google.com/file/d/1PZ6MHxBd5_XOqMTGnrS6OEB_tGlL6uGX/view?usp=drive_link) | + +**QuantumNodes runs locally for this demo. Render hosts the separate full Toju +Node/Express server**, including its storage, payment, database and lifecycle-job +routes. `backend/toju-client` is a payment/verification worker, not a replacement +storage server. The retired `services/toju-demo` prototype must not be deployed. + +Agent storage payments use **Base Sepolia** (`eip155:84532`) and faucet USDC. +The server's separate Solana route uses **Solana testnet**, with program +`CSXnfQsFWxdPB5pnS73TQDA6ivK6kcFnRwtt6TgFquxH`. This agent demo does not execute +a Solana payment. Neither route automatically falls back to mainnet. + +### Run the integration locally + +Use Python 3.12 (the tested runtime), `uv`, **Node 22**, Bun, and a running local +MongoDB. Start with the backend/frontend setup below. The frontend uses Bun; +the isolated payment worker has its own npm lockfile. + +```sh +# From the repository root +cd backend +uv sync --locked +cd toju-client +npm ci +cd ../../frontend +bun install +``` + +Create ignored `backend/.env` and `frontend/.env.local` files from their example +settings. Merge the values below into your local files; do not overwrite existing +credentials. Generate **one** bridge secret with `openssl rand -hex 32` and use +that same value on both sides. Generate a separate Better Auth secret. + +Backend configuration (`backend/.env`): + +```dotenv +QB2_MONGODB_TARGET=local +QB2_MONGODB_LOCAL_URI=mongodb://127.0.0.1:27017 +QB2_MONGODB_DATABASE=qds +AGENT_AUTH_BRIDGE_SECRET= + +AGENT_LLM_PROVIDER=openrouter +OPENROUTER_API_KEY= +OPENROUTER_MODEL=nvidia/nemotron-3.5-lightning:free + +CDP_API_KEY_ID= +CDP_API_KEY_SECRET= +CDP_WALLET_SECRET= +NETWORK=base-sepolia + +TOJU_API_URL=https://toju-devcon-testnet.onrender.com +TOJU_PAYMENT_RECIPIENT=0xbB2Fc1cc295b53a75217319eA421256dABB3eacb +TOJU_NODE_BINARY=node +``` + +`TOJU_NODE_BINARY` must resolve to Node 22; use its absolute path if your shell's +default differs. All agent services use the selected `QB2_MONGODB_TARGET` and database; set +`QB2_MONGODB_REMOTE_URI` with target `remote` for Atlas. + +For Groq instead, set `AGENT_LLM_PROVIDER=groq`, `GROQ_API_KEY`, and +`GROQ_MODEL=openai/gpt-oss-20b`. OpenRouter accepts only free model names and sends +zero-price provider limits. For NEAR AI Cloud private inference in a Confidential TEE, +set `AGENT_LLM_PROVIDER=near`, `NEARAI_API_KEY`, and `NEAR_MODEL` (canonical Confidential TEE +model ID from `https://cloud-api.near.ai/v1/model/list` with `attestationSupported=true`); +configure a per-key spend limit in the NEAR dashboard. Each completion round is verified +with `nearai-inference-sdk` and surfaced as an attestation badge (`Model TEE verified`, +`Gateway verified`, or `Unverified`) on the chat message. + +The agent chat UI includes a provider dropdown (Groq / OpenRouter / NEAR) that overrides the env default +per browser; `AGENT_LLM_PROVIDER` remains the server fallback. There is no paid fallback or automatic +rate-limit retry. Free model availability can change. + +Frontend configuration (`frontend/.env.local`): + +```dotenv +NEXT_PUBLIC_BACKEND_URL=http://127.0.0.1:8081 +MONGODB_URI=mongodb://127.0.0.1:27017 +MONGODB_DB_NAME=quantum_platform +BETTER_AUTH_URL=http://localhost:3000 +BETTER_AUTH_SECRET= +AGENT_AUTH_BRIDGE_SECRET= +RESEND_API_KEY= +OTP_EMAIL_FROM= +NEXT_PUBLIC_TRIAL_DISABLED=true +``` + +The trial flag is for local development; it does **not** bypass sign-in or export +ownership checks. Email OTP authentication requires a working Resend sender. +MongoDB stores chat/agent state and authentication; **Neon belongs to the Toju +server**, not these MongoDB settings. Provider keys, the auth bridge and CDP +signing credentials are server-only; never prefix them with `NEXT_PUBLIC_`. + +Start the two processes in separate terminals: + +```sh +cd backend +uv run quantum-backend --host 127.0.0.1 --port 8081 +``` + +```sh +cd frontend +bun run dev +``` + +Sign in at [localhost:3000](http://localhost:3000), open **Agents → Agent Chat**, +and use your own agent and its CDP Base Sepolia wallet. Fund that wallet with +test USDC using the [Circle faucet](https://faucet.circle.com/) before approving +an export. Do not use mainnet funds or another user's wallet. + +### Try a draft, approval and receipt + +1. Ask: `Draft a short Bell-pair experiment proposal. Clearly label it as a proposed experiment that has not been run. Show only the draft for now.` +2. Then ask: `Let's store this proposal so we can access it later.` +3. Review the exact content, public-IPFS notice, recipient, SHA-256, 30-day + retention and price. Click **Cancel** to decline or **Approve & store** once + to publish. A direct `yes` applies only to the current pending approval. +4. Wait for **Stored and retrieved. SHA-256 verified.** Open the artifact and + testnet payment links. The worker checks the on-chain USDC Transfer and the + downloaded bytes; an upload response alone is not treated as proof. +5. Reload the saved chat to see the persisted receipt. For an uncertain outcome, + use **Check receipt (no new payment)**; never repay automatically. + +The model can also propose exporting an owned workflow run ID. The backend reads +that user's stored run snapshots and exports selected metadata/results rather +than trusting model-invented results. Exports are limited to 256 KiB, approval +expires after 15 minutes, and the current payment policy caps each export at +24 micro-USDC (`0.000024` test USDC). Changing server pricing requires reviewing +the client's matching policy; mismatched quotes fail closed. + +### Host your own full Toju server on Render + +Follow the [complete hosting guide](https://github.com/QMeshPy/documentations/blob/main/toju/toju-render-testnet-guide.md), which +includes every environment variable, pricing initialization, migrations and +signed QStash schedule. The reproduced installation uses Render Free, Neon Free +and QStash Free; check your account's quotas before inviting additional users. + +- Deploy the **afara server fork**, branch `devcon/testnet`, with Render's + root directory blank. Do not deploy this entire QuantumNodes repository as Toju. +- Use Node 22, `NODE_ENV=staging`, `X402_NETWORK=sepolia`, + `FILECOIN_NETWORK=calibration`, and a Solana testnet RPC/IDL. +- Provide your own Neon `DATABASE_URL`, Pinata JWT/gateway, Resend sender/key, + QStash current/next signing keys, admin API secret and Base Sepolia recipient. + Keep all secrets in Render's environment settings. +- Build: `corepack pnpm --filter server... install --frozen-lockfile --prod=false && corepack pnpm --dir server run build` +- Start: `corepack pnpm --dir server exec tsx scripts/migrate.ts staging && node server/dist/index.js` +- Set Render's health-check path to `/health/live`. Check `/health` separately + for database readiness and initialize pricing as documented in the guide. +- Configure the five signed lifecycle/usage jobs described in the guide. + Do not schedule `/jobs/destroy`. +- Point your QuantumNodes backend's `TOJU_API_URL` at your new service and set + `TOJU_PAYMENT_RECIPIENT` to its exact `BASE_USDC_WALLET` public address. + +The demo endpoint above is a shared test service, not a production availability +guarantee. Its hosted `/health` returning healthy does not by itself verify +payment settlement, IPFS retrieval or lifecycle cleanup. + +### Verification and known limits + +The earlier **DevCon Demo** draft completed a testnet payment and verified +retrieval: [public report](https://gateway.pinata.cloud/ipfs/bafybeiaw3wgnhn5yort3hvvzohvyd6ms43xmjat4qhj4jaazr3e2rnuvie/report.md) +and [Base Sepolia transaction](https://sepolia.basescan.org/tx/0x4b9e24bb741676455f1b5aa611d7459bd2b6fbf8666bf8353a2eca84602fa1fe). +A later approved draft was pinned but settlement failed; its uncertain receipt +is preserved. The exact historical settlement failure was not captured. + +Run the focused checks without consuming model quota or making payments: + +```sh +cd backend +uv run pytest tests/services/test_agent_bridge.py tests/services/test_groq_agent.py tests/services/test_toju_storage.py +cd toju-client +npm test +cd ../../frontend +bun run tsc --noEmit +``` + +The storage test uses a temporary local MongoDB database and removes only that +test database. The provider/payment calls in these checks are mocked. Live +coverage includes one completed export, receipt recovery and a signed QStash +usage snapshot. A full quantum experiment, Solana/Filecoin payments, renewal +email flow and time-based deletion have not been verified end to end. Public +IPFS copies may remain after pin retention ends. Free-tier cold starts, quotas +and external settlement failures remain demo limitations. + +--- diff --git a/repository/CONTEXT.md b/repository/CONTEXT.md new file mode 100644 index 0000000..666b892 --- /dev/null +++ b/repository/CONTEXT.md @@ -0,0 +1,219 @@ +# CONTEXT + +Project context for contributors working in this repository. + +Covers: what the platform is, how the code is organized, where the important code lives, and caveats to know before making changes. + +Last verified: May 2026. + +--- + +## What This Platform Is + +A research-oriented platform that treats quantum-style operations as **discoverable network services**, orchestrated through a Python FastAPI coordinator over real `py-libp2p`, exposed through a Next.js operator console, and persisted to Postgres + MongoDB. + +A secondary research workflow uses **QAOA-based portfolio optimization**, demonstrating where quantum computing gains a scaling advantage over classical algorithms for financial optimization problems. + +## What This Platform Is Not + +- A real quantum hardware control stack (uses Qiskit statevector simulation) +- A production multi-tenant platform (auth is dev-mode stubs) +- A multi-coordinator consensus system +- A finished bring-your-own-node marketplace + +Treat it as a serious proof-of-concept orchestration platform with a growing operator console. + +--- + +## Directory Structure + +| Directory | Role | Tech | +|---|---|---| +| `backend/` | Primary backend | Python 3.11, FastAPI, py-libp2p (Trio), Qiskit, SQLAlchemy (Postgres), Beanie (MongoDB) | +| `frontend/` | Primary frontend | Next.js 16, React 19, TypeScript, Tailwind 4, shadcn/ui | +| `docs/` | Documentation corpus | Markdown | +| `dataset/` | S&P 500 benchmark data | CSV | +| `deploy/` | Caddy + deployment config | Caddyfile | +| `scripts/` | Utility scripts | Python / Shell | + +### Root Files + +| File | Purpose | +|---|---| +| `docker-compose.yaml` | Runs `backend` + `frontend` + Caddy. Uses Neon Postgres and Atlas MongoDB by default. | +| `DEPLOYMENT-MANUAL.md` | EC2 + Docker + Caddy + Vercel deployment runbook | +| `.env.example` | Template for Docker/deployment env vars | +| `deploy/Caddyfile` | Reverse proxy config | + +--- + +## Backend Architecture + +Package: `backend/src/quantum_backend_v2/` + +### Entrypoints + +- CLI: `quantum_backend_v2.main:main` (registered as `quantum-backend` console script) +- Bootstrap: `bootstrap/application.py` → `create_application()` builds the FastAPI app +- Lifespan: `api/app.py` handles startup (init persistence, start discovery, run recovery) and shutdown + +### Layer Map + +| Layer | Path | Responsibility | +|---|---|---| +| **API** | `api/routers/`, `api/models/`, `api/deps/`, `api/errors/` | FastAPI routers, request/response models, auth deps, error contracts | +| **Application** | `application/` | Business logic: `CircuitJobService`, `FinancialJobService`, `EnrollmentService`, distributed statevector execution | +| **Identity** | `identity/` | User roles (ADMIN/OPERATOR/DEVELOPER/VIEWER), trust tiers, token claims, API keys | +| **Persistence** | `persistence/postgres.py` | SQLAlchemy ORM: users, enrollments, workflow runs, execution plans, financial jobs, reservation/execution events (append-only) | +| **Persistence** | `persistence/mongodb.py` | Beanie documents: peer capabilities, topology projections, benchmark results, provenance bundles | +| **Persistence** | `persistence/local_log.py` | JSONL append-only peer log (fsync) | +| **Libp2p** | `libp2p/` | Real py-libp2p host (Ed25519), GossipSub pubsub, stream RPC, Trio thread bridged to asyncio, embedded dev worker swarm | +| **Discovery** | `discovery/` | `DiscoveryService` + `PeerRegistry`: drain pubsub events → upsert MongoDB projections, stale-peer TTL, enrollment visibility | +| **Protocols** | `protocols/` | Wire schemas: execution, reservation, quality, peersync | +| **Reservations** | `reservations/` | Event-sourced state machine: REQUESTED → ACCEPTED → COMMITTED → CANCELLED/EXPIRED/REJECTED | +| **Runtime** | `runtime/` | Execution state machine, crash recovery on startup | +| **Quality** | `quality/` | Service quality catalog: transpiles each service circuit against Qiskit BasicSimulator for fidelity | +| **Workflows** | `workflows/` | Benchmark models and service | + +### Configuration + +All config flows through `QB2_*` environment variables → `AppSettings.from_env()` Pydantic model. + +| Variable | Purpose | +|---|---| +| `QB2_ENVIRONMENT` | development / staging / production / test | +| `QB2_API_HOST` / `QB2_API_PORT` | Bind address (default 0.0.0.0:8081) | +| `QB2_AUTH_REQUIRED` | Auth toggle (false for local dev) | +| `QB2_POSTGRES_TARGET` | `local` or `neon` | +| `QB2_POSTGRES_LOCAL_DSN` / `QB2_POSTGRES_NEON_*_DSN` | Postgres connection strings | +| `QB2_MONGODB_TARGET` | `local` or `remote` | +| `QB2_MONGODB_*_URI` | MongoDB connection strings | +| `QB2_LIBP2P_ENABLED` | Enable/disable real libp2p | +| `QB2_LIBP2P_DEV_SERVICE_PEER_COUNT` | Embedded worker peers (default 4) | + +### API Surface (28+ endpoints) + +**System**: `GET /`, `GET /api/v1/health`, `GET /api/v1/ready` + +**Bootstrap**: `GET /api/v1/bootstrap/libp2p`, `GET /api/v1/bootstrap/libp2p/runtime` + +**Discovery**: `GET /api/v1/discovery/peers`, `GET /api/v1/discovery/peers/{peer_id}`, `GET /api/v1/discovery/topology` + +**Enrollment**: `POST /api/v1/enrollment/peers`, `GET /api/v1/enrollment/peers`, `POST /api/v1/enrollment/peers/{peer_id}/action` + +**Circuits**: `POST /api/v1/circuits/submit` + +**Jobs**: `GET /api/v1/jobs`, `GET /api/v1/jobs/{job_id}` + +**Plans**: `GET /api/v1/plans/{plan_id}` + +**Services**: `GET /api/v1/services` + +**Finance**: `POST /api/v1/finance/submit`, `GET /api/v1/finance/{job_id}`, `GET /api/v1/finance/{job_id}/comparison`, `GET /api/v1/finance` + +**Workflows**: `POST /api/v1/workflows/runs`, `GET /api/v1/workflows/runs/{run_id}`, `POST /api/v1/workflows/benchmarks`, `GET /api/v1/workflows/benchmarks/{benchmark_id}` + +**Reservations**: `POST /api/v1/reservations`, `GET /api/v1/reservations/{reservation_id}`, `POST /api/v1/reservations/{reservation_id}/cancel` + +### Quantum Execution Flow + +1. Client submits OpenQASM to `POST /api/v1/circuits/submit` +2. `CircuitJobService` creates a `WorkflowRunRecord` (QUEUED) +3. Background: waits for service peers → compiles execution plan → distributes fragments via libp2p RPC → streams results → assembles final quantum state +4. Status: QUEUED → COMPILING → EXECUTING → COMPLETED / FAILED +5. Qiskit builds the quantum result: counts, probabilities, statevector, Bloch vectors, entanglement entropy, density matrices, fidelity, observable expectations + +### Quantum Gate/Service Types (11) + +`hadamard`, `cnot`, `cz`, `controlled_unitary`, `programmable_gate`, `qft`, `teleportation`, `bell_pair`, `syndrome_extraction`, `distillation`, `measurement_feedforward` + +### Running Locally + +```bash +cd backend +make install # uv sync --extra dev +make run # scripts/demo-start.sh +make run-clean # flush runtime artifacts, then start +make test # uv run pytest +make lint # ruff + mypy +``` + +--- + +## Frontend Architecture + +Framework: Next.js 16 (App Router), React 19, TypeScript. + +### Routing + +| Route | Description | +|---|---| +| `/dashboard` | 3D network graph, stats, charts, data table | +| `/runs` | Unified run list (circuit + financial) | +| `/runs/new` | Visual circuit builder + OpenQASM editor | +| `/runs/[runId]` | Run detail, quantum analysis, fragment flow | +| `/finance` | CSV upload, portfolio optimization, quantum-vs-classical comparison | + +### BFF Proxy Pattern + +The browser never calls the Python backend directly. Next.js API routes (`src/app/api/`) proxy all requests: + +- `src/lib/backend-client.ts` — server-only HTTP client (reads `QUANTUM_BACKEND_URL`) +- `src/proxy.ts` — proxy middleware +- `src/lib/*-transformers.ts` — reshape backend snake_case to frontend camelCase + +### Running Locally + +```bash +cd frontend +npm install +npm run dev # http://localhost:3000 +``` + +Set `QUANTUM_BACKEND_URL` if backend is not at `http://127.0.0.1:8081`. + +--- + +## Docker Compose (Full Stack) + +```bash +cp .env.example .env +docker compose up --build +``` + +Services: `backend` (port 8081), `frontend` (port 3000), `caddy` (ports 80/443). + +--- + +## Important Caveats + +1. **Trio/asyncio bridge**: The libp2p layer runs in a Trio thread, bridged to asyncio via `queue.SimpleQueue`. Do not mix Trio and asyncio primitives. + +2. **No WebSockets**: Both backend and frontend use REST polling. No WS endpoints exist. + +3. **Auth is dev-mode**: `QB2_AUTH_REQUIRED=false` by default. Production JWT is a stub. Do not build features that assume real auth exists. + +4. **Embedded dev swarm**: The libp2p "network" in local dev is a coordinator + N worker peers all running in the same process — a controlled demo fabric, not an open heterogeneous node ecosystem. + +5. **Quantum modeling simplifications**: Teleportation = ancilla-free SWAP. Syndrome extraction and distillation = orchestration steps (no stabilizer measurement). Fidelity = consistency reference against ideal compiled state, not hardware tomography. + +6. **uv, not pip**: backend uses `uv` for Python package management. Use `uv sync` and `uv run`. + +7. **npm**: frontend uses npm. Use `npm install` and `npm run dev`. + +--- + +## Where To Start By Task + +| Task | Entry Point | +|---|---| +| Backend API or orchestration | `backend/src/quantum_backend_v2/api/routers/` → `application/` | +| Circuit parsing or planning | `backend/src/quantum_backend_v2/planning/` and `application/distributed_statevector.py` | +| Libp2p or peer discovery | `backend/src/quantum_backend_v2/libp2p/` → `discovery/` | +| Persistence or data models | `backend/src/quantum_backend_v2/persistence/postgres.py` and `persistence/mongodb.py` | +| Alembic migrations | `backend/alembic/` | +| Frontend pages or components | `frontend/src/app/(main)/` for pages, `frontend/src/components/` for components | +| Frontend API layer | `frontend/src/app/api/` and `frontend/src/lib/backend-client.ts` | +| Frontend state or hooks | `frontend/src/store/` and `frontend/src/hooks/` | +| Docker or deployment | `docker-compose.yaml`, `backend/Dockerfile`, `frontend/Dockerfile`, `DEPLOYMENT-MANUAL.md` | +| Docs | `docs/README.md` | diff --git a/repository/DEPLOYMENT-MANUAL.md b/repository/DEPLOYMENT-MANUAL.md new file mode 100644 index 0000000..a72c9fa --- /dev/null +++ b/repository/DEPLOYMENT-MANUAL.md @@ -0,0 +1,1087 @@ +# Production Deployment Manual + +This manual explains how to deploy the platform with frontend on Vercel and backend on AWS Lightsail, with separate managed databases (Neon PostgreSQL + MongoDB Atlas). + +The deployed stack is: + +- `frontend`: Next.js app on Vercel with global CDN +- `backend`: FastAPI + libp2p on AWS Lightsail in Docker +- `caddy`: reverse proxy and HTTPS terminator on Lightsail +- `neon`: PostgreSQL database (managed) +- `mongodb atlas`: MongoDB database (managed) + +Hostname layout: + +- Production frontend: `https://distributed-quantum.com` (Vercel) +- Production backend API: `https://api.distributed-quantum.com` (Lightsail) +- Local frontend: `http://localhost:3000` +- Local backend API: `http://localhost:8081` + +## Table Of Contents + +- [Architecture](#architecture) +- [Prerequisites](#prerequisites) +- [Part 1: Setup Databases](#part-1-setup-databases) +- [Part 2: Deploy Backend to Lightsail](#part-2-deploy-backend-to-lightsail) +- [Part 3: Configure DNS](#part-3-configure-dns) +- [Part 4: Deploy Frontend to Vercel](#part-4-deploy-frontend-to-vercel) +- [Part 5: Verify Integration](#part-5-verify-integration) +- [Environment Variables Reference](#environment-variables-reference) +- [Updating the Deployment](#updating-the-deployment) +- [Monitoring and Maintenance](#monitoring-and-maintenance) +- [Troubleshooting](#troubleshooting) +- [Cost Breakdown](#cost-breakdown) + +--- + +## Architecture + +``` +Internet + ↓ +┌──────────────────┬─────────────────┐ +│ │ │ +Vercel AWS Lightsail +(Frontend) (Backend + Caddy) +distributed- api.distributed- +quantum.com quantum.com +│ │ +Next.js App Docker Compose: +Better Auth - FastAPI backend + - Caddy (HTTPS) + - libp2p networking + │ + ┌────┴────┐ + │ │ + Neon MongoDB + (Postgres) (Atlas) + managed managed +``` + +Key architectural decisions: + +- **Frontend on Vercel**: Automatic deployments, global CDN, zero-config scaling +- **Backend on Lightsail**: Single VM deployment, Docker Compose, predictable costs +- **Managed Databases**: Neon (PostgreSQL) and MongoDB Atlas for reliability +- **Caddy for HTTPS**: Automatic Let's Encrypt certificates, reverse proxy + +--- + +## Prerequisites + +Before starting, gather these accounts and credentials: + +### Required Accounts + +1. **AWS Account** - for Lightsail instance +2. **Vercel Account** - for frontend hosting (free tier works) +3. **Neon Account** - for PostgreSQL database (free tier works) +4. **MongoDB Atlas Account** - for MongoDB database (free tier works) +5. **Resend Account** - for email OTP (free tier: 3000 emails/month) +6. **Domain** - registered domain with DNS access + +### Required Tools (Local) + +- Git +- SSH client +- Text editor +- cURL (for testing) + +### Optional Services + +- **Lighthouse Storage** - IPFS pinning for VAULT feature +- **Pinata** - Alternative IPFS pinning +- **Sentry** - Error tracking (recommended for production) + +--- + +## Part 1: Setup Databases + +### 1.1 Neon PostgreSQL + +1. **Sign up**: Visit [neon.tech](https://neon.tech) and create account +2. **Create project**: + - Click "New Project" + - Name: `quantum-backend-prod` + - Region: Choose closest to your users (e.g., AWS us-east-1) + - Click "Create Project" + +3. **Get connection strings**: + - Go to project dashboard + - Click "Connection Details" + - Copy **Pooled connection** string (for application) + - Copy **Direct connection** string (for migrations) + - Format: `postgresql+asyncpg://user:pass@host/dbname?sslmode=require` + +4. **Save credentials** (you'll need these later): + ``` + Pooled: postgresql+asyncpg://neondb_owner:xxx@ep-xxx-pooler.neon.tech/neondb?sslmode=require + Direct: postgresql+asyncpg://neondb_owner:xxx@ep-xxx.neon.tech/neondb?sslmode=require + ``` + +### 1.2 MongoDB Atlas + +1. **Sign up**: Visit [mongodb.com/cloud/atlas](https://www.mongodb.com/cloud/atlas) +2. **Create cluster**: + - Click "Build a Database" + - Choose "M0 Free" tier (512MB, good for starting) + - Provider: AWS + - Region: Choose closest to Lightsail region + - Cluster name: `quantum-cluster` + - Click "Create" + +3. **Create database user**: + - Go to "Database Access" (left sidebar) + - Click "Add New Database User" + - Username: `quantum_user` + - Password: Generate strong password (save it!) + - Database User Privileges: "Read and write to any database" + - Click "Add User" + +4. **Configure network access**: + - Go to "Network Access" + - Click "Add IP Address" + - For testing: Click "Allow Access from Anywhere" (0.0.0.0/0) + - For production: Add your Lightsail instance IP later + - Click "Confirm" + +5. **Get connection URI**: + - Go to "Database" → Your cluster + - Click "Connect" + - Choose "Connect your application" + - Driver: Python, Version: 3.11 or later + - Copy connection string + - Format: `mongodb+srv://quantum_user:PASSWORD@cluster.mongodb.net/?appName=quantum` + - **Replace `` with your actual password** + +6. **Save URI** (you'll need this later): + ``` + mongodb+srv://quantum_user:YOUR_PASSWORD@cluster.mongodb.net/?appName=quantum-backend + ``` + +### 1.3 Resend (Email Service) + +1. **Sign up**: Visit [resend.com](https://resend.com) +2. **Get API key**: + - Go to "API Keys" + - Click "Create API Key" + - Name: `quantum-backend-prod` + - Permission: "Sending access" + - Click "Create" + - **Copy the API key immediately** (shown only once) + +3. **Save API key**: + ``` + re_xxxxxxxxxxxxxxxxxxxxxxxxxxxx + ``` + +4. **Verify domain** (optional but recommended): + - Go to "Domains" + - Add your domain + - Add DNS records as shown + - Wait for verification + +--- + +## Part 2: Deploy Backend to Lightsail + +### 2.1 Create Lightsail Instance + +1. **Login**: Go to [lightsail.aws.amazon.com](https://lightsail.aws.amazon.com) + +2. **Create instance**: + - Click "Create instance" + - **Instance location**: Choose region closest to users (e.g., Virginia) + - **Platform**: Linux/Unix + - **Blueprint**: OS Only → Ubuntu 22.04 LTS + - **Instance plan**: + - Minimum: $20/month (2 GB RAM, 2 vCPUs, 60 GB SSD) + - Recommended: $40/month (4 GB RAM, 2 vCPUs, 80 GB SSD) + - **Instance name**: `quantum-backend-prod` + - Click "Create instance" + +3. **Wait for instance** to start (~2 minutes) + +### 2.2 Configure Networking + +1. **Create static IP**: + - Go to instance → Networking tab + - Click "Create static IP" + - Name: `quantum-backend-ip` + - Click "Create" + - **Note the IP address**: `54.123.45.67` (example) + +2. **Configure firewall**: + - Still in Networking tab + - IPv4 Firewall rules should include: + - SSH: TCP 22 (already exists) + - HTTP: TCP 80 (add if missing) + - HTTPS: TCP 443 (add if missing) + - Custom TCP: 4011 (add - for libp2p) + - Click "Add rule" for any missing + +### 2.3 Connect and Setup Server + +1. **SSH into instance**: + ```bash + # Using Lightsail console (browser-based) + # Click "Connect using SSH" button + + # OR using terminal with downloaded key + chmod 400 ~/Downloads/LightsailDefaultKey-us-east-1.pem + ssh -i ~/Downloads/LightsailDefaultKey-us-east-1.pem ubuntu@54.123.45.67 + ``` + +2. **Update system**: + ```bash + sudo apt-get update + sudo apt-get upgrade -y + ``` + +3. **Install Docker**: + ```bash + curl -fsSL https://get.docker.com -o get-docker.sh + sudo sh get-docker.sh + sudo usermod -aG docker ubuntu + sudo systemctl enable docker + sudo systemctl start docker + + # Re-login for group changes + exit + # SSH back in + ``` + +4. **Verify Docker**: + ```bash + docker --version + docker ps + ``` + +5. **Install Docker Compose**: + ```bash + sudo apt-get install docker-compose-plugin -y + docker compose version + ``` + +6. **Install Git**: + ```bash + sudo apt-get install git -y + ``` + +### 2.4 Clone Repository and Configure + +1. **Clone repository**: + ```bash + cd ~ + git clone https://github.com/YOUR_USERNAME/nodes-quantum-gates.git + cd nodes-quantum-gates + ``` + + If private repository: + ```bash + # Generate SSH key + ssh-keygen -t ed25519 -C "lightsail-deploy" + cat ~/.ssh/id_ed25519.pub + # Add to GitHub → Settings → SSH Keys + + # Then clone + git clone git@github.com:YOUR_USERNAME/nodes-quantum-gates.git + ``` + +2. **Configure backend environment**: + ```bash + cd ~/nodes-quantum-gates + nano backend/.env + ``` + + Paste and update with your credentials: + ```env + # App Configuration + QB2_ENVIRONMENT=production + QB2_SERVICE_NAME=quantum-backend + QB2_API_HOST=0.0.0.0 + QB2_API_PORT=8080 + QB2_LOG_LEVEL=INFO + QB2_JSON_LOGS=true + QB2_AUTH_REQUIRED=false + + # PostgreSQL (Neon) - PASTE YOUR CREDENTIALS + QB2_POSTGRES_TARGET=neon + QB2_POSTGRES_NEON_POOLED_DSN=postgresql+asyncpg://neondb_owner:PASS@ep-xxx-pooler.neon.tech/neondb?sslmode=require + QB2_POSTGRES_NEON_DIRECT_DSN=postgresql+asyncpg://neondb_owner:PASS@ep-xxx.neon.tech/neondb?sslmode=require + QB2_POSTGRES_DATABASE=qds + QB2_POSTGRES_ECHO=false + QB2_POSTGRES_POOL_PRE_PING=true + + # MongoDB Atlas - PASTE YOUR URI + QB2_MONGODB_TARGET=remote + QB2_MONGODB_REMOTE_URI=mongodb+srv://quantum_user:PASS@cluster.mongodb.net/?appName=quantum-backend + QB2_MONGODB_DATABASE=qds + QB2_MONGODB_SERVER_SELECTION_TIMEOUT_MS=5000 + + # Peer Log + QB2_PEER_LOG_DIR=/workspace/backend/quantum-backend/peer-logs + QB2_PEER_ID=qb2-production-peer + QB2_PEER_LOG_FSYNC=true + + # Libp2p + QB2_LIBP2P_ENABLED=true + QB2_LIBP2P_PEER_ID=qb2-production-peer + QB2_LIBP2P_LISTEN_MULTIADDRS=/ip4/0.0.0.0/tcp/4011 + QB2_LIBP2P_ADVERTISE_MULTIADDRS= + QB2_LIBP2P_BOOTSTRAP_PEERS= + QB2_LIBP2P_RENDEZVOUS_NAMESPACE=quantum-backend + QB2_LIBP2P_PEERSTORE_PATH=/workspace/backend/quantum-backend/libp2p/peerstore.sqlite3 + QB2_LIBP2P_ACTIVATE_LISTENERS=true + QB2_LIBP2P_DEV_SERVICE_PEER_COUNT=0 + QB2_LIBP2P_DEV_SERVICE_BASE_PORT=4021 + ``` + + Save and exit (Ctrl+X, Y, Enter) + +3. **Configure Docker Compose environment**: + ```bash + nano .env + ``` + + Update with your domain: + ```env + # Production domains + API_DOMAIN=api.distributed-quantum.com + FRONTEND_DOMAIN=https://distributed-quantum.com + + # Legacy (keep for now) + CADDY_FRONTEND_SITE_ADDRESS=distributed-quantum.com + CADDY_API_SITE_ADDRESS=api.distributed-quantum.com + LEGACY_FRONTEND_PORT=3003 + ``` + + Save and exit + +4. **Create required directories**: + ```bash + mkdir -p backend/quantum-backend/peer-logs + mkdir -p backend/quantum-backend/libp2p + ``` + +### 2.5 Deploy Backend + +1. **Start services**: + ```bash + cd ~/nodes-quantum-gates + docker compose up -d + ``` + +2. **Check status**: + ```bash + docker compose ps + ``` + + Expected output: + ``` + NAME IMAGE STATUS + backend ... Up (healthy) + caddy caddy:2.10 Up + ``` + +3. **View logs**: + ```bash + docker compose logs -f + ``` + + Press Ctrl+C to exit logs + +4. **Test backend locally**: + ```bash + curl http://localhost:8080/api/v1/health + ``` + + Expected: `{"status":"healthy",...}` + +### 2.6 Run Database Migrations + +```bash +docker compose exec backend alembic upgrade head +docker compose exec backend alembic current +``` + +### 2.7 Setup Auto-Start on Reboot + +1. **Create systemd service**: + ```bash + sudo nano /etc/systemd/system/quantum-backend.service + ``` + + Paste: + ```ini + [Unit] + Description=Quantum Backend Service + Requires=docker.service + After=docker.service + + [Service] + Type=oneshot + RemainAfterExit=yes + WorkingDirectory=/home/ubuntu/nodes-quantum-gates + ExecStart=/usr/bin/docker compose up -d + ExecStop=/usr/bin/docker compose down + User=ubuntu + Group=docker + + [Install] + WantedBy=multi-user.target + ``` + +2. **Enable service**: + ```bash + sudo systemctl daemon-reload + sudo systemctl enable quantum-backend.service + sudo systemctl start quantum-backend.service + ``` + +--- + +## Part 3: Configure DNS + +### 3.1 Add DNS Records + +In your DNS provider (Cloudflare, Route53, etc.): + +| Type | Name | Value | TTL | +|------|------|-------|-----| +| A | api | 54.123.45.67 (your Lightsail IP) | Auto | +| A | @ | (wait - will set for Vercel) | Auto | + +**Important**: +- `api.distributed-quantum.com` → points to Lightsail +- `distributed-quantum.com` → will point to Vercel (set later) + +### 3.2 Wait for DNS Propagation + +```bash +# Check DNS (from your local machine) +nslookup api.distributed-quantum.com + +# Should return your Lightsail IP +``` + +Propagation usually takes 5-30 minutes. + +### 3.3 Test HTTPS + +Once DNS propagates, Caddy automatically gets SSL certificate: + +```bash +# From your local machine +curl https://api.distributed-quantum.com/api/v1/health +``` + +If certificate error, wait a few minutes. Caddy needs DNS fully propagated. + +--- + +## Part 4: Deploy Frontend to Vercel + +### 4.1 Prepare Environment Variables + +Before deploying, collect these values: + +```env +# Backend API +NEXT_PUBLIC_BACKEND_URL=https://api.distributed-quantum.com + +# Auth Secret (generate new) +BETTER_AUTH_SECRET= +BETTER_AUTH_URL=https://distributed-quantum.com + +# MongoDB (SAME cluster as backend) +MONGODB_URI=mongodb+srv://quantum_user:PASS@cluster.mongodb.net +MONGODB_DATABASE=distributed-quantum + +# Email (Resend) +RESEND_API_KEY=re_xxxxxxxxxxxxxxxx +OTP_EMAIL_FROM=noreply@distributed-quantum.com + +# Trial config +TRIAL_BYPASS_EMAILS=admin@distributed-quantum.com +NEXT_PUBLIC_TRIAL_BYPASS_EMAILS=admin@distributed-quantum.com +NEXT_PUBLIC_TRIAL_DISABLED=false + +# IPFS (optional - for VAULT feature) +NEXT_PUBLIC_LIGHTHOUSE_KEY= +NEXT_PUBLIC_PINATA_JWT= +``` + +### 4.2 Deploy to Vercel + +1. **Login**: Go to [vercel.com](https://vercel.com) + +2. **Import project**: + - Click "Add New..." → "Project" + - Import your GitHub repository + - **Root Directory**: Select `frontend` + - **Framework Preset**: Next.js (auto-detected) + +3. **Configure build**: + - Build Command: `bun run build` (or leave default) + - Output Directory: `.next` (default) + - Install Command: `bun install` (or `npm install` if Bun not supported) + +4. **Add environment variables**: + - Click "Environment Variables" + - Add each variable from 4.1 above + - Select "Production" environment + - Click "Add" for each + +5. **Deploy**: + - Click "Deploy" + - Wait 3-5 minutes for build + - Note deployment URL: `your-project.vercel.app` + +### 4.3 Configure Custom Domain + +1. **Add domain in Vercel**: + - Go to Settings → Domains + - Add: `distributed-quantum.com` + - Add: `www.distributed-quantum.com` (optional) + +2. **Configure DNS**: + - Follow Vercel's instructions + - Usually add A record: `@` → Vercel's IP + - Or CNAME: `www` → `cname.vercel-dns.com` + +3. **Wait for SSL** (~5 minutes) + - Vercel auto-provisions certificate + - Green checkmark = ready + +4. **Test**: + ```bash + curl https://distributed-quantum.com + open https://distributed-quantum.com + ``` + +### 4.4 Update Backend CORS + +SSH back into Lightsail: + +```bash +ssh ubuntu@54.123.45.67 +cd ~/nodes-quantum-gates +``` + +Backend CORS is handled by Caddy. No additional config needed if you used the updated Caddyfile. + +--- + +## Part 5: Verify Integration + +### 5.1 Backend Health + +```bash +curl https://api.distributed-quantum.com/api/v1/health +``` + +Expected: `{"status":"healthy",...}` + +### 5.2 Frontend Loads + +```bash +open https://distributed-quantum.com +``` + +Should load Next.js app. + +### 5.3 API Integration + +Open browser console on `https://distributed-quantum.com`: + +```javascript +fetch('https://api.distributed-quantum.com/api/v1/health') + .then(r => r.json()) + .then(d => console.log('Backend health:', d)) + .catch(e => console.error('Backend error:', e)) +``` + +Should return health data without CORS errors. + +### 5.4 Authentication Flow + +1. Try signing up/logging in +2. Should receive OTP email +3. Should authenticate successfully + +If issues, check: +- `RESEND_API_KEY` in Vercel +- `MONGODB_URI` in Vercel matches backend's MongoDB +- `BETTER_AUTH_SECRET` is set +- Browser console for errors + +--- + +## Environment Variables Reference + +### Backend (backend/.env) + +**Required**: +```env +QB2_POSTGRES_TARGET=neon +QB2_POSTGRES_NEON_POOLED_DSN= +QB2_POSTGRES_NEON_DIRECT_DSN= +QB2_MONGODB_TARGET=remote +QB2_MONGODB_REMOTE_URI= +``` + +**Optional but recommended**: +```env +QB2_ENVIRONMENT=production +QB2_LOG_LEVEL=INFO +QB2_LIBP2P_PEER_ID=qb2-production-peer +QB2_LIBP2P_DEV_SERVICE_PEER_COUNT=0 +``` + +**Agent Chat & Private Inference (`AGENT_LLM_PROVIDER`)**: +```env +AGENT_AUTH_BRIDGE_SECRET= +# Explicit provider selection: groq | openrouter | near (no silent fallback) +AGENT_LLM_PROVIDER=groq +GROQ_API_KEY= +GROQ_MODEL=openai/gpt-oss-20b +OPENROUTER_API_KEY= +OPENROUTER_MODEL=nvidia/nemotron-3.5-lightning:free +NEARAI_API_KEY= +NEAR_MODEL= +TOJU_API_URL=https://toju-devcon-testnet.onrender.com +TOJU_PAYMENT_RECIPIENT=0xbB2Fc1cc295b53a75217319eA421256dABB3eacb +TOJU_NODE_BINARY=node +``` + +### Frontend (Vercel Dashboard) + +**Required**: +```env +NEXT_PUBLIC_BACKEND_URL=https://api.distributed-quantum.com +BETTER_AUTH_SECRET=<32+ char random> +BETTER_AUTH_URL=https://distributed-quantum.com +MONGODB_URI= +RESEND_API_KEY= +OTP_EMAIL_FROM=noreply@your-domain.com +``` + +**Optional**: +```env +NEXT_PUBLIC_LIGHTHOUSE_KEY= +NEXT_PUBLIC_PINATA_JWT= +TRIAL_BYPASS_EMAILS=admin@domain.com +NEXT_PUBLIC_TRIAL_BYPASS_EMAILS=admin@domain.com +``` + +### Docker Compose (root .env) + +**Required for production**: +```env +API_DOMAIN=api.distributed-quantum.com +FRONTEND_DOMAIN=https://distributed-quantum.com +``` + +--- + +## Updating the Deployment + +### Update Backend (Lightsail) + +```bash +ssh ubuntu@LIGHTSAIL_IP +cd ~/nodes-quantum-gates +git pull origin main +docker compose down +docker compose up -d --build + +# Run migrations if needed +docker compose exec backend alembic upgrade head +``` + +### Update Frontend (Vercel) + +Automatic on git push to main branch, or: + +1. Vercel Dashboard → Deployments +2. Select latest deployment → "Redeploy" + +### Rollback Backend + +```bash +cd ~/nodes-quantum-gates +git log # Find last working commit +git reset --hard COMMIT_HASH +docker compose down +docker compose up -d --build +``` + +### Rollback Frontend + +1. Vercel Dashboard → Deployments +2. Find working deployment +3. Click "..." → "Promote to Production" + +--- + +## Monitoring and Maintenance + +### Daily Checks + +```bash +# SSH into Lightsail +ssh ubuntu@LIGHTSAIL_IP + +# Check services +docker compose ps + +# Check logs +docker compose logs --tail=100 backend + +# Check resources +docker stats --no-stream +df -h +free -h +``` + +### Health Monitoring Script + +Create on Lightsail: + +```bash +cat > ~/health-check.sh << 'EOF' +#!/bin/bash +HEALTH_URL="http://localhost:8080/api/v1/health" +RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" $HEALTH_URL) + +if [ $RESPONSE -eq 200 ]; then + echo "$(date): Service healthy" +else + echo "$(date): Service unhealthy (HTTP $RESPONSE)" + docker compose restart backend +fi +EOF + +chmod +x ~/health-check.sh +``` + +Add to crontab: + +```bash +crontab -e +# Add: */5 * * * * /home/ubuntu/health-check.sh >> /home/ubuntu/health-check.log 2>&1 +``` + +### Backups + +**Database backups**: +- Neon: Automatic backups included (point-in-time recovery) +- MongoDB Atlas: Automatic backups included + +**Application data**: + +```bash +# Backup peer logs +tar -czf ~/backup-$(date +%Y%m%d).tar.gz \ + -C ~/nodes-quantum-gates/backend/quantum-backend peer-logs/ libp2p/ + +# Keep last 7 days +find ~/backup-*.tar.gz -mtime +7 -delete +``` + +### Log Rotation + +```bash +sudo nano /etc/logrotate.d/quantum-backend +``` + +``` +/home/ubuntu/nodes-quantum-gates/backend/quantum-backend/peer-logs/*.log { + daily + rotate 7 + compress + delaycompress + missingok + notifempty +} +``` + +--- + +## Troubleshooting + +### Backend won't start + +**Check logs**: +```bash +docker compose logs backend +``` + +**Common issues**: +- Database connection: Check DSN in `backend/.env` +- Port conflict: Check if port 8080 in use +- Out of memory: Upgrade instance to $40/month + +**Fix**: +```bash +docker compose down +docker compose up -d --build +``` + +### HTTPS not working (Caddy) + +**Check DNS**: +```bash +nslookup api.distributed-quantum.com +``` + +**Check Caddy logs**: +```bash +docker compose logs caddy +``` + +**Common issues**: +- DNS not propagated: Wait 30 minutes +- Ports not open: Check Lightsail firewall (80, 443) +- Domain doesn't resolve: Check DNS records + +**Force cert renewal**: +```bash +docker compose restart caddy +``` + +### CORS errors in frontend + +**Verify backend responds**: +```bash +curl https://api.distributed-quantum.com/api/v1/health +``` + +**Check CORS headers**: +```bash +curl -i -H "Origin: https://distributed-quantum.com" \ + https://api.distributed-quantum.com/api/v1/health +``` + +Should include: `access-control-allow-origin: https://distributed-quantum.com` + +**If missing**, check Caddyfile configuration and restart: +```bash +docker compose restart caddy +``` + +### Database connection failures + +**Test PostgreSQL**: +```bash +docker compose exec backend python -c " +from sqlalchemy import create_engine +engine = create_engine('YOUR_NEON_DSN') +print('PostgreSQL OK') +" +``` + +**Test MongoDB**: +```bash +docker compose exec backend python -c " +from pymongo import MongoClient +client = MongoClient('YOUR_MONGODB_URI') +print('MongoDB OK') +" +``` + +**Common issues**: +- Wrong credentials: Double-check DSN/URI +- Firewall: MongoDB Atlas - allow Lightsail IP +- SSL required: Ensure `?sslmode=require` in PostgreSQL DSN + +### Frontend build fails (Vercel) + +**Check build logs** in Vercel deployment. + +**Common issues**: +- TypeScript errors: Fix in code +- Environment variables missing: Add in Vercel dashboard +- Out of memory: Usually auto-handled by Vercel + +**Redeploy**: +```bash +git commit --allow-empty -m "trigger rebuild" +git push origin main +``` + +### High memory usage (Lightsail) + +**Check usage**: +```bash +docker stats +free -h +``` + +**Restart services**: +```bash +docker compose restart backend +``` + +**Clean up**: +```bash +docker system prune -f +``` + +**Long-term**: Upgrade to $40/month instance (4GB RAM). + +### Disk space issues + +**Check usage**: +```bash +df -h +docker system df +``` + +**Clean up**: +```bash +docker system prune -a --volumes -f +docker builder prune -a -f +sudo apt-get clean +sudo journalctl --vacuum-time=3d +``` + +**Resize volume** (if needed): +```bash +# Increase in AWS Console first, then: +sudo growpart /dev/nvme0n1 1 +sudo resize2fs /dev/nvme0n1p1 +df -h +``` + +--- + +## Cost Breakdown + +### Minimal Setup (~$30-35/month) + +| Service | Plan | Cost | +|---------|------|------| +| AWS Lightsail | 2GB RAM | $20 | +| Neon PostgreSQL | Free tier | $0 | +| MongoDB Atlas | M0 Free | $0 | +| Vercel | Hobby | $0 | +| Resend | Free tier | $0 | +| Domain | Varies | ~$10-15 | +| **Total** | | **~$30-35** | + +### Recommended Production (~$146/month) + +| Service | Plan | Cost | +|---------|------|------| +| AWS Lightsail | 4GB RAM | $40 | +| Neon PostgreSQL | Pro | $19 | +| MongoDB Atlas | M10 | $57 | +| Vercel | Pro | $20 | +| Resend | Pro | $20 | +| Domain | Varies | ~$10-15 | +| **Total** | | **~$166** | + +### Cost Optimization Tips + +1. Start with free tiers, upgrade as needed +2. Use Cloudflare free tier for DDoS protection +3. Monitor bandwidth (Lightsail includes 1-3TB) +4. Optimize images and bundles +5. Enable caching aggressively + +--- + +## Security Checklist + +- [x] HTTPS enforced (Caddy + Vercel) +- [x] Security headers configured +- [x] CORS limited to frontend domain +- [x] Database SSL connections +- [x] Firewall configured (ports 22, 80, 443, 4011) +- [ ] SSH key-only authentication +- [ ] Fail2ban installed +- [ ] Cloudflare proxy enabled (optional) +- [ ] Error tracking (Sentry) +- [ ] Uptime monitoring +- [ ] Log aggregation + +### Enable SSH Key-Only + +```bash +sudo nano /etc/ssh/sshd_config + +# Set: +PasswordAuthentication no +PubkeyAuthentication yes + +# Restart +sudo systemctl restart sshd +``` + +### Enable Fail2ban + +```bash +sudo apt-get install fail2ban -y +sudo systemctl enable fail2ban +sudo systemctl start fail2ban +``` + +--- + +## Quick Reference + +### Production URLs + +- Frontend: https://distributed-quantum.com +- Backend: https://api.distributed-quantum.com +- Health: https://api.distributed-quantum.com/api/v1/health +- API Docs: https://api.distributed-quantum.com/docs + +### SSH Access + +```bash +ssh ubuntu@YOUR_LIGHTSAIL_IP +cd ~/nodes-quantum-gates +``` + +### Essential Commands + +```bash +# Check status +docker compose ps + +# View logs +docker compose logs -f + +# Restart +docker compose restart + +# Update +git pull && docker compose up -d --build + +# Migrations +docker compose exec backend alembic upgrade head + +# Health check +curl http://localhost:8080/api/v1/health +``` + +### Emergency + +- **Frontend down**: Vercel → Deployments → Rollback +- **Backend down**: `docker compose restart backend` +- **Database issues**: Check connection strings in `backend/.env` +- **Full restart**: `docker compose down && docker compose up -d` + +--- + +**Deployment complete!** + +Your quantum computing platform is now live on: +- Frontend: Vercel (https://distributed-quantum.com) +- Backend: Lightsail (https://api.distributed-quantum.com) +- Databases: Neon + MongoDB Atlas + +For support, check logs first, then consult troubleshooting section above. diff --git a/repository/PROGRESS.md b/repository/PROGRESS.md new file mode 100644 index 0000000..3ed4181 --- /dev/null +++ b/repository/PROGRESS.md @@ -0,0 +1,348 @@ +# AgentKit Integration Implementation Progress + +**Branch**: `feat/agentkit-integration` +**Start Date**: 2026-05-20 +**Last Updated**: 2026-05-20 +**Design Doc**: `/docs/superpowers/specs/2026-05-20-agentkit-crowdfunding-design.md` + +--- + +## 📊 Overall Progress + +- [x] **Phase 1**: Payment Infrastructure (30/32 tasks - 94%) ✅ +- [x] **Phase 2**: Worker Marketplace (14/18 tasks - 78%) ✅ +- [x] **Phase 3**: AI Research Agents (17/22 tasks - 77%) ✅ +- [x] **Phase 4**: Research Crowdfunding (38/44 tasks - 86%) ✅ + +**Total Progress**: 99/116 tasks (85%) - CODE IMPLEMENTATION COMPLETE! + +**Implementation Status**: All 4 phases are 100% code-complete (32,171 lines of production code, 27 API endpoints, 9 services). +**Test Status**: 25/30 passing (83%) — 5 AgentKit failures need live CDP testnet; all Proposal tests now 100% passing. +**Remaining**: Credentials/infra setup, integration tests, frontend (optional). + +--- + +## 🎯 Phase 1: Payment Infrastructure (IMPLEMENTATION COMPLETE) + +**Status**: 🟢 94% Complete (code done, pending infra setup + unit tests) + +### Setup & Dependencies (3/8) +- [x] Install `coinbase-agentkit` Python SDK +- [x] Install `boto3` for AWS Bedrock +- [x] CDP credentials configured in `.env` +- [ ] Create platform wallet on Base Sepolia testnet +- [ ] Fund platform wallet with testnet ETH + USDC +- [ ] Set up Resend email account +- [ ] Add Resend API key to `.env` +- [ ] Configure AWS Bedrock access + +### MongoDB Schema (6/6) ✅ +- [x] Create `wallets` collection with indexes +- [x] Create `payments` collection with indexes +- [x] Create `worker_pricing` collection with indexes +- [x] Create `research_proposals` collection with indexes +- [x] Create `ai_agents` collection with indexes +- [x] Create `notifications` collection with indexes + +### AgentKit Service (10/10) ✅ +- [x] Create service file with all methods implemented +- [x] Implement `create_wallet()` with Fernet encryption +- [x] Implement `get_balance()` via Web3 +- [x] Implement `transfer_usdc()` with ERC-20 +- [x] Implement `aave_supply()` for escrow +- [x] Implement `aave_withdraw()` for release +- [x] Implement `request_testnet_funds()` faucet +- [x] Implement `_load_wallet()` helper +- [x] Implement `_load_platform_wallet()` helper +- [x] Write unit tests + +### Wallet API (6/6) ✅ +- [x] Create router with all 6 endpoints +- [x] Implement POST /wallet/create +- [x] Implement GET /wallet/balance +- [x] Implement POST /wallet/transfer +- [x] Implement POST /wallet/fund-testnet +- [x] Implement GET /wallet/transactions +- [x] Implement POST /wallet/export + +### Payment Models (2/2) ✅ +- [x] Create `wallet.py` Pydantic models +- [x] Create `payment.py` Pydantic models + +--- + +## 🛍️ Phase 2: Worker Marketplace + +**Status**: 🟢 78% Complete (code done, pending integration tests) + +### Marketplace Service (8/8) ✅ +- [x] Create `marketplace_service.py` +- [x] Implement `register_worker_pricing()` +- [x] Implement `estimate_job_cost()` with OpenQASM parser +- [x] Implement `route_operations()` cheapest worker +- [x] Implement `distribute_payment_to_workers()` +- [x] Implement `update_worker_reputation()` +- [x] Implement `_find_cheapest_worker()` helper +- [x] Implement `_parse_circuit_operations()` parser + +### Marketplace API (4/4) ✅ +- [x] Create router skeleton +- [x] Wire up all endpoints to service +- [x] Add cost estimation to job submission +- [x] Test payment distribution + +### Job Integration (4/4) ✅ +- [x] Enhance circuit_service with cost estimation +- [x] Integrate marketplace routing +- [x] Add payment escrow to job flow +- [x] Add payment distribution on completion + +### Phase 2 Testing (0/2) +- [ ] Unit tests for marketplace service +- [ ] Integration test: job → payment → worker + +--- + +## 🤖 Phase 3: AI Research Agents + +**Status**: 🟢 77% Complete (code done, pending agent model file + integration tests) + +### AI Agent Service (6/6) ✅ +- [x] Create `ai_agent_service.py` +- [x] Implement `create_agent()` with wallet +- [x] Implement `analyze_proposal()` with Claude 3.5 +- [x] Implement `form_coalition()` multi-agent +- [x] Implement `_execute_funding()` with limits +- [x] Implement `_get_daily_remaining()` budget tracking + +### AI Agent API (6/6) ✅ +- [x] Create agents.py router +- [x] Implement POST /agents (create) +- [x] Implement GET /agents (list) +- [x] Implement GET /agents/:id +- [x] Implement PUT /agents/:id/config +- [x] Implement POST /agents/:id/analyze +- [x] Implement GET /agents/:id/spending + +### AWS Bedrock Setup (4/4) ✅ +- [x] Create AWS IAM user +- [x] Generate AWS access keys +- [x] Add credentials to `.env` +- [x] Test Bedrock access + +### Agent Models (1/1) ✅ +- [x] Create `agent.py` Pydantic models + +### Phase 3 Testing (0/5) +- [ ] Test agent creation +- [ ] Test proposal analysis +- [ ] Test autonomous funding +- [ ] Test spending limits +- [ ] Integration test: agent → analyze → fund + +--- + +## 🔬 Phase 4: Research Crowdfunding (THE NOVEL FEATURE!) + +**Status**: 🟢 86% Complete (code done, pending infra keys + integration tests) + +### IPFS Setup (1/3) +- [x] Create IPFS utility +- [ ] Get Web3.Storage API key +- [ ] Test IPFS uploads + +### Notification Service (6/6) ✅ +- [x] Create `notification_service.py` +- [x] Implement `notify_new_proposal()` multi-channel +- [x] Implement `notify_proposal_funded()` +- [x] Implement `notify_payment_received()` +- [x] Implement `send_bulk_email()` Resend API +- [x] Implement email templates (5 templates) + +### Proposal Service (8/8) ✅ THE CROWN JEWEL! +- [x] Create `proposal_service.py` +- [x] Implement `create_proposal()` +- [x] Implement `fund_proposal()` with Aave +- [x] Implement `claim_fragment()` +- [x] Implement `submit_results()` to IPFS +- [x] Implement `_auto_fragment_proposal()` Claude AI +- [x] Implement `_broadcast_new_proposal()` +- [x] Implement `_pay_fragment_researcher()` +- [x] Implement `_release_all_escrow()` + +### Proposal API (8/8) ✅ +- [x] Create proposals.py router skeleton +- [x] Wire up all 8 endpoints +- [x] Test proposal creation +- [x] Test auto-fragmentation +- [x] Test funding flow +- [x] Test fragment claiming +- [x] Test result submission +- [x] Test IPFS publishing + +### Notification API (5/5) ✅ +- [x] Create notifications.py router +- [x] Implement GET /notifications (with pagination and filters) +- [x] Implement PUT /notifications/:id/read +- [x] Implement POST /notifications/preferences +- [x] Implement DELETE /notifications/:id + +### Proposal Models (2/2) ✅ +- [x] Create `proposal.py` Pydantic models +- [x] Create `notification.py` Pydantic models + +### Phase 4 Testing (0/12) +- [ ] Test proposal creation +- [ ] Test auto-fragmentation (Bedrock) +- [ ] Test direct funding +- [ ] Test agent coalition funding +- [ ] Test email notifications +- [ ] Test GossipSub broadcasts +- [ ] Test fragment claiming +- [ ] Test result submission +- [ ] Test Aave escrow +- [ ] Test IPFS publishing +- [ ] Integration test: full lifecycle +- [ ] Integration test: multi-fragment workflow + +--- + +## 🎨 Frontend Implementation (Optional - Not Blocking) + +**Status**: Not started (0/22 tasks) + +Skipped for now - backend can be demoed via API. + +--- + +## 🚀 Deployment Checklist + +**Status**: Not started (0/14 tasks) + +Will start after Phase 1-4 complete. + +--- + +## 📝 Documentation + +**Status**: Complete! ✅ + +### Technical Docs (6/6) ✅ +- [x] Design specification (30+ pages) +- [x] Progress tracker (this file!) +- [x] Context guide for AI handoff +- [x] Proposal service documentation +- [x] Notification service documentation +- [x] AWS Bedrock setup guide + +--- + +## 🧪 Testing Summary + +### Unit Tests (25/30 Passing - 83%) + +**Note**: Failures in AgentKit tests are complex SDK mocking issues (live CDP/Web3 needed). Proposal service tests are now 100% passing after DI refactor + address fixes today. + +- [x] AgentKit service tests (9/14 passing - 64%) — 5 failures need live CDP testnet +- [ ] Marketplace service tests (8 tests) — not yet written +- [ ] AI agent service tests (6 tests) — not yet written +- [x] Proposal service tests (16/16 passing - 100%) ✅ Fixed today! +- [ ] Notification service tests (5 tests) — not yet written + +### Integration Tests (Written, Not Validated End-to-End) +- [x] Agent funding flow test (written) +- [x] Crowdfunding flow test (written) +- [x] Marketplace flow test (written) +- [x] Payment flow test (written) +- [ ] End-to-end validation with live services + +--- + +## 🎯 Success Metrics + +### Phase 1 Metrics +- [x] Wallet creation working (needs CDP keys) +- [ ] USDC transfers functional +- [ ] Transactions visible on Basescan +- [ ] 0 failed transfers + +### Phase 2 Metrics +- [ ] Workers can register pricing +- [ ] Job routing selects cheapest worker +- [ ] Payments distributed correctly + +### Phase 3 Metrics +- [ ] Agents can analyze proposals +- [ ] Autonomous funding works +- [ ] Spending limits enforced + +### Phase 4 Metrics +- [ ] Proposals auto-fragment correctly +- [ ] Funding threshold system works +- [ ] IPFS results publish successfully +- [ ] Email notifications deliver + +--- + +## 🐛 Known Issues & Blockers + +### Current Blockers +- Test pass rate at 83% (25/30). AgentKit 5 failures need live CDP testnet credentials +- Platform wallet not yet created on Base Sepolia +- Missing API keys: Resend, Web3.Storage + +### Completed Agents +1. ✅ MongoDB Schema Setup +2. ✅ Pydantic Models (all 20 models) +3. ✅ Basescan Utility +4. ✅ AgentKit Service (825 lines) +5. ✅ Wallet API Router (558 lines) +6. ✅ AgentKit Service Implementation +7. ✅ Wallet API Wiring +8. ✅ .env.example Template +9. ✅ Marketplace Service (889 lines) +10. ✅ Notification Service (819 lines) +11. ✅ AI Agent Service (842 lines) +12. ✅ IPFS Utility (252 lines) +13. ✅ Proposal Service (1,093 lines) — THE CROWN JEWEL! +14. ✅ Agents API Router (689 lines) +15. ✅ Marketplace API Router (440 lines) +16. ✅ Notifications API Router (437 lines) +17. ✅ Proposals API Router (901 lines) +18. ✅ Unit Tests (AgentKit + Proposal services) +19. ✅ Integration Tests (4 flow tests written) + +--- + +## 📅 Milestones + +### Phase 1 Complete +- [x] Dependencies installed +- [x] MongoDB schema defined +- [x] AgentKit service implemented +- [x] Wallet API functional +- [ ] Demo: USDC transfer on Basescan + +### Phase 2 In Progress +- [x] Marketplace service complete +- [x] API endpoints wired (marketplace_router registered in app.py) +- [ ] Demo: Job routes to cheapest worker + +### Phase 3 In Progress +- [x] AI agent service complete +- [x] Agents API complete +- [x] AWS Bedrock configured +- [ ] Demo: Agent autonomously funds + +### Phase 4 In Progress +- [x] Proposal service complete (THE BIG ONE!) +- [x] Notification service complete +- [x] IPFS utility complete +- [x] API endpoints wired (proposals_router registered in app.py) +- [ ] Demo: Full research lifecycle + +--- + +**Last Updated**: 2026-05-20 +**Next Review**: After each agent completion +**Maintained By**: Soham + Claude + 14 Background Agents 🤖 diff --git a/repository/README.md b/repository/README.md new file mode 100644 index 0000000..d798517 --- /dev/null +++ b/repository/README.md @@ -0,0 +1,74 @@ +# Application repository documents + +These 41 documents moved from the application repository into the documentation +submodule on 2026-11-01. General repository guides retain their original paths +under `repository/`; demo material, design specs, and plans join the existing +`superpowers/` directory. Files named `README.md` remain beside the source code. + +> **Historical status:** Progress trackers, implementation summaries, test results, +> design reviews, and deployment notes record the state when they were written. +> Their completion claims, commands, paths in examples, and test counts were not +> revalidated during this move. Use the application repository's current +> [root README](https://github.com/Winter-Soren/distributed-quantum-services/blob/main/README.md), +> [backend README](https://github.com/Winter-Soren/distributed-quantum-services/blob/main/backend/README.md), +> and [frontend README](https://github.com/Winter-Soren/distributed-quantum-services/blob/main/frontend/README.md) +> for current setup instructions. + +## Frontend guidance + +Read [architecture.md](frontend/architecture.md), [DESIGN.md](frontend/DESIGN.md), +[AGENTS.md](frontend/AGENTS.md), [CLAUDE.md](frontend/CLAUDE.md), and +[SKILL.md](frontend/SKILL.md) for the preserved frontend guidance. They describe +work in the application's `frontend/` directory. Moving `AGENTS.md` and +`CLAUDE.md` out of that directory removes automatic discovery by tools that +search those filenames beside the source; load these documents explicitly. + +## Migration index + +Old paths are relative to the application repository root. New paths below are +relative to this documentation repository; in a full application checkout, +prefix them with `documentations/`. + +| Old application path | New documentation path | +| --- | --- | +| `CONTEXT.md` | [repository/CONTEXT.md](CONTEXT.md) | +| `DEPLOYMENT-MANUAL.md` | [repository/DEPLOYMENT-MANUAL.md](DEPLOYMENT-MANUAL.md) | +| `PROGRESS.md` | [repository/PROGRESS.md](PROGRESS.md) | +| `backend/AWS_BEDROCK_SETUP.md` | [repository/backend/AWS_BEDROCK_SETUP.md](backend/AWS_BEDROCK_SETUP.md) | +| `backend/CHAIN_REGISTRY.md` | [repository/backend/CHAIN_REGISTRY.md](backend/CHAIN_REGISTRY.md) | +| `backend/IPFS_SETUP.md` | [repository/backend/IPFS_SETUP.md](backend/IPFS_SETUP.md) | +| `backend/NOTIFICATION_SERVICE_IMPLEMENTATION.md` | [repository/backend/NOTIFICATION_SERVICE_IMPLEMENTATION.md](backend/NOTIFICATION_SERVICE_IMPLEMENTATION.md) | +| `backend/NOTIFICATION_SERVICE_INTEGRATION_GUIDE.md` | [repository/backend/NOTIFICATION_SERVICE_INTEGRATION_GUIDE.md](backend/NOTIFICATION_SERVICE_INTEGRATION_GUIDE.md) | +| `backend/PAYMENT_LEDGER.md` | [repository/backend/PAYMENT_LEDGER.md](backend/PAYMENT_LEDGER.md) | +| `backend/PROGRESS.md` | [repository/backend/PROGRESS.md](backend/PROGRESS.md) | +| `backend/PROPOSAL_SERVICE_DOCUMENTATION.md` | [repository/backend/PROPOSAL_SERVICE_DOCUMENTATION.md](backend/PROPOSAL_SERVICE_DOCUMENTATION.md) | +| `backend/PROVENANCE.md` | [repository/backend/PROVENANCE.md](backend/PROVENANCE.md) | +| `backend/ROUTER_REGISTRATION_COMPLETE.md` | [repository/backend/ROUTER_REGISTRATION_COMPLETE.md](backend/ROUTER_REGISTRATION_COMPLETE.md) | +| `backend/STARKNET_SESSION_GUIDE.md` | [repository/backend/STARKNET_SESSION_GUIDE.md](backend/STARKNET_SESSION_GUIDE.md) | +| `backend/TEST_RESULTS.md` | [repository/backend/TEST_RESULTS.md](backend/TEST_RESULTS.md) | +| `backend/docs/ens-resolution.md` | [repository/backend/docs/ens-resolution.md](backend/docs/ens-resolution.md) | +| `backend/docs/ens-worker-subnames.md` | [repository/backend/docs/ens-worker-subnames.md](backend/docs/ens-worker-subnames.md) | +| `backend/src/services/README_NOTIFICATION_SERVICE.md` | [repository/backend/src/services/README_NOTIFICATION_SERVICE.md](backend/src/services/README_NOTIFICATION_SERVICE.md) | +| `design-qa.md` | [repository/design-qa.md](design-qa.md) | +| `docs/HELIA_INTEGRATION.md` | [repository/docs/HELIA_INTEGRATION.md](docs/HELIA_INTEGRATION.md) | +| `docs/payments/stripe-connect-vs-onchain-payouts.md` | [repository/docs/payments/stripe-connect-vs-onchain-payouts.md](docs/payments/stripe-connect-vs-onchain-payouts.md) | +| `frontend/AGENTS.md` | [repository/frontend/AGENTS.md](frontend/AGENTS.md) | +| `frontend/CLAUDE.md` | [repository/frontend/CLAUDE.md](frontend/CLAUDE.md) | +| `frontend/DESIGN.md` | [repository/frontend/DESIGN.md](frontend/DESIGN.md) | +| `frontend/SKILL.md` | [repository/frontend/SKILL.md](frontend/SKILL.md) | +| `frontend/architecture.md` | [repository/frontend/architecture.md](frontend/architecture.md) | +| `docs/superpowers/3MIN_SCRIPT.md` | [superpowers/3MIN_SCRIPT.md](../superpowers/3MIN_SCRIPT.md) | +| `docs/superpowers/DEMO_CHEAT_SHEET.md` | [superpowers/DEMO_CHEAT_SHEET.md](../superpowers/DEMO_CHEAT_SHEET.md) | +| `docs/superpowers/DEMO_DAY_CHECKLIST.md` | [superpowers/DEMO_DAY_CHECKLIST.md](../superpowers/DEMO_DAY_CHECKLIST.md) | +| `docs/superpowers/DEMO_FLOW_DIAGRAM.md` | [superpowers/DEMO_FLOW_DIAGRAM.md](../superpowers/DEMO_FLOW_DIAGRAM.md) | +| `docs/superpowers/GROUND_TRUTH_DEMO_COMPLETE.md` | [superpowers/GROUND_TRUTH_DEMO_COMPLETE.md](../superpowers/GROUND_TRUTH_DEMO_COMPLETE.md) | +| `docs/superpowers/GROUND_TRUTH_PRESENTATION.md` | [superpowers/GROUND_TRUTH_PRESENTATION.md](../superpowers/GROUND_TRUTH_PRESENTATION.md) | +| `docs/superpowers/PRESENTATION_README.md` | [superpowers/PRESENTATION_README.md](../superpowers/PRESENTATION_README.md) | +| `docs/superpowers/plans/2026-05-09-quantum-pharma-pipeline.md` | [superpowers/plans/2026-05-09-quantum-pharma-pipeline.md](../superpowers/plans/2026-05-09-quantum-pharma-pipeline.md) | +| `docs/superpowers/plans/2026-05-14-autonomous-workspace.md` | [superpowers/plans/2026-05-14-autonomous-workspace.md](../superpowers/plans/2026-05-14-autonomous-workspace.md) | +| `docs/superpowers/plans/2026-10-31-architecture-scroll.md` | [superpowers/plans/2026-10-31-architecture-scroll.md](../superpowers/plans/2026-10-31-architecture-scroll.md) | +| `docs/superpowers/specs/2026-05-08-vault-nft-storage-integration.md` | [superpowers/specs/2026-05-08-vault-nft-storage-integration.md](../superpowers/specs/2026-05-08-vault-nft-storage-integration.md) | +| `docs/superpowers/specs/2026-05-09-quantum-pharma-implementation-design.md` | [superpowers/specs/2026-05-09-quantum-pharma-implementation-design.md](../superpowers/specs/2026-05-09-quantum-pharma-implementation-design.md) | +| `docs/superpowers/specs/2026-05-14-autonomous-workspace-design.md` | [superpowers/specs/2026-05-14-autonomous-workspace-design.md](../superpowers/specs/2026-05-14-autonomous-workspace-design.md) | +| `docs/superpowers/specs/2026-05-20-agentkit-frontend-design.md` | [superpowers/specs/2026-05-20-agentkit-frontend-design.md](../superpowers/specs/2026-05-20-agentkit-frontend-design.md) | +| `docs/superpowers/specs/quantum-pharma-docking-design.md` | [superpowers/specs/quantum-pharma-docking-design.md](../superpowers/specs/quantum-pharma-docking-design.md) | diff --git a/repository/backend/AWS_BEDROCK_SETUP.md b/repository/backend/AWS_BEDROCK_SETUP.md new file mode 100644 index 0000000..a8c3a94 --- /dev/null +++ b/repository/backend/AWS_BEDROCK_SETUP.md @@ -0,0 +1,621 @@ +# AWS Bedrock Setup Guide + +Complete guide to setting up AWS Bedrock for Claude 3.5 Sonnet access in the Quantum Backend project. + +--- + +## Table of Contents + +1. [Overview](#overview) +2. [Prerequisites](#prerequisites) +3. [Step 1: Create AWS IAM User](#step-1-create-aws-iam-user) +4. [Step 2: Configure Required Permissions](#step-2-configure-required-permissions) +5. [Step 3: Generate Access Keys](#step-3-generate-access-keys) +6. [Step 4: Enable Bedrock Model Access](#step-4-enable-bedrock-model-access) +7. [Step 5: Configure Environment Variables](#step-5-configure-environment-variables) +8. [Step 6: Test Connection](#step-6-test-connection) +9. [Troubleshooting](#troubleshooting) +10. [Cost Estimates](#cost-estimates) + +--- + +## Overview + +AWS Bedrock provides serverless access to Claude 3.5 Sonnet for: +- **Phase 3**: AI Research Agents (proposal analysis, coalition formation) +- **Phase 4**: Auto-fragmentation (breaking research proposals into tasks) + +**Model**: `anthropic.claude-3-5-sonnet-20241022-v2:0` + +**Why Bedrock?** +- Pay-per-use pricing (no monthly fees) +- Built-in AWS security and compliance +- No separate Anthropic API key needed +- Scales automatically with demand + +--- + +## Prerequisites + +- AWS Account (create at https://aws.amazon.com) +- Basic familiarity with AWS Console +- Admin access to create IAM users (or contact your AWS admin) + +--- + +## Step 1: Create AWS IAM User + +### 1.1 Navigate to IAM Console + +1. Log into AWS Console: https://console.aws.amazon.com +2. Search for "IAM" in the top search bar +3. Click on "IAM" (Identity and Access Management) + +### 1.2 Create New User + +1. In the left sidebar, click **Users** +2. Click **Create user** button +3. Enter user details: + - **User name**: `quantum-backend-bedrock` + - **Provide user access to AWS Management Console**: UNCHECK (we only need programmatic access) +4. Click **Next** + +--- + +## Step 2: Configure Required Permissions + +### 2.1 Attach Policy Directly + +On the "Set permissions" page: + +1. Select **Attach policies directly** +2. Search for `AmazonBedrockFullAccess` +3. Check the box next to **AmazonBedrockFullAccess** +4. Click **Next** + +### 2.2 Review and Create + +1. Review the user details: + - User name: `quantum-backend-bedrock` + - Permissions: `AmazonBedrockFullAccess` +2. Click **Create user** + +### 2.3 Custom Policy (Alternative - More Restrictive) + +If your organization requires least-privilege access, use this custom policy instead: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "BedrockInvokeModel", + "Effect": "Allow", + "Action": [ + "bedrock:InvokeModel", + "bedrock:InvokeModelWithResponseStream" + ], + "Resource": [ + "arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-5-sonnet-20241022-v2:0" + ] + } + ] +} +``` + +To use custom policy: +1. Select **Attach policies directly** +2. Click **Create policy** (opens new tab) +3. Click **JSON** tab +4. Paste the policy above +5. Click **Next**, name it `BedrockClaudeInvokeOnly` +6. Return to user creation tab and attach the new policy + +--- + +## Step 3: Generate Access Keys + +### 3.1 Navigate to User Security Credentials + +1. After creating the user, click on the user name `quantum-backend-bedrock` +2. Click the **Security credentials** tab +3. Scroll down to **Access keys** section +4. Click **Create access key** + +### 3.2 Select Use Case + +1. Select **Application running outside AWS** +2. Check the confirmation box at the bottom +3. Click **Next** + +### 3.3 Add Description Tag (Optional) + +1. Description: `Quantum Backend Claude API Access` +2. Click **Create access key** + +### 3.4 Download Credentials + +**CRITICAL**: This is your ONLY chance to see the secret key! + +1. You'll see: + - **Access key ID**: Starts with `AKIA...` + - **Secret access key**: Long random string +2. Click **Download .csv file** (recommended) +3. OR copy both values to a secure location immediately + +**Security Best Practices**: +- NEVER commit these keys to git +- Store in password manager or AWS Secrets Manager +- Rotate keys every 90 days + +--- + +## Step 4: Enable Bedrock Model Access + +AWS requires explicit permission to use Claude models. + +### 4.1 Navigate to Bedrock Console + +1. Search for "Bedrock" in AWS Console +2. Click **Amazon Bedrock** +3. In left sidebar, click **Model access** (under "Bedrock configurations") + +### 4.2 Request Model Access + +1. Find **Anthropic** in the provider list +2. Check the box next to **Claude 3.5 Sonnet v2** +3. Click **Request model access** (or **Modify model access** if already configured) +4. Review terms and conditions +5. Submit request + +**Access Approval**: +- Usually instant for most AWS accounts +- May take 1-2 business days for new accounts +- You'll receive email notification when approved + +### 4.3 Verify Access + +1. Return to **Model access** page +2. Status should show **Access granted** (green) next to Claude 3.5 Sonnet v2 +3. Model ID: `anthropic.claude-3-5-sonnet-20241022-v2:0` + +--- + +## Step 5: Configure Environment Variables + +### 5.1 Update `.env` File + +Navigate to your backend directory and edit `.env`: + +```bash +cd /path/to/nodes-quantum-gates/backend +nano .env # or use your preferred editor +``` + +### 5.2 Add AWS Credentials + +```env +# AI & Agent Configuration +# AWS Bedrock Configuration (for Claude API via AWS) +AWS_BEDROCK_ENABLED=true + +# AWS Region for Bedrock service +# Supported regions: us-east-1, us-west-2, ap-southeast-1, ap-northeast-1, eu-central-1 +AWS_REGION=us-east-1 + +# AWS credentials for Bedrock access +# Source: IAM -> Users -> quantum-backend-bedrock -> Security credentials +AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE +AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY + +# Claude model ID for Bedrock +BEDROCK_MODEL_ID=anthropic.claude-3-5-sonnet-20241022-v2:0 +``` + +**Replace**: +- `AKIAIOSFODNN7EXAMPLE` with your actual Access Key ID +- `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY` with your actual Secret Access Key + +### 5.3 Verify `.env` is Gitignored + +```bash +# Check .gitignore contains .env +cat .gitignore | grep "^\.env$" + +# If not found, add it +echo ".env" >> .gitignore +``` + +--- + +## Step 6: Test Connection + +### 6.1 Quick Test Script + +Create a test file to verify Bedrock access: + +```bash +cd /path/to/nodes-quantum-gates/backend +nano test_bedrock.py +``` + +Paste the following script: + +```python +#!/usr/bin/env python3 +""" +Test AWS Bedrock connection for Claude 3.5 Sonnet access. +Run: python test_bedrock.py +""" + +import json +import os +import sys +from datetime import datetime + +import boto3 +from botocore.exceptions import ClientError, NoCredentialsError + + +def test_bedrock_connection(): + """Test AWS Bedrock connection and Claude model invocation.""" + + print("=" * 60) + print("AWS BEDROCK CONNECTION TEST") + print("=" * 60) + print(f"Timestamp: {datetime.now().isoformat()}\n") + + # Step 1: Verify environment variables + print("Step 1: Checking environment variables...") + + aws_access_key = os.getenv("AWS_ACCESS_KEY_ID") + aws_secret_key = os.getenv("AWS_SECRET_ACCESS_KEY") + aws_region = os.getenv("AWS_REGION", "us-east-1") + model_id = os.getenv("BEDROCK_MODEL_ID", "anthropic.claude-3-5-sonnet-20241022-v2:0") + + if not aws_access_key: + print("ERROR: AWS_ACCESS_KEY_ID not found in environment") + return False + + if not aws_secret_key: + print("ERROR: AWS_SECRET_ACCESS_KEY not found in environment") + return False + + print(f" AWS_ACCESS_KEY_ID: {aws_access_key[:10]}... (FOUND)") + print(f" AWS_SECRET_ACCESS_KEY: ****** (HIDDEN)") + print(f" AWS_REGION: {aws_region}") + print(f" BEDROCK_MODEL_ID: {model_id}\n") + + # Step 2: Create Bedrock Runtime client + print("Step 2: Creating Bedrock Runtime client...") + + try: + client = boto3.client( + service_name="bedrock-runtime", + region_name=aws_region, + aws_access_key_id=aws_access_key, + aws_secret_access_key=aws_secret_key + ) + print(" SUCCESS: Client created\n") + except Exception as e: + print(f" ERROR: Failed to create client: {e}") + return False + + # Step 3: Prepare test request + print("Step 3: Preparing test request...") + + request_body = { + "anthropic_version": "bedrock-2023-05-31", + "max_tokens": 100, + "messages": [ + { + "role": "user", + "content": "Say 'Hello from Claude on AWS Bedrock!' and nothing else." + } + ] + } + + print(f" Request body: {json.dumps(request_body, indent=2)}\n") + + # Step 4: Invoke model + print("Step 4: Invoking Claude model...") + + try: + response = client.invoke_model( + modelId=model_id, + body=json.dumps(request_body) + ) + print(" SUCCESS: Model invoked\n") + except ClientError as e: + error_code = e.response["Error"]["Code"] + error_message = e.response["Error"]["Message"] + print(f" ERROR: AWS ClientError - {error_code}") + print(f" Message: {error_message}") + + if error_code == "AccessDeniedException": + print("\n Possible causes:") + print(" 1. Model access not granted in Bedrock console") + print(" 2. IAM user lacks bedrock:InvokeModel permission") + print(" 3. Incorrect model ID") + elif error_code == "ResourceNotFoundException": + print("\n Possible causes:") + print(" 1. Model not available in this region") + print(" 2. Incorrect model ID") + + return False + except NoCredentialsError: + print(" ERROR: AWS credentials not found or invalid") + return False + except Exception as e: + print(f" ERROR: Unexpected error: {e}") + return False + + # Step 5: Parse response + print("Step 5: Parsing response...") + + try: + response_body = json.loads(response["body"].read()) + + print(f" Response ID: {response_body.get('id', 'N/A')}") + print(f" Model: {response_body.get('model', 'N/A')}") + print(f" Stop reason: {response_body.get('stop_reason', 'N/A')}") + + if "content" in response_body and len(response_body["content"]) > 0: + content = response_body["content"][0] + if content["type"] == "text": + print(f"\n Claude's response:\n \"{content['text']}\"\n") + + # Check usage + usage = response_body.get("usage", {}) + input_tokens = usage.get("input_tokens", 0) + output_tokens = usage.get("output_tokens", 0) + + print(f" Usage:") + print(f" Input tokens: {input_tokens}") + print(f" Output tokens: {output_tokens}") + print(f" Total tokens: {input_tokens + output_tokens}\n") + + except Exception as e: + print(f" ERROR: Failed to parse response: {e}") + return False + + # Success! + print("=" * 60) + print("SUCCESS: AWS Bedrock connection test passed!") + print("=" * 60) + print("\nYour backend is ready to use Claude 3.5 Sonnet for:") + print(" - AI Research Agent proposal analysis") + print(" - Autonomous coalition formation") + print(" - Research proposal auto-fragmentation") + + return True + + +if __name__ == "__main__": + # Load .env file if present + try: + from dotenv import load_dotenv + load_dotenv() + print("INFO: Loaded variables from .env file\n") + except ImportError: + print("INFO: python-dotenv not installed, using system environment variables\n") + + success = test_bedrock_connection() + sys.exit(0 if success else 1) +``` + +### 6.2 Run Test Script + +```bash +# Make sure you're in the backend directory with .env file +cd /path/to/nodes-quantum-gates/backend + +# Install boto3 if not already installed +pip install boto3 python-dotenv + +# Run test +python test_bedrock.py +``` + +### 6.3 Expected Output + +``` +INFO: Loaded variables from .env file + +============================================================ +AWS BEDROCK CONNECTION TEST +============================================================ +Timestamp: 2026-05-20T15:30:45.123456 + +Step 1: Checking environment variables... + AWS_ACCESS_KEY_ID: AKIAIOSFO... (FOUND) + AWS_SECRET_ACCESS_KEY: ****** (HIDDEN) + AWS_REGION: us-east-1 + BEDROCK_MODEL_ID: anthropic.claude-3-5-sonnet-20241022-v2:0 + +Step 2: Creating Bedrock Runtime client... + SUCCESS: Client created + +Step 3: Preparing test request... + Request body: { + "anthropic_version": "bedrock-2023-05-31", + "max_tokens": 100, + "messages": [ + { + "role": "user", + "content": "Say 'Hello from Claude on AWS Bedrock!' and nothing else." + } + ] +} + +Step 4: Invoking Claude model... + SUCCESS: Model invoked + +Step 5: Parsing response... + Response ID: msg_abc123xyz... + Model: claude-3-5-sonnet-20241022 + Stop reason: end_turn + + Claude's response: + "Hello from Claude on AWS Bedrock!" + + Usage: + Input tokens: 15 + Output tokens: 8 + Total tokens: 23 + +============================================================ +SUCCESS: AWS Bedrock connection test passed! +============================================================ + +Your backend is ready to use Claude 3.5 Sonnet for: + - AI Research Agent proposal analysis + - Autonomous coalition formation + - Research proposal auto-fragmentation +``` + +--- + +## Troubleshooting + +### Error: AccessDeniedException + +**Symptom**: `An error occurred (AccessDeniedException) when calling the InvokeModel operation` + +**Solutions**: +1. Verify model access granted in Bedrock console: + - Go to Bedrock > Model access + - Check Claude 3.5 Sonnet v2 shows "Access granted" +2. Verify IAM permissions: + - IAM > Users > quantum-backend-bedrock > Permissions + - Should have `AmazonBedrockFullAccess` or custom policy with `bedrock:InvokeModel` +3. Wait 5-10 minutes after requesting model access (propagation delay) + +### Error: NoCredentialsError + +**Symptom**: `Unable to locate credentials` + +**Solutions**: +1. Verify `.env` file exists in backend directory +2. Verify `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are set +3. Check for typos in key names (case-sensitive) +4. Restart your terminal/IDE to reload environment + +### Error: ResourceNotFoundException + +**Symptom**: `Could not find model with ID: anthropic.claude-3-5-sonnet...` + +**Solutions**: +1. Verify region supports Claude: + - Supported: us-east-1, us-west-2, ap-southeast-1, ap-northeast-1, eu-central-1 + - Change `AWS_REGION` in `.env` if needed +2. Verify model ID spelling: + - Correct: `anthropic.claude-3-5-sonnet-20241022-v2:0` +3. Check for model version updates (AWS occasionally changes IDs) + +### Error: ThrottlingException + +**Symptom**: `Rate exceeded` or `TooManyRequestsException` + +**Solutions**: +1. Add exponential backoff to your code +2. Request quota increase in AWS Service Quotas console +3. Default limits: + - 10,000 tokens/minute (input + output combined) + - 100 requests/minute + +### Connection Times Out + +**Symptom**: Request hangs for 60+ seconds + +**Solutions**: +1. Check network connectivity to AWS +2. Verify no firewall blocking port 443 +3. Try different AWS region +4. Check AWS service health: https://status.aws.amazon.com + +--- + +## Cost Estimates + +### Pricing (as of May 2026) + +**Claude 3.5 Sonnet v2 on Bedrock**: +- Input tokens: $3.00 per million tokens +- Output tokens: $15.00 per million tokens + +### Example Costs + +**AI Research Agent Proposal Analysis**: +- Input: 5,000 tokens (proposal text) +- Output: 500 tokens (analysis JSON) +- Cost per analysis: $0.0225 + +**Auto-Fragmentation**: +- Input: 3,000 tokens (proposal) +- Output: 1,000 tokens (fragment list) +- Cost per fragmentation: $0.024 + +**Monthly Estimates**: +- 100 proposals analyzed: $2.25 +- 50 proposals fragmented: $1.20 +- Total: ~$3.45/month (low usage) +- High usage (1,000 proposals/month): ~$34.50/month + +**Comparison to Direct Anthropic API**: +- Bedrock pricing is identical to Anthropic's API pricing +- Bedrock adds AWS infrastructure benefits (IAM, CloudWatch, VPC, etc.) +- No additional AWS charges for Bedrock itself (only model usage) + +### Cost Optimization Tips + +1. Use prompt caching for repeated context +2. Set reasonable `max_tokens` limits +3. Implement response caching for duplicate requests +4. Monitor usage in AWS Cost Explorer +5. Set up billing alerts in AWS Budgets + +--- + +## Next Steps + +After successful setup: + +1. **Phase 3 (AI Agents)**: + - Create AI agents via `/agents` API + - Test proposal analysis: `POST /agents/:id/analyze` + - Test autonomous funding workflows + +2. **Phase 4 (Research Crowdfunding)**: + - Create research proposals: `POST /proposals` + - Test auto-fragmentation (uses Claude) + - Verify fragments stored in MongoDB + +3. **Production Considerations**: + - Implement retry logic with exponential backoff + - Add structured logging for Bedrock calls + - Set up CloudWatch alarms for errors/throttling + - Consider AWS Secrets Manager for credentials + - Enable AWS CloudTrail for audit logging + +4. **Monitoring**: + - Track token usage in CloudWatch Metrics + - Set billing alerts (e.g., alert if >$50/month) + - Monitor response latencies + - Log all API errors + +--- + +## Additional Resources + +- **AWS Bedrock Documentation**: https://docs.aws.amazon.com/bedrock/ +- **Claude on Bedrock Guide**: https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-anthropic-claude.html +- **boto3 Bedrock Reference**: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/bedrock-runtime.html +- **Anthropic API Reference**: https://docs.anthropic.com/ +- **Cost Calculator**: https://calculator.aws/ + +--- + +**Document Version**: 1.0 +**Last Updated**: 2026-05-20 +**Maintained By**: Quantum Backend Team diff --git a/repository/backend/CHAIN_REGISTRY.md b/repository/backend/CHAIN_REGISTRY.md new file mode 100644 index 0000000..49db97b --- /dev/null +++ b/repository/backend/CHAIN_REGISTRY.md @@ -0,0 +1,305 @@ +# Chain registry — Part of #2 + +The registry lists network metadata and supported operations. Base Sepolia's +existing wallet service now reads token, escrow, and explorer metadata from it. +Ethereum Sepolia remains available for metadata and explorer links only. + +## Configuration and identifiers + +`src/quantum_backend_v2/chains/registry.toml` ships inside the Python package. +Bootstrap loads and validates it once, before creating persistence and peer runtimes, +then passes the immutable registry to the API. Invalid configuration prevents +startup. Parsing uses `tomllib` or the existing `tomli` dependency on Python 3.10. + +Base Sepolia remains the default. In a CAIP-2 ID, `eip155` identifies the +Ethereum-compatible family and the number identifies the network. + +| Network | Canonical ID | AgentKit alias | CDP alias | +| --- | --- | --- | --- | +| Base Sepolia | `eip155:84532` | `base-sepolia` | `base-sepolia` | +| Ethereum Sepolia | `eip155:11155111` | `ethereum-sepolia` | `ethereum-sepolia` | +| Starknet Sepolia | `starknet:SN_SEPOLIA` | — | — | + +Starknet Sepolia is a read-only registry entry (`explorer_url`, `verify_transaction`) plus an opt-in session policy. Base USDC funding stays on the Coinbase path. Alias: `starknet-sepolia`. Native token: STRK, 18 decimals. Explorer templates point at Voyager Sepolia. `QB2_STARKNET_SEPOLIA_RPC_URL` overrides the public RPC. + +`registry.resolve()` returns the configured default; `registry.resolve(name)` +accepts a canonical ID, legacy alias, or provider alias. An empty, unknown, or +disabled name raises an error rather than silently selecting Base Sepolia. +Matching is exact and case-sensitive. Duplicate IDs and ambiguous aliases across +entries, including provider mappings, are rejected. There is no global active network. + +`QB2_CHAIN_REGISTRY_PATH` optionally selects another TOML file through application +settings; unset uses package data. Relative paths use the working directory. +`NETWORK` selects the wallet network by alias or canonical ID. When unset, the +registry default applies; the packaged default remains Base Sepolia. Empty or +unknown values fail startup. Wallet endpoints use an application-scoped service +with the same immutable registry as the public chain endpoint. Marketplace, +proposal, and AI agent API services share this wallet service, including proposal +broadcasts, agent wallet creation, autonomous funding, and escrow release. These +services are initialized lazily per application; their MongoDB clients and the proposal +service’s Bedrock client close on application shutdown. Standalone proposal and +agent callers can inject a wallet service; otherwise each lazily creates and +reuses its own service. Standalone `AgentKitService` callers load these settings +too, or accept an injected registry and explicit network. Injected registries use +their own default, without reading process-wide network selection. + +Mainnet entries must have `enabled = false`. Disabled entries cannot be resolved +for operations and are omitted from the public list. Existing wallet records must +resolve to the selected network before balance queries, provider reconstruction, +transfers, escrow operations, or faucet instructions. Mismatches fail instead of +using the same address on a different chain. Stored network aliases and unique +wallet indexes are unchanged; this is not yet multi-network wallet persistence. + +## Credentials and operation capabilities + +`rpc_env_var` stores an environment-variable **name**, never a URL or key. +`public_rpc_url` is an optional keyless fallback: HTTPS only, with no credentials, +query string or fragment. `chains.rpc.resolve_rpc_url()` uses the variable when it +is set and the public endpoint otherwise. The packaged fallbacks are +`https://sepolia.base.org`, the endpoint AgentKit used before the registry, and +`https://ethereum-sepolia-rpc.publicnode.com`, the endpoint the ENS service uses. + +Overrides must be HTTPS, or plain HTTP to `localhost`/`127.0.0.1`/`::1` for a local +node such as anvil. A resolved URL may carry an API key. It is never stored in the +registry, returned by `GET /api/v1/chains` or logged: RPC failures are logged by +exception type only, and configuration errors name the variable, not its value. + +Base Sepolia declares `explorer_url`, `create_wallet`, `get_balance`, +`transfer_stablecoin`, `aave_supply`, `aave_withdraw`, and `faucet_instructions`. +The service checks each operation before provider or RPC calls. Transaction +capabilities require explorer metadata; transfer and supply also require balance +queries. Aave operations require an explicit pool address. Wallet capabilities +are currently restricted to the existing Base Sepolia provider mapping; adding +another network requires provider validation and implementation coverage. + +Ethereum Sepolia declares `explorer_url` and `verify_transaction`. Listing its +provider aliases does not enable wallet operations. `verify_transaction` is +read-only and needs no wallet provider, so it is the one RPC capability allowed +outside the Base Sepolia wallet mapping. The explorer adapter and the legacy +`utils.basescan` wrappers perform no network I/O. They validate reference shape, +not checksums, existence, ownership, or transaction success. + +`anchor_root` (#21) marks a network that may carry provenance Merkle-root anchors. It +is allowed only on testnets with `verify_transaction`, and it does not use the wallet +provider: anchors come from a dedicated low-balance account, never a user or platform +wallet. Ethereum Sepolia declares it. See +[Anchor transactions](PROVENANCE.md#anchor-transactions-21). + +Robinhood Chain Testnet (`eip155:46630`, #18) declares `explorer_url` and +`verify_transaction` only, with no stablecoin. Official Stock Tokens and Chainlink +feeds exist only on Robinhood Chain mainnet, which stays out of this registry; see +[Robinhood Chain market data](#robinhood-chain-market-data). + +The faucet endpoint returns manual instructions; it does not send funds. Balance +queries use Web3 through the registry RPC, not the CDP token-balance endpoint. + +## Adapters + +| Adapter | Operations | Network I/O | +| --- | --- | --- | +| `EvmExplorerAdapter` | `explorer_url` | None | +| `EvmRpcAdapter` | `get_balance`, `verify_transaction` | Read-only JSON-RPC through an injected `Web3` | + +`EvmRpcAdapter` checks the chain's capabilities and validates every address and +hash before any RPC call. Before its first query it compares `eth_chainId` with the +registry entry and raises `ChainMismatchError` on a mismatch, so a mistyped +override cannot read another network. + +- `get_balance(address)` returns the native and stablecoin balances, scaled by the + registry's decimals. +- `verify_transaction(tx_hash, expected_transfer=None, min_confirmations=1)` + returns `not_found`, `pending` (in the mempool, or mined with too few + confirmations), `failed` (reverted) or `confirmed`, with the block and + confirmation count. With an `expected_transfer`, it also reports whether the + receipt has a Transfer log from the registry's stablecoin with that exact sender, + recipient and amount. `verified` is true only for a confirmed transaction whose + expected transfer, if any, matched. + +Transfers and Aave calls stay in `AgentKitService`: they are signed by the CDP +wallet provider, which uses its own RPC client. No adapter holds keys. + +Explorer templates deliberately support only HTTPS URLs of the form +`https://HOST/tx/{tx_hash}`, `/address/{address}`, and `/block/{block_number}`. +Credentials, query strings, fragments, and extra formatting fields are forbidden. +Custom explorer URL shapes require a deliberate schema/adapter change. + +Loader errors report validation reasons without raw values, field locations, +file paths, or parser details. Keep validator messages free of user input; unknown +field names can also contain secrets. Do not log raw TOML or direct Pydantic errors. + +## Robinhood Chain market data + +`chains/market_data.toml` is a reviewed allowlist of tokenized stocks on Robinhood +Chain mainnet (`eip155:4663`) whose Chainlink price history may be read for analysis. +The maintainers approved this read-only mainnet exception in #18. It is separate from +the registry and has no capabilities: `TokenPriceHistoryReader` sends only `eth_call` +through Multicall3, never wallets, signing or transactions. The RPC is resolved like +registry entries, through `QB2_ROBINHOOD_MAINNET_RPC_URL` or the public endpoint. + +Each entry joins Robinhood's asset list with Chainlink's Robinhood feed directory and +was checked on-chain at `reviewed_block`: the token's `symbol()`, 18 decimals and +`uiMultiplier()`, and the feed's `description()` and 8 decimals. Feeds report the +**token** price, which already includes the multiplier; it is never applied again. +Before reading, the reader compares each feed's description, decimals and phase with +the allowlist and fails if they changed or the feed has been silent for over 7 days +(weekends plus a market holiday reach about 104 hours). A Chainlink phase upgrade +therefore blocks reads until the allowlist is reviewed, even for older windows. + +History rules (`chains/price_history.py`): + +- Rounds before `first_valid_round` are excluded. Each feed's launch rounds + (2026-06-22 to 2026-06-23 13:50 UTC) were published at about 1e8x the price. +- Every round in the window must have a positive answer, ordered timestamps, + `answeredInRound >= roundId`, and move under 50% from the previous round; any + violation rejects the request. Missing data is never zero-filled or substituted. +- Each weekday's price is the last valid round at or before 21:00 UTC (after the + 16:00 ET close in summer and winter time), if it is at most 36 hours old. Feeds + publish on a 0.5% move or a 24-hour heartbeat, so a quiet day keeps the last + published price; that is the feed's value at the cutoff, not a substitute. +- A weekday on which no selected feed published (a market holiday such as + 2026-07-03 or Labor Day) is dropped rather than adding a zero return. Dates where any + selected asset lacks a price are dropped too; fewer than 60 remaining dates is an + error. The range is limited to 366 days, must contain at least 60 weekdays, and must + not end in the future. A cutoff after the read block is not yet a close. +- `PriceHistory` records the chain, block number/hash, token and feed addresses, + decimals, and the round ID and `updatedAt` of every close. `to_wide_csv()` produces + the optimizer's existing wide price format. + +Chainlink rounds are immutable, so the recorded round IDs reproduce a series later. +The public RPC is not an archive node and does not serve old block state. + +To add an asset, confirm it in `GET https://api.robinhood.com/rhj/assets` and +[Chainlink's Robinhood feeds](https://docs.chain.link/data-feeds/price-feeds/addresses?network=robinhood), +verify the on-chain fields above, read its rounds to find the first valid one, and +update `reviewed_block`. + +## Public API + +`GET /api/v1/chains` is public and requires no wallet credentials, RPC, or database query. + +The response has `default_chain_id` and an ordered `chains` list. Each enabled +entry contains exactly: + +- `chain_id`, `family`, `display_name`, `testnet` +- `aliases`: legacy network names, such as the `network` stored on wallets +- `native_token`: `symbol`, `decimals` +- `stablecoin`: `symbol`, `decimals`, `address`, or `null` +- `explorers`: `transaction`, `address`, `block` templates +- `capabilities`: the operations declared above for each network + +RPC configuration, provider aliases, and escrow configuration are excluded from +responses and OpenAPI. + +## Starknet session policies + +Ready (Argent) is the owner wallet. In this version the owner signs `register_session`, `spend`, and `revoke_session` on `backend/contracts/starknet`. The backend stores only the session private key, encrypted with `WALLET_ENCRYPTION_KEY`. That key is not registered on the Ready account and `spend` does not check it. `spend` reverts on-chain when the amount exceeds `max_per_proposal` (`per_call`), the UTC day total exceeds `daily_budget` (`daily`), the session is past `expires_at` (`expired`), or it has been revoked (`revoked`). + +AVNU's Sepolia paymaster (`https://sepolia.paymaster.avnu.fi`, header `x-paymaster-api-key` from `AVNU_PAYMASTER_API_KEY`) may sponsor `deploy_account` and `register_session` only. Spends and revokes are not sponsored. + +Contributor toolchain, not imported by the running API until a session is created: [Scarb](https://docs.swmansion.com/scarb/) to build the contract (`scarb build` in `backend/contracts/starknet`, Cairo 2.8) and [`starknet.py`](https://pypi.org/project/starknet.py/) (`uv add starknet.py` from `backend`) to generate the session key. Scarb is a separate install of a few hundred megabytes. `starknet.py` is an extra Python dependency used only for the Stark curve keypair. Policy decisions and the paymaster gate are covered by `tests/unit/test_starknet_policy.py` without either tool and without a funded account. + +## Adding another EVM testnet + +1. Verify its ID, token metadata, and explorer against official sources. Check + locked provider versions for each operation before adding provider mappings. +2. Copy a `[[chains]]` entry and replace network-specific fields. Use a unique + `eip155:NUMBER` without leading zeros and unique aliases. Unknown provider + mappings and stablecoin metadata can be omitted. +3. Set `testnet = true`, choose `enabled`, and declare only implemented capabilities. +4. Set `rpc_env_var`, add a keyless `public_rpc_url` if one exists, and document + the variable in `.env.example`. +5. Add metadata expectations and run the focused tests below. + +## Verified sources and provider limits + +Checked October 21, 2026: + +- [CAIP-2](https://standards.chainagnostic.org/CAIPs/caip-2) defines the namespaced + identifier. This slice intentionally accepts only canonical decimal EVM IDs. +- [Base network details](https://docs.base.org/get-started/connect-to-base) + specify Base Sepolia ID `84532`, ETH, and its explorer. +- [Ethereum client Sepolia metadata](https://github.com/eth-clients/sepolia) + specifies ID `11155111` and its explorers. +- [Robinhood Chain's connection page](https://docs.robinhood.com/chain/connecting/) + specifies testnet ID `46630`, ETH, its public RPC and explorer (checked October 28, + 2026; `eth_chainId` returned `0xb626`). Robinhood's asset API lists every Stock + Token on mainnet `4663` only. +- [Ethereum's ether reference](https://ethereum.org/developers/docs/intro-to-ether/) + defines 18 decimal places. Both entries use ETH as the native token. +- [Circle's contract list](https://developers.circle.com/stablecoins/usdc-contract-addresses) + supplies both test USDC addresses. [Circle's USDC reference](https://github.com/circlefin/skills/blob/master/plugins/circle/skills/use-usdc/SKILL.md) + documents the six-decimal ERC-20 representation. These are test tokens. +- The [AgentKit 0.7.4 distribution](https://pypi.org/project/coinbase-agentkit/0.7.4/) + was inspected using the wheel URL and SHA-256 from `uv.lock`: + `coinbase_agentkit/network/network.py` maps both IDs; the wallet provider's + `_get_cdp_sdk_network` maps both aliases. +- The [CDP SDK 1.45.1 distribution](https://pypi.org/project/cdp-sdk/1.45.1/) + was inspected the same way. Its `ListEvmTokenBalancesNetwork` enum includes + `base`, `base-sepolia`, and `ethereum`, **not `ethereum-sepolia`**. + [Coinbase's balance documentation](https://docs.cdp.coinbase.com/server-wallets/v2/using-the-wallet-api/managing-accounts) + lists the same subset. Thus Ethereum Sepolia is suitable for this metadata/link + layer, but general wallet support cannot be inferred from its alias. This + restriction concerns CDP token-balance queries; the existing service reads + balances directly through Web3. + +## Validation + +From `backend`, with project dependencies installed: + +```bash +uv run --no-sync pytest tests/unit/test_chain_registry.py tests/unit/test_chain_adapter.py tests/unit/test_chains_api.py -q +uv run --no-sync pytest tests/unit/test_chain_rpc.py tests/unit/test_chain_rpc_adapter.py -q +uv run --no-sync pytest tests/unit/test_chains_bootstrap.py tests/unit/test_wallet_registry_service.py tests/unit/test_basescan_registry.py -q +uv run --no-sync ruff check src/quantum_backend_v2/chains src/quantum_backend_v2/api/routers/chains.py tests/unit/test_chain*.py +uv run --no-sync mypy --follow-imports=silent src/quantum_backend_v2/chains src/quantum_backend_v2/api/routers/chains.py +uv run --no-sync pytest tests/unit/test_market_data.py tests/unit/test_token_price_history.py -q +# Optional read-only mainnet check (eth_call only): +QB2_ROBINHOOD_LIVE_TEST=1 uv run --no-sync pytest tests/live/test_robinhood_price_history.py -m live_robinhood -q +``` + +`test_token_price_history.py` replays real rounds recorded from Robinhood Chain +mainnet (`tests/fixtures/robinhood_chainlink_rounds.json`, SPY and QQQ at the block +it names) through an in-process JSON-RPC provider; the live check compares the same +rounds with the chain. + +The first command runs isolated registry/API tests. Bootstrap tests exercise the +real app factory and lifespan with persistence, recovery, peers, and legacy external +service clients mocked; +they require the backend's other import-time dependencies. None of these use a +database, blockchain, or wallet: the RPC adapter tests answer JSON-RPC from a +scripted in-process provider. They do not verify live peer or wallet integration. + +The dependency fix pins the published libp2p 0.6.0 release and removes the moving +archive URL; see [Local development](https://github.com/Winter-Soren/distributed-quantum-services/blob/main/backend/README.md#local-development). The bootstrap +unit tests still mock peer services. The existing `test_libp2p_peerstore.py` tests +exercise the real library with temporary SQLite files. +`tests/network/test_libp2p_transport.py` verifies real bidirectional RPC, discovery +delivery, and shutdown between two local peers without external services. + +## Existing escrow configuration + +The Base Sepolia Aave pool `0x07eA79F68B2B3df564D0A34F8e19D9B1e339814b` was +moved from the service to the registry without changing it. Its capability flags +preserve existing transaction-building paths; they do not verify deployment or +reserve compatibility. The [current Aave address book](https://github.com/bgd-labs/aave-address-book/blob/main/src/AaveV3BaseSepolia.sol) +lists a different pool and a different test USDC reserve from this application's +Circle test USDC. Do not substitute one of those addresses alone: pool/token +compatibility and existing positions need a separate escrow review. No live +Aave transaction or contract verification was performed for this migration. + +## Frontend + +`src/features/chains` in the frontend reads `/api/v1/chains` through `/api/chains`. +Wallet, funding and storage-export screens resolve the wallet's stored `network` +through `aliases` and build explorer links from the registry templates; none of +them hard-code an explorer or network. The wallet transfer tab and the proposal +funding dialog show a chain selector. Only the wallet's own network is selectable, +and only when it has the operation's capability (`transfer_stablecoin`, +`aave_supply`), because wallet and funding requests do not take a network yet. + +## Out of scope for #2 + +Enabling a second wallet network, letting requests choose a network, and +network-aware wallet/payment persistence wait until a second chain supports +wallets. Confirming payouts with `verify_transaction` belongs to the payout work +in #8. Existing payment recording and confirmation semantics remain unchanged. diff --git a/repository/backend/IPFS_SETUP.md b/repository/backend/IPFS_SETUP.md new file mode 100644 index 0000000..6781006 --- /dev/null +++ b/repository/backend/IPFS_SETUP.md @@ -0,0 +1,479 @@ +# IPFS Setup Guide for Research Results Storage + +This guide walks you through setting up IPFS via Web3.Storage for decentralized storage of research results in the Quantum Backend system. + +## Overview + +**What is IPFS?** +- InterPlanetary File System: Content-addressed, peer-to-peer hypermedia protocol +- Files are identified by their content (CID), not location +- Perfect for immutable research results that need permanent, verifiable storage + +**Why Web3.Storage?** +- Free tier: 5GB storage (plenty for research results) +- Simple HTTP API (no local IPFS node required) +- Built-in pinning service (keeps your content available) +- Public gateway URLs for easy access + +**Use Case in Phase 4:** +- Research results uploaded to IPFS after completion +- Content-addressable by CID (Content Identifier) +- Publicly verifiable and permanently stored +- Gateway URLs provided for easy viewing + +--- + +## Step 1: Sign Up for Web3.Storage + +1. **Visit the signup page** + ``` + https://web3.storage/login + ``` + +2. **Create your account** + - Sign in with **Email** or **GitHub** + - Verify your email if prompted + - No payment information required for free tier + +3. **Confirm account activation** + - Check your email for verification link + - Click to activate account + +--- + +## Step 2: Generate API Token + +1. **Navigate to API Tokens** + - After logging in, go to **Account** section + - Click **Create API Token** or **New Token** + +2. **Create the token** + - Give it a descriptive name: `Quantum Backend - Research Results` + - Click **Create** + +3. **Copy the token** + - Token format: `eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...` + - **IMPORTANT:** Copy immediately - it won't be shown again + - Store securely in your password manager + +--- + +## Step 3: Configure Environment Variables + +1. **Open your `.env` file** + ```bash + cd /path/to/backend + nano .env # or use your preferred editor + ``` + +2. **Add the Web3.Storage API key** + ```bash + # IPFS / Decentralized Storage (Web3.Storage) + WEB3_STORAGE_API_KEY=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.your_actual_token_here + ``` + +3. **Save and close** the file + +4. **Verify the configuration** + ```bash + # Check that the key is set (don't print full value for security) + grep WEB3_STORAGE_API_KEY .env + ``` + +--- + +## Step 4: Test IPFS Upload + +We've provided a test script to verify your IPFS integration is working correctly. + +### Test Script: `test_ipfs_upload.py` + +Save this as `/backend/scripts/test_ipfs_upload.py`: + +```python +#!/usr/bin/env python3 +""" +Test script for IPFS integration via Web3.Storage. +Verifies that the IPFSService can upload and retrieve content. +""" + +import asyncio +import sys +from pathlib import Path + +# Add src to path for imports +sys.path.insert(0, str(Path(__file__).parent.parent / "src")) + +from utils.ipfs import IPFSService, IPFSError + + +async def test_json_upload(): + """Test uploading JSON data to IPFS.""" + print("\n=== Testing JSON Upload to IPFS ===\n") + + try: + # Initialize service (reads WEB3_STORAGE_API_KEY from environment) + service = IPFSService() + print("✓ IPFSService initialized successfully") + + # Sample research results + test_data = { + "research_id": "test-quantum-teleportation-001", + "algorithm": "Quantum Teleportation", + "timestamp": "2026-05-20T10:00:00Z", + "results": { + "fidelity": 0.9845, + "gate_count": 12, + "circuit_depth": 4, + "success_rate": 0.98 + }, + "metadata": { + "researcher": "test_user", + "proposal_id": "prop_12345", + "fragment_id": "frag_67890" + } + } + + print("Uploading test research results to IPFS...") + metadata = await service.upload_research_results(test_data) + + print(f"\n✓ Upload successful!") + print(f" CID: {metadata['cid']}") + print(f" Gateway URL: {metadata['url']}") + print(f"\n📝 You can view the content at: {metadata['url']}") + + return True + + except ValueError as e: + print(f"\n✗ Configuration error: {e}") + print("\nMake sure WEB3_STORAGE_API_KEY is set in your .env file") + return False + + except IPFSError as e: + print(f"\n✗ IPFS upload failed: {e}") + return False + + except Exception as e: + print(f"\n✗ Unexpected error: {e}") + import traceback + traceback.print_exc() + return False + + +async def test_file_upload(): + """Test uploading a file to IPFS.""" + print("\n=== Testing File Upload to IPFS ===\n") + + try: + service = IPFSService() + + # Create a temporary test file + test_file = Path("/tmp/quantum_test_result.txt") + test_content = """ +Quantum Circuit Execution Results +================================== +Date: 2026-05-20 +Algorithm: Shor's Algorithm +Input: N = 15 + +Results: +- Period found: r = 4 +- Factors: 3, 5 +- Success: True +- Fidelity: 0.9876 +""" + test_file.write_text(test_content) + print(f"Created test file: {test_file}") + + print("Uploading test file to IPFS...") + cid = await service.upload_file(str(test_file)) + url = service.get_gateway_url(cid) + + print(f"\n✓ File upload successful!") + print(f" CID: {cid}") + print(f" Gateway URL: {url}") + print(f"\n📝 You can view the file at: {url}") + + # Clean up + test_file.unlink() + + return True + + except Exception as e: + print(f"\n✗ File upload test failed: {e}") + return False + + +async def test_gateway_url_generation(): + """Test gateway URL generation.""" + print("\n=== Testing Gateway URL Generation ===\n") + + try: + service = IPFSService() + + # Test with sample CID + test_cid = "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi" + url = service.get_gateway_url(test_cid) + + expected = f"https://{test_cid}.ipfs.w3s.link" + + if url == expected: + print(f"✓ Gateway URL generated correctly:") + print(f" {url}") + return True + else: + print(f"✗ Gateway URL mismatch:") + print(f" Expected: {expected}") + print(f" Got: {url}") + return False + + except Exception as e: + print(f"\n✗ Gateway URL test failed: {e}") + return False + + +async def main(): + """Run all IPFS tests.""" + print("=" * 60) + print("IPFS Integration Test Suite") + print("=" * 60) + + results = [] + + # Test 1: Gateway URL generation (no API call) + results.append(await test_gateway_url_generation()) + + # Test 2: JSON upload (requires API key) + results.append(await test_json_upload()) + + # Test 3: File upload (requires API key) + results.append(await test_file_upload()) + + # Summary + print("\n" + "=" * 60) + print("Test Summary") + print("=" * 60) + passed = sum(results) + total = len(results) + print(f"\n{passed}/{total} tests passed") + + if passed == total: + print("\n✓ All tests passed! IPFS integration is working correctly.") + return 0 + else: + print("\n✗ Some tests failed. Check the error messages above.") + return 1 + + +if __name__ == "__main__": + exit_code = asyncio.run(main()) + sys.exit(exit_code) +``` + +### Running the Test Script + +1. **Make the script executable** + ```bash + chmod +x scripts/test_ipfs_upload.py + ``` + +2. **Run the test** + ```bash + # From the backend directory + python scripts/test_ipfs_upload.py + ``` + +3. **Expected output** + ``` + ============================================================ + IPFS Integration Test Suite + ============================================================ + + === Testing Gateway URL Generation === + ✓ Gateway URL generated correctly: + https://bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi.ipfs.w3s.link + + === Testing JSON Upload to IPFS === + ✓ IPFSService initialized successfully + Uploading test research results to IPFS... + ✓ Upload successful! + CID: bafybeiabc123... + Gateway URL: https://bafybeiabc123....ipfs.w3s.link + + === Testing File Upload to IPFS === + Created test file: /tmp/quantum_test_result.txt + Uploading test file to IPFS... + ✓ File upload successful! + CID: bafybeiabc456... + Gateway URL: https://bafybeiabc456....ipfs.w3s.link + + ============================================================ + Test Summary + ============================================================ + 3/3 tests passed + ✓ All tests passed! IPFS integration is working correctly. + ``` + +--- + +## Step 5: Integration with Proposal Service + +The `IPFSService` is already integrated into the proposal service. Here's how it works: + +### Automatic Upload on Result Submission + +```python +from utils.ipfs import get_ipfs_service + +# In proposal_service.py +async def submit_results(proposal_id: str, fragment_id: str, results: dict): + """Submit research results and upload to IPFS.""" + + # Upload results to IPFS + ipfs_service = get_ipfs_service() + ipfs_metadata = await ipfs_service.upload_research_results(results) + + # Store CID and gateway URL in MongoDB + await proposals_collection.update_one( + {"_id": proposal_id, "fragments.id": fragment_id}, + { + "$set": { + "fragments.$.results": results, + "fragments.$.ipfs_cid": ipfs_metadata["cid"], + "fragments.$.ipfs_url": ipfs_metadata["url"], + "fragments.$.status": "completed" + } + } + ) +``` + +### API Endpoint Usage + +**POST /api/proposals/{proposal_id}/fragments/{fragment_id}/results** + +```json +{ + "results": { + "algorithm": "Quantum Teleportation", + "fidelity": 0.9845, + "measurements": [...], + "circuit": "OPENQASM 2.0; ..." + } +} +``` + +**Response:** + +```json +{ + "message": "Results submitted and uploaded to IPFS", + "ipfs_cid": "bafybeiabc123...", + "ipfs_url": "https://bafybeiabc123....ipfs.w3s.link", + "fragment_id": "frag_67890", + "status": "completed" +} +``` + +--- + +## Troubleshooting + +### Error: "Web3.Storage API key is required" + +**Cause:** Environment variable not set or `.env` file not loaded + +**Solution:** +1. Verify `WEB3_STORAGE_API_KEY` exists in `.env` +2. Restart the backend server to reload environment +3. Check for typos in variable name + +### Error: "HTTP 401 Unauthorized" + +**Cause:** Invalid or expired API token + +**Solution:** +1. Generate a new token at https://web3.storage +2. Update `.env` with the new token +3. Restart the server + +### Error: "Upload timeout" or "Network error" + +**Cause:** Network connectivity issues or Web3.Storage downtime + +**Solution:** +1. Check your internet connection +2. Verify Web3.Storage status: https://status.web3.storage +3. Try again in a few minutes +4. Check firewall settings (port 443 must be open) + +### Error: "5GB storage limit exceeded" + +**Cause:** Free tier storage limit reached + +**Solution:** +1. Delete old uploads via Web3.Storage dashboard +2. Upgrade to paid plan for more storage +3. Implement cleanup strategy for old research results + +--- + +## Storage Best Practices + +### 1. Content Optimization +- Compress large datasets before upload +- Use efficient JSON serialization +- Remove unnecessary metadata + +### 2. CID Management +- Always store CIDs in your database +- CIDs are permanent - content cannot be edited +- Use CIDs for verification and deduplication + +### 3. Gateway URLs +- Gateway URLs are for viewing, not storage +- Multiple gateways can serve the same CID +- Default gateway: `https://{cid}.ipfs.w3s.link` + +### 4. Rate Limiting +- Web3.Storage has rate limits on free tier +- Batch uploads when possible +- Implement retry logic with exponential backoff + +--- + +## Next Steps + +1. ✓ Sign up for Web3.Storage account +2. ✓ Generate and configure API token +3. ✓ Run test script to verify integration +4. ✓ IPFS automatically used by proposal service +5. Start submitting real research results! + +--- + +## Resources + +- **Web3.Storage Documentation**: https://web3.storage/docs/ +- **IPFS Documentation**: https://docs.ipfs.tech/ +- **CID Inspector**: https://cid.ipfs.tech/ +- **Public Gateway Checker**: https://ipfs.github.io/public-gateway-checker/ + +--- + +## Support + +If you encounter issues: + +1. Check Web3.Storage status: https://status.web3.storage +2. Review Web3.Storage docs: https://web3.storage/docs/ +3. Test with curl: + ```bash + curl -X POST https://api.web3.storage/upload \ + -H "Authorization: Bearer YOUR_API_KEY" \ + -H "Content-Type: application/json" \ + -d '{"test": "data"}' + ``` + +--- + +**Setup completed!** Your IPFS integration is ready for decentralized research result storage. diff --git a/repository/backend/NOTIFICATION_SERVICE_IMPLEMENTATION.md b/repository/backend/NOTIFICATION_SERVICE_IMPLEMENTATION.md new file mode 100644 index 0000000..2cf95d3 --- /dev/null +++ b/repository/backend/NOTIFICATION_SERVICE_IMPLEMENTATION.md @@ -0,0 +1,276 @@ +# NotificationService Implementation Summary + +## Overview + +Successfully implemented a comprehensive multi-channel notification system for the quantum research crowdfunding platform. + +## Location + +- **Main Service**: `/backend/src/services/notification_service.py` (819 lines) +- **Package Export**: `/backend/src/services/__init__.py` (updated) +- **Documentation**: [Notification service guide](src/services/README_NOTIFICATION_SERVICE.md) +- **Examples**: `/backend/src/services/notification_service_example.py` + +## Implementation Status + +### ✅ Core Service Class + +- `NotificationService` class with async methods +- MongoDB integration via Motor (AsyncIOMotorClient) +- Resend API integration via httpx (no SDK needed) +- WebSocket integration via existing ConnectionManager +- Optional GossipSub PubSub support + +### ✅ Notification Methods + +1. **notify_new_proposal(proposal: dict)** + - Sends to all users (with preference support ready) + - Email via Resend batch API + - WebSocket broadcast to all connected users + - GossipSub broadcast on "proposals" topic + - Creates NotificationDocument for each user + +2. **notify_proposal_funded(proposal_id, funder_name, amount, total_raised)** + - Emails researcher + all funders + - WebSocket broadcast + - Template: "✅ {funder_name} contributed {amount} USDC" + - Persisted to notifications collection + +3. **notify_payment_received(user_id, amount, job_id, transaction_hash, basescan_url)** + - Emails specific user + - Template: "💰 You received {amount} USDC for job #{job_id}" + - Includes BaseScan URL for blockchain verification + - WebSocket notification if user is online + +4. **notify_fragment_claimed(proposal_id, fragment_id, researcher_id)** + - Emails original proposal owner + - Template: "🔬 Researcher claimed fragment: {fragment_title}" + - WebSocket notification + +5. **notify_results_published(proposal_id, ipfs_hash)** + - Emails researcher + all funders + - Template: "📊 Results published to IPFS: {ipfs_url}" + - IPFS URL: `https://ipfs.io/ipfs/{hash}` + - WebSocket + GossipSub broadcast + +6. **send_bulk_email(recipients, template_type, data)** + - Resend batch send API (up to 100 per request) + - 5 professional HTML email templates + - Rate limit handling (logs warnings) + - Graceful error handling + +### ✅ Email Templates + +Five professional HTML email templates: + +1. **new_proposal** - "🔬 New Research Proposal: {title}" +2. **proposal_funded** - "✅ Proposal Fully Funded: {title}" +3. **payment_received** - "💰 Payment Received: {amount} USDC" +4. **fragment_claimed** - "🔬 Research Fragment Claimed" +5. **results_published** - "📊 Research Results Published" + +Each template includes: +- Professional HTML styling +- Responsive design (max-width: 600px) +- Color-coded by notification type +- Call-to-action buttons +- Privacy footer + +### ✅ Notification Channels + +| Channel | Status | Features | +|---------|--------|----------| +| **Email (Resend)** | ✅ Implemented | Batch send, HTML templates, rate limit handling | +| **WebSocket** | ✅ Implemented | Real-time browser updates via ConnectionManager | +| **GossipSub PubSub** | ✅ Implemented | P2P network broadcasts (optional) | +| **MongoDB** | ✅ Implemented | Persistent audit trail in notifications collection | + +### ✅ MongoDB Integration + +- Uses existing `NotificationDocument` from `db.agentkit_collections` +- Persists all notifications for user history +- Tracks email delivery status +- Supports read/unread status + +### ✅ Error Handling + +- Email failures: Logged, doesn't crash +- WebSocket failures: Skips offline users +- GossipSub failures: Logged, continues +- MongoDB failures: Logged per document +- Missing config: Warnings, channels disabled + +### ✅ Configuration + +Environment variables (already in `.env.example`): + +```bash +RESEND_API_KEY=re_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx +RESEND_FROM_EMAIL=onboarding@resend.dev +RESEND_TO_EMAIL=your-email@example.com +PLATFORM_BASE_URL=http://localhost:3000 +QB2_MONGODB_LOCAL_URI=mongodb://127.0.0.1:27017 +QB2_MONGODB_DATABASE=qds +``` + +### ✅ Dependencies + +- **httpx**: ✅ Already in pyproject.toml (>=0.27.0) +- **motor**: ✅ Already installed (>=3.3.0) +- **Resend SDK**: ❌ NOT needed (using REST API directly) + +## Usage Example + +```python +from services.notification_service import NotificationService + +# Initialize +service = NotificationService() + +# Notify new proposal +await service.notify_new_proposal({ + "proposal_id": "prop_123", + "title": "Quantum Portfolio Optimization", + "description": "...", + "researcher_id": "researcher_001", + "budget_required": Decimal("5000.00"), + "funding_threshold": Decimal("2500.00"), + "deadline": datetime.now() + timedelta(days=30) +}) + +# Notify funding +await service.notify_proposal_funded( + proposal_id="prop_123", + funder_name="Dr. Alice Smith", + amount="1000.00", + total_raised="3500.00" +) + +# Notify payment +await service.notify_payment_received( + user_id="user_001", + amount="250.00", + job_id="job_456", + transaction_hash="0xabcdef...", + basescan_url="https://sepolia.basescan.org/tx/0xabcdef..." +) + +# Cleanup +await service.close() +``` + +## Integration Points + +### With FastAPI + +```python +from fastapi import APIRouter, Depends +from services.notification_service import get_notification_service + +@router.post("/proposals/{proposal_id}/fund") +async def fund_proposal( + proposal_id: str, + notification_service: NotificationService = Depends(get_notification_service) +): + # ... process funding + await notification_service.notify_proposal_funded(...) +``` + +### With GossipSub + +```python +from quantum_backend_v2.libp2p.pubsub import create_gossipsub_pubsub +from services.notification_service import NotificationService + +gossipsub, pubsub = create_gossipsub_pubsub(host) +service = NotificationService(pubsub_client=pubsub) +``` + +### With WebSocket + +Uses existing `ConnectionManager` from: +- `/backend/src/quantum_backend_v2/agent/websocket.py` + +## Testing + +Run example script: + +```bash +cd /path/to/backend/src +python -m services.notification_service_example +``` + +## Resend API Integration + +### No SDK Required + +We use Resend's REST API directly via httpx (cleaner, fewer dependencies). + +### API Endpoints Used + +- `POST /emails/batch` - Batch email sending + +### Authentication + +```python +headers = { + "Authorization": f"Bearer {RESEND_API_KEY}", + "Content-Type": "application/json" +} +``` + +### Rate Limits + +- **Free tier**: 100 emails/day, 3,000/month +- **Batch limit**: 100 emails per request +- **Sender**: Use `onboarding@resend.dev` for dev, verify domain for prod + +## File Structure + +``` +backend/src/services/ +├── __init__.py (updated with NotificationService exports) +├── notification_service.py (819 lines - main implementation) +├── notification_service_example.py (example usage) +└── README_NOTIFICATION_SERVICE.md (comprehensive documentation) +``` + +## Next Steps + +### Immediate Integration + +1. **Add to API routes**: Import in FastAPI routers that need notifications +2. **Initialize on startup**: Create global instance in app startup +3. **Test email sending**: Configure RESEND_API_KEY and test +4. **Wire GossipSub**: Pass pubsub client when available + +### Future Enhancements + +1. **User Preferences**: Add `notification_preferences` to PlatformUserDocument +2. **Email Queue**: Use Celery/RQ for async processing +3. **Retry Logic**: Exponential backoff for failed sends +4. **Analytics**: Track open/click rates via Resend webhooks +5. **More Templates**: Add templates for other event types +6. **SMS/Push**: Add Twilio (SMS) and Firebase (mobile push) + +## Summary + +✅ **All requirements met:** + +1. ✅ Multi-channel notifications (Email, WebSocket, GossipSub) +2. ✅ All 5 notification methods implemented +3. ✅ Professional HTML email templates +4. ✅ Resend API integration via httpx +5. ✅ MongoDB persistence +6. ✅ WebSocket broadcasts +7. ✅ GossipSub pubsub support +8. ✅ Graceful error handling +9. ✅ Rate limit handling +10. ✅ Comprehensive documentation +11. ✅ Example usage scripts + +**Total lines of code**: 819 (service) + 150 (examples) + 300 (docs) = ~1,270 lines + +**Dependencies**: All already installed (httpx, motor, pymongo) + +**Ready for production**: Yes, with proper RESEND_API_KEY configuration diff --git a/repository/backend/NOTIFICATION_SERVICE_INTEGRATION_GUIDE.md b/repository/backend/NOTIFICATION_SERVICE_INTEGRATION_GUIDE.md new file mode 100644 index 0000000..8ae268f --- /dev/null +++ b/repository/backend/NOTIFICATION_SERVICE_INTEGRATION_GUIDE.md @@ -0,0 +1,458 @@ +# NotificationService Integration Guide + +## Quick Start + +This guide shows how to integrate NotificationService into your FastAPI routes and background tasks. + +## Step 1: Import the Service + +```python +from services.notification_service import NotificationService, get_notification_service +``` + +## Step 2: Initialize on App Startup + +In your main FastAPI app (`main.py` or `app.py`): + +```python +from fastapi import FastAPI +from services.notification_service import NotificationService, set_notification_service + +app = FastAPI() + +@app.on_event("startup") +async def startup_event(): + # Initialize notification service + notification_service = NotificationService() + set_notification_service(notification_service) + print("✅ NotificationService initialized") + +@app.on_event("shutdown") +async def shutdown_event(): + # Cleanup notification service + from services.notification_service import get_notification_service + service = get_notification_service() + await service.close() + print("👋 NotificationService closed") +``` + +## Step 3: Use in API Routes + +### Example 1: Create New Proposal + +```python +from fastapi import APIRouter, Depends, BackgroundTasks +from services.notification_service import get_notification_service, NotificationService + +router = APIRouter(prefix="/proposals", tags=["proposals"]) + +@router.post("/") +async def create_proposal( + proposal_data: dict, + background_tasks: BackgroundTasks, + notification_service: NotificationService = Depends(get_notification_service) +): + # Save proposal to database + proposal = await db.research_proposals.insert_one({ + "proposal_id": generate_id(), + "title": proposal_data["title"], + "description": proposal_data["description"], + # ... other fields + "created_at": datetime.now(timezone.utc) + }) + + # Send notifications asynchronously + background_tasks.add_task( + notification_service.notify_new_proposal, + proposal + ) + + return {"status": "success", "proposal_id": proposal["proposal_id"]} +``` + +### Example 2: Fund Proposal + +```python +@router.post("/{proposal_id}/fund") +async def fund_proposal( + proposal_id: str, + funding_data: dict, + background_tasks: BackgroundTasks, + notification_service: NotificationService = Depends(get_notification_service) +): + # Process funding transaction + proposal = await db.research_proposals.find_one({"proposal_id": proposal_id}) + + # Update proposal with new funding + new_funder = { + "funder_id": funding_data["funder_id"], + "wallet_address": funding_data["wallet_address"], + "amount_usdc": funding_data["amount"], + "funded_at": datetime.now(timezone.utc) + } + + await db.research_proposals.update_one( + {"proposal_id": proposal_id}, + { + "$push": {"funders": new_funder}, + "$inc": {"budget_raised": funding_data["amount"]} + } + ) + + # Get updated proposal + updated_proposal = await db.research_proposals.find_one({"proposal_id": proposal_id}) + + # Send notifications + background_tasks.add_task( + notification_service.notify_proposal_funded, + proposal_id=proposal_id, + funder_name=funding_data["funder_name"], + amount=str(funding_data["amount"]), + total_raised=str(updated_proposal["budget_raised"]) + ) + + return {"status": "success", "total_raised": updated_proposal["budget_raised"]} +``` + +### Example 3: Process Payment + +```python +@router.post("/payments/callback") +async def payment_callback( + payment_data: dict, + background_tasks: BackgroundTasks, + notification_service: NotificationService = Depends(get_notification_service) +): + # Verify blockchain transaction + transaction_hash = payment_data["transaction_hash"] + basescan_url = f"https://sepolia.basescan.org/tx/{transaction_hash}" + + # Update payment record + await db.payments.update_one( + {"payment_id": payment_data["payment_id"]}, + { + "$set": { + "status": "confirmed", + "transaction_hash": transaction_hash, + "basescan_url": basescan_url, + "completed_at": datetime.now(timezone.utc) + } + } + ) + + # Send notification + background_tasks.add_task( + notification_service.notify_payment_received, + user_id=payment_data["to_user_id"], + amount=str(payment_data["amount"]), + job_id=payment_data["job_id"], + transaction_hash=transaction_hash, + basescan_url=basescan_url + ) + + return {"status": "confirmed"} +``` + +### Example 4: Claim Research Fragment + +```python +@router.post("/{proposal_id}/fragments/{fragment_id}/claim") +async def claim_fragment( + proposal_id: str, + fragment_id: str, + researcher_id: str, + background_tasks: BackgroundTasks, + notification_service: NotificationService = Depends(get_notification_service) +): + # Assign fragment to researcher + await db.research_proposals.update_one( + {"proposal_id": proposal_id}, + {"$push": {"claimed_fragments": { + "fragment_id": fragment_id, + "researcher_id": researcher_id, + "claimed_at": datetime.now(timezone.utc) + }}} + ) + + # Send notification + background_tasks.add_task( + notification_service.notify_fragment_claimed, + proposal_id=proposal_id, + fragment_id=fragment_id, + researcher_id=researcher_id + ) + + return {"status": "claimed"} +``` + +### Example 5: Publish Research Results + +```python +@router.post("/{proposal_id}/results/publish") +async def publish_results( + proposal_id: str, + results_data: dict, + background_tasks: BackgroundTasks, + notification_service: NotificationService = Depends(get_notification_service) +): + # Upload results to IPFS + ipfs_hash = await upload_to_ipfs(results_data["file"]) + + # Update proposal with results + await db.research_proposals.update_one( + {"proposal_id": proposal_id}, + { + "$set": { + "results_ipfs_hash": ipfs_hash, + "status": "completed", + "completed_at": datetime.now(timezone.utc) + } + } + ) + + # Send notifications + background_tasks.add_task( + notification_service.notify_results_published, + proposal_id=proposal_id, + ipfs_hash=ipfs_hash + ) + + return { + "status": "published", + "ipfs_hash": ipfs_hash, + "ipfs_url": f"https://ipfs.io/ipfs/{ipfs_hash}" + } +``` + +## Step 4: Initialize with GossipSub (Optional) + +If you want P2P network broadcasts: + +```python +from quantum_backend_v2.libp2p.pubsub import create_gossipsub_pubsub +from services.notification_service import NotificationService, set_notification_service + +@app.on_event("startup") +async def startup_event(): + # Create libp2p host (existing setup) + host = await create_libp2p_host() + + # Create GossipSub pubsub + gossipsub, pubsub = create_gossipsub_pubsub(host) + + # Start pubsub service + await pubsub.start() + + # Initialize notification service with pubsub + notification_service = NotificationService(pubsub_client=pubsub) + set_notification_service(notification_service) +``` + +## Step 5: User Notification Preferences (Future) + +When you add notification preferences to `PlatformUserDocument`: + +```python +# In mongodb.py or models.py +class PlatformUserDocument(Document): + # ... existing fields + notification_preferences: dict[str, bool] = Field(default_factory=lambda: { + "new_proposals": True, + "proposal_funded": True, + "payment_received": True, + "fragment_claimed": True, + "results_published": True, + "email_enabled": True, + "websocket_enabled": True + }) +``` + +Then update the service to check preferences: + +```python +# In notification_service.py, notify_new_proposal method: +users_cursor = self.db.platform_users.find({ + "is_active": True, + "notification_preferences.new_proposals": True, + "notification_preferences.email_enabled": True +}) +``` + +## Testing Notifications + +### Local Testing + +1. Set up Resend API key in `.env`: +```bash +RESEND_API_KEY=re_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx +RESEND_FROM_EMAIL=onboarding@resend.dev +RESEND_TO_EMAIL=your-test-email@example.com +``` + +2. Run the example script: +```bash +cd backend/src +python -m services.notification_service_example +``` + +3. Check email in your inbox and Resend logs at https://resend.com/logs + +### Integration Testing + +```python +import pytest +from services.notification_service import NotificationService + +@pytest.fixture +async def notification_service(): + service = NotificationService() + yield service + await service.close() + +@pytest.mark.asyncio +async def test_notify_new_proposal(notification_service): + proposal = { + "proposal_id": "test_123", + "title": "Test Proposal", + "description": "Test description", + "researcher_id": "researcher_test", + "budget_required": Decimal("1000.00"), + "funding_threshold": Decimal("500.00"), + "deadline": datetime.now() + timedelta(days=7) + } + + # Should not raise exception + await notification_service.notify_new_proposal(proposal) + + # Check notification was created in MongoDB + notification = await db.notifications.find_one({"data.proposal_id": "test_123"}) + assert notification is not None + assert notification["type"] == "other" +``` + +## Common Patterns + +### Pattern 1: Fire-and-Forget (Background Task) + +Use FastAPI's `BackgroundTasks` for non-critical notifications: + +```python +background_tasks.add_task( + notification_service.notify_new_proposal, + proposal +) +``` + +### Pattern 2: Await Notification (Synchronous) + +For critical notifications where you need to know if they were sent: + +```python +success = await notification_service.send_bulk_email( + recipients=recipients, + template_type="payment_received", + data=template_data +) + +if not success: + logger.error("Failed to send payment notification email") + # Maybe retry or alert admin +``` + +### Pattern 3: Batch Notifications + +For sending to many users: + +```python +# Get all users who need notification +users = await db.platform_users.find({ + "is_active": True, + "notification_preferences.new_proposals": True +}).to_list(length=1000) + +# Split into batches of 100 (Resend limit) +for i in range(0, len(users), 100): + batch = users[i:i+100] + recipients = [{"email": u["email"], "name": u["display_name"]} for u in batch] + + await notification_service.send_bulk_email( + recipients=recipients, + template_type="new_proposal", + data=template_data + ) +``` + +## Troubleshooting + +### Issue: Emails not sending + +**Check:** +1. `RESEND_API_KEY` is set in `.env` +2. Email is valid format +3. Rate limits not exceeded (100/day free tier) +4. Check Resend logs: https://resend.com/logs + +**Solution:** +```bash +# Verify API key +curl https://api.resend.com/emails/batch \ + -H "Authorization: Bearer $RESEND_API_KEY" \ + -H "Content-Type: application/json" \ + -d '[{"from":"onboarding@resend.dev","to":["your-email@example.com"],"subject":"Test","html":"

Test

"}]' +``` + +### Issue: WebSocket not receiving notifications + +**Check:** +1. WebSocket connection is established +2. Session ID matches between connection and notification +3. `ConnectionManager` is initialized + +**Solution:** +```python +# Check active WebSocket connections +from quantum_backend_v2.agent.websocket import manager +print(f"Active connections: {manager.active_connections}") +``` + +### Issue: GossipSub not broadcasting + +**Check:** +1. `pubsub_client` is passed to NotificationService +2. GossipSub is started +3. Topic "proposals" is subscribed + +**Solution:** +```python +# Verify pubsub is working +topics = await pubsub.get_subscribed_topics() +print(f"Subscribed topics: {topics}") +``` + +## Environment Variables Checklist + +```bash +# Required +✅ RESEND_API_KEY=re_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx +✅ QB2_MONGODB_LOCAL_URI=mongodb://127.0.0.1:27017 +✅ QB2_MONGODB_DATABASE=qds + +# Optional but recommended +✅ RESEND_FROM_EMAIL=onboarding@resend.dev +✅ PLATFORM_BASE_URL=http://localhost:3000 +``` + +## Next Steps + +1. ✅ Import NotificationService in your API routes +2. ✅ Add `Depends(get_notification_service)` to endpoints +3. ✅ Use `BackgroundTasks` for async notifications +4. ✅ Configure Resend API key in `.env` +5. ✅ Test with example script +6. ✅ Deploy and monitor Resend logs + +## Support + +- Resend Docs: https://resend.com/docs +- Resend Logs: https://resend.com/logs +- NotificationService README: [Notification service guide](src/services/README_NOTIFICATION_SERVICE.md) diff --git a/repository/backend/PAYMENT_LEDGER.md b/repository/backend/PAYMENT_LEDGER.md new file mode 100644 index 0000000..14d7d93 --- /dev/null +++ b/repository/backend/PAYMENT_LEDGER.md @@ -0,0 +1,277 @@ +# Payment events and credit ledger — Part of #3 + +Every payment rail (x402, cards, Zcash, swaps, on-chain transfers) records the +same kind of payment event, and value that enters the platform's internal credit +system is written to one append-only, USDC-denominated ledger. The aim is +"accept anywhere and settle once": idempotency, reconciliation and +double-payment protection exist once instead of once per rail. + +Code: `src/quantum_backend_v2/payments/` (`models.py`, `ledger.py`, `errors.py`). + +## Collections + +| Collection | Contents | Unique on | +| --- | --- | --- | +| `payment_events` | One document per payment, with its status history | `idempotency_key` | +| `credit_ledger` | Append-only entries: `credit`, `hold`, `capture`, `release` | `idempotency_key` | +| `credit_holds` | Guard: amount still capturable or releasable per hold | `hold_id` | +| `credit_balances` | Guard: cached `available` and `held` per account | `account_id` | + +`payment_events` also indexes `job_id`, `proposal_id`, `status` and +`(status, created_at)`. Money is stored as `Decimal128`, never as floats. +Indexes are created at app startup and before a `LedgerService` first writes. + +## Payment events + +`record_event` stores an event once per idempotency key. Replaying a key returns +the stored event with `created=False`. Replaying it with a different rail, +purpose, asset, amount or ledger account raises `IdempotencyConflictError`. + +``` +pending ──> confirmed ──> refunded + │ └──> failed + └──> uncertain ──> confirmed | failed (admin reconciliation only) +``` + +- `confirm` (pending → confirmed) credits the event's `ledger_account_id`, if it + has one. Calling it again is harmless and finishes a credit a crash interrupted. +- `mark_uncertain` is for payments whose outcome is unknown, such as a timeout after a + transaction may have been broadcast. Replaying an uncertain key raises + `PaymentUncertainError`, and `confirm` refuses it: **uncertain payments are + never retried automatically.** +- `reconcile` lets an admin resolve a pending or uncertain event as confirmed or + failed after checking the rail. The admin's id and reason go into the status history. +- `refunded` is reserved. Refund policy is out of scope for #3. + +### Choosing idempotency keys + +Use a key that names the payment itself, so a retry produces the same key: + +| Payment | Key | +| --- | --- | +| Stripe credit pack (#10) | `stripe:checkout:` | +| Fragment payout | `fragment_payout:::` | +| Full escrow release | `escrow_release::` | +| Proposal funding | `proposal_funding::` | +| USDC transfer | `metadata["idempotency_key"]`, else `transfer:` | + +Use the Stripe session id rather than the event id: an asynchronously paid +session produces two events with different ids for the same payment. + +## Credit ledger + +Ledger entries are never updated or deleted. An account's balance is defined by +`derive_balance`: credits add to `available`; a hold moves value from +`available` to `held`; a capture spends held value; a release returns it. +`LedgerService.balance` always sums the entries; it never reads the guards. + +```python +await ledger.hold(account, Decimal("12"), hold_id=job_id, job_id=job_id) # on submit +await ledger.capture(job_id, actual_cost, idempotency_key=f"capture:{job_id}") +await ledger.release(job_id, idempotency_key=f"release:{job_id}") # the rest +``` + +### Concurrency without transactions + +Our MongoDB runs as a standalone server, which has no multi-document +transactions. Instead, each hold, capture or release changes its guard +document with one atomic conditional update: + +- a hold succeeds only if `credit_balances.available >= amount`; +- a capture or release succeeds only if `credit_holds.remaining >= amount`; +- a full release compares and sets the remainder it read. + +The ledger entry's unique key decides between concurrent replays of the same +operation; the loser undoes its guard change and returns the winner's entry. + +Writes are ordered so a crash between two steps always **fails closed**: value +can end up reserved and unusable until reconciled, but never spendable twice. +Credits write the ledger before the guard; holds, captures and releases change +the guard first. A hold guard lost between two inserts is rebuilt from the +ledger on the next capture or release. + +## Reconciliation + +`run_reconciliation` runs every `QB2_PAYMENT_RECONCILE_INTERVAL_SECONDS` +(default 300; `0` disables) and on demand. It: + +1. marks pending events older than 30 minutes that have **no external + reference** as uncertain. Their outcome was never recorded. Events + with a tx hash or Stripe id are awaiting their rail and are left alone; +2. compares each account's `credit_balances` guard with its ledger sum and + reports drift. It does not repair drift, so an operator can find the cause first. + +## Admin API + +All routes require the admin role. None of them can write ledger entries. + +| Method | Path | Purpose | +| --- | --- | --- | +| GET | `/api/v1/admin/payments/events?status=uncertain` | List events (also `job_id`, `proposal_id`, `limit`) | +| GET | `/api/v1/admin/payments/events/{key}` | One event with its status history | +| POST | `/api/v1/admin/payments/events/{key}/reconcile` | `{"outcome": "confirmed" \| "failed", "reason": "...", "external_ref": "0x..."}` | +| POST | `/api/v1/admin/payments/reconciliation` | Run the sweep now | +| GET | `/api/v1/admin/payments/accounts/{account_id}/balance` | Balance derived from the ledger | + +## First consumers + +The existing Base Sepolia paths write events without changing their responses: + +- `AgentKitService.transfer_usdc` records a pending event before sending and + confirms it with the tx hash. +- `ProposalService.fund_proposal` records a confirmed event for the escrow deposit. +- `_pay_fragment_researcher` and `_release_all_escrow` record a pending event per + attempt before withdrawing. A new attempt is allowed only after every earlier + one failed, so resubmitting results no longer pays a researcher twice. + +A send that fails with `ValueError` (validation, before anything is sent) marks the +event failed. Any other error marks it uncertain. If updating an event fails +after funds moved, the error is logged and the sweep flags the event later. + +On-chain events are confirmed when the transaction is broadcast, as the legacy +`payments` records are. Receipt-based confirmation is a follow-up. + +## Integrating a new rail + +1. `record_event(...)` with a key that names the payment, before value moves. + Set `ledger_account_id` if the payment should become credit. +2. Move the value only if `created` is true. +3. `confirm(key, external_ref=...)` when it settles, `fail` when it definitely + did not, and `mark_uncertain` when you cannot tell. + +## Prepaid credits (#10) + +Users can buy credit packs by card and spend the credits on circuit runs. One +credit is one USDC, so card-bought credits sit in the same ledger as everything else. + +``` +Settings → POST /api/billing/credits/checkout → Stripe Checkout (mode: payment) +Stripe → /api/auth/stripe/webhook (signature check, event-id de-dupe) + → line items → credits (from STRIPE_CREDIT_PACKS, never the client) + → POST /api/v1/internal/billing/credit-purchases (X-Billing-Internal-Secret) + → payment event stripe:checkout: → ledger credit +``` + +**Buying.** Packs are Stripe one-time Prices mapped to credit amounts in +`STRIPE_CREDIT_PACKS` (frontend). The webhook handles credit-pack sessions itself +and does not forward them to the subscription plugin. It works out the credits from +the session's line items as Stripe recorded them. A session paid by a delayed +method completes unpaid and is credited on +`checkout.session.async_payment_succeeded`. + +**Exactly once.** The frontend de-duplicates Stripe events by id. The backend +keys the purchase on the Checkout Session (`stripe:checkout:`), so a +redelivered event, or the two events of one async session, credit once. A +redelivery also finishes a credit that a crash interrupted. + +**Spending.** With `QB2_CIRCUIT_JOB_CREDIT_COST` above 0, submitting a circuit holds +its cost under `job:`, or returns `402 insufficient_credits`. A completed job +captures the hold and a failed one releases it. If the server stops mid-job, the +hold stays reserved and is never spent twice. Releasing holds for jobs that never +finish is a follow-up; there is no admin route for it yet. The default of 0 +keeps runs free. x402 runs are paid per request and never use credits. + +**Identity.** Runs and balances use the signed `qn1` assertion, so both sides need +the same `AGENT_AUTH_BRIDGE_SECRET`. Without it, runs are submitted anonymously as +before and are not charged to a user. + +### Configuration + +| Side | Variable | Purpose | +| --- | --- | --- | +| Frontend | `STRIPE_SECRET_KEY`, `STRIPE_PRICE_ID`, `STRIPE_WEBHOOK_SECRET` | Stripe billing (#42); required for packs too | +| Frontend | `STRIPE_CREDIT_PACKS` | `price_abc:10,price_def:50` | +| Both | `BILLING_INTERNAL_SECRET` / `QB2_BILLING_INTERNAL_SECRET` | Same 32+ character secret; guards the purchase route | +| Both | `AGENT_AUTH_BRIDGE_SECRET` | Signs the user's identity for runs and balances | +| Backend | `QB2_CIRCUIT_JOB_CREDIT_COST` | Credits per circuit run; `0` disables charging | +| Frontend | `STRIPE_STABLECOIN_PAYMENTS` | `true` also offers USD packs in USDC (#76); off by default | + +### Manual test (Stripe CLI, test mode) + +1. In the Stripe Dashboard (test mode), create a product with a **one-time** price, + e.g. $10, and set `STRIPE_CREDIT_PACKS=price_...:10`. +2. Set the secrets above, `QB2_CIRCUIT_JOB_CREDIT_COST=2`, and restart both apps. +3. `stripe listen --forward-to localhost:3000/api/auth/stripe/webhook` +4. Settings → Credits → **Buy 10 credits · $10.00**, pay with `4242 4242 4242 4242`. + The balance shows 10. +5. `stripe events resend `: the webhook returns `duplicate: true` and the + balance stays 10. `GET /api/v1/admin/payments/events/stripe:checkout:` + shows one confirmed card event. +6. Submit a circuit run: the balance drops to 8, with 2 reserved while it runs and + spent once it completes. With fewer than 2 credits, submitting shows the 402 message. + +### Stablecoin credit packs (#76) + +With `STRIPE_STABLECOIN_PAYMENTS=true`, each USD-priced pack can also be bought +in stablecoins through Stripe's `crypto` payment method. The customer pays from +their own wallet on crypto.stripe.com and Stripe settles to our balance in USD. We +hold no wallet keys, run no on-chain code and store no payment details. + +``` +Settings → "… in USDC" → POST /api/billing/credits/checkout {priceId, method: "crypto"} + → Checkout Session with payment_method_types: ["crypto"] +Stripe → webhook → session retrieved with payment_intent.payment_method expanded + → POST …/credit-purchases {…, payment_method: "crypto"} + → payment event stripe:checkout:, rail "stablecoin" +``` + +- **Hidden when off.** Once stablecoins are enabled in the Stripe Dashboard, + Stripe shows them on every Checkout. Card sessions therefore always pass + `excluded_payment_method_types: ["crypto"]`, and a stablecoin session is only + created when the flag is on and the pack's Price is in USD. +- **Exactly once.** Nothing new: the purchase is still keyed on the Checkout + Session, so a redelivered event, or `completed` followed by + `async_payment_succeeded`, credits once. The rail is part of the event's + identity, so one session can never be recorded as both card and stablecoin. +- **Rail.** Only Stripe's `crypto` method is recorded on the `stablecoin` rail. + Cards and Stripe's other dynamic methods stay on `card`. + +**Not supported (per Stripe's docs):** + +- **Disputes.** Stablecoin payments cannot be disputed, so there are no chargebacks + to claw credits back. +- **Manual capture.** Stablecoin payments cannot be authorized and captured later. + Credit packs never relied on that: credits are granted once the session is paid, + and job holds are taken from the internal ledger, not from Stripe. +- **Refunds** are supported, full or partial, and go back as stablecoins to the + paying wallet. Refunding does not yet debit the credits it granted; that applies + to card refunds too and is a follow-up. +- **Limits.** Stablecoin Checkout needs USD prices, US$10,000 at most per + transaction, and a Stripe account approved for "Stablecoins and Cryptocurrency". + Stripe currently offers it to US businesses (not New York); other regions are in + private preview. Stablecoin subscriptions are out of scope. + +**Manual test (sandbox).** + +1. Dashboard → Settings → Payment methods: request **Stablecoins and Cryptocurrency** + and wait until it is active. +2. Use a USD one-time price in `STRIPE_CREDIT_PACKS`, set + `STRIPE_STABLECOIN_PAYMENTS=true` and restart the frontend. +3. Fund a MetaMask wallet on Polygon Amoy with test USDC (faucet.circle.com) and a + little POL for gas (faucet.polygon.technology). +4. `stripe listen --forward-to localhost:3000/api/auth/stripe/webhook`, then + Settings → Credits → **Buy 10 credits · $10.00 in USDC** and pay on + crypto.stripe.com. The balance shows 10. +5. `stripe events resend `: `duplicate: true`, the balance stays 10, and + `GET /api/v1/admin/payments/events/stripe:checkout:` shows one confirmed + `stablecoin` event. +6. With the flag off, the USDC buttons disappear and card Checkout no longer + lists crypto. + +## Tests + +```bash +uv run pytest tests/unit/test_payment_ledger_models.py tests/unit/test_payments_admin_api.py tests/unit/test_credits_api.py +# Frontend webhook and credit-pack tests: +(cd ../frontend && bun run test:billing) +# Idempotency and concurrency need a real server (disposable local MongoDB only): +QB2_MONGODB_TEST_URI=mongodb://127.0.0.1:27017 uv run pytest tests/live -q +``` + +The live tests cover replayed keys, concurrent confirms, concurrent holds that +would overspend, concurrent double captures and releases, rebuilding a lost hold +guard, drift detection, randomized concurrent traffic that must keep guards equal +to ledger sums, and the payout paths (paid once, never retried when uncertain). +`tests/live/test_credits_mongo.py` covers credit purchases delivered twice or +concurrently, the two events of one async session, crash recovery, and job +metering that charges only completed jobs. diff --git a/repository/backend/PROGRESS.md b/repository/backend/PROGRESS.md new file mode 100644 index 0000000..9979ec7 --- /dev/null +++ b/repository/backend/PROGRESS.md @@ -0,0 +1,663 @@ +# AgentKit Integration Progress Report + +**Last Updated**: May 20, 2026 05:15 UTC +**Branch**: `feat/agentkit-integration` +**Status**: 🎉 **PHASE 1-4 COMPLETE** | **Tests: 16/30 Passing (53%)** + +--- + +## 📊 Executive Summary + +### Completion Status: 100% ✅ + +**Total Implementation:** +- **172 Python files** in src/ +- **32,171 lines** of production code +- **8,688 lines** of test code +- **232 commits** in May 2026 +- **47 files changed** from main branch +- **19,915+ lines** of new code + +### Phase Breakdown +- ✅ **Phase 1**: AgentKit Integration (100%) +- ✅ **Phase 2**: Marketplace & AI Services (100%) +- ✅ **Phase 3**: Notification System (100%) +- ✅ **Phase 4**: Proposal Service (100%) + +--- + +## 🎯 Completed Features + +### Phase 1: AgentKit Integration (100% Complete) + +#### Core Infrastructure ✅ +- [x] **AgentKit SDK Integration** + - AWS Bedrock client setup with Claude Sonnet 4.5 + - Model configuration and streaming support + - Cost tracking and usage monitoring + - Error handling and retry logic + +- [x] **Agent Service Layer** (`src/services/agentkit_service.py` - 825 lines) + - Agent CRUD operations with MongoDB + - Wallet management and payment integration + - AI-powered agent creation and configuration + - Agent discovery and search capabilities + - Session management + +- [x] **AI Agent Service** (`src/services/ai_agent_service.py` - 842 lines) + - Claude API integration with Bedrock + - Conversation management + - Context preservation + - Token usage tracking + - Streaming responses + +#### Data Models ✅ +- [x] **Agent Models** (`src/models/agent.py` - 228 lines) + - `Agent` - Complete agent configuration + - `AgentSession` - Session state management + - `AgentMessage` - Message history + - `AgentCapability` - Skill definitions + - `AIModel` - Model configurations + +- [x] **Wallet Models** (`src/models/wallet.py` - 193 lines) + - `Wallet` - Multi-currency wallet + - `Transaction` - Payment history + - `WalletBalance` - Token balances + +- [x] **Payment Models** (`src/models/payment.py` - 151 lines) + - `Payment` - Payment processing + - `PaymentMethod` - Payment sources + - `Refund` - Refund tracking + +#### API Endpoints ✅ +- [x] **Agent Router** (`src/api/v1/agents.py` - 689 lines) + - `POST /api/v1/agents/` - Create agent + - `GET /api/v1/agents/` - List agents with filters + - `GET /api/v1/agents/{agent_id}` - Get agent details + - `PUT /api/v1/agents/{agent_id}` - Update agent + - `DELETE /api/v1/agents/{agent_id}` - Delete agent + - `POST /api/v1/agents/{agent_id}/sessions` - Create session + - `POST /api/v1/agents/{agent_id}/chat` - Chat with agent + - `GET /api/v1/agents/search` - Search agents + +- [x] **Wallet Router** (`src/api/v1/wallet.py` - 558 lines) + - `POST /api/v1/wallet/` - Create wallet + - `GET /api/v1/wallet/{wallet_id}` - Get wallet + - `GET /api/v1/wallet/{wallet_id}/balance` - Get balance + - `POST /api/v1/wallet/{wallet_id}/deposit` - Deposit funds + - `POST /api/v1/wallet/{wallet_id}/withdraw` - Withdraw funds + - `POST /api/v1/wallet/{wallet_id}/transfer` - Transfer funds + - `GET /api/v1/wallet/{wallet_id}/transactions` - Transaction history + +#### Database Layer ✅ +- [x] **MongoDB Collections** (`src/db/agentkit_collections.py` - 316 lines) + - Agent collection with indexes + - Session collection with TTL + - Message collection with partitioning + - Transaction collection with audit trail + - Wallet collection with balance tracking + +#### Documentation ✅ +- [x] **Technical Specs** + - `docs/superpowers/AGENTKIT_CONTEXT.md` (683 lines) + - `docs/adr/2026-05-20-agentkit-crowdfunding-design.md` (1,514 lines) + - AWS Bedrock setup guide + - Integration patterns + +- [x] **API Documentation** + - Complete OpenAPI schemas + - Request/response examples + - Authentication flows + - Error handling patterns + +--- + +### Phase 2: Marketplace & AI Services (100% Complete) + +#### Marketplace Service ✅ +- [x] **Core Implementation** (`src/services/marketplace_service.py` - 889 lines) + - Agent listing and discovery + - Search with filters (categories, pricing, ratings) + - Reputation system with reviews + - Featured agents and promotions + - Analytics and metrics tracking + +- [x] **API Router** (`src/api/v1/marketplace.py` - 440 lines) + - `GET /api/v1/marketplace/agents` - Browse agents + - `GET /api/v1/marketplace/agents/{agent_id}` - Agent details + - `POST /api/v1/marketplace/agents/{agent_id}/reviews` - Submit review + - `GET /api/v1/marketplace/categories` - List categories + - `GET /api/v1/marketplace/featured` - Featured agents + - `GET /api/v1/marketplace/analytics` - Marketplace stats + +#### AI Services Integration ✅ +- [x] **AWS Bedrock Setup** + - Claude Sonnet 4.5 model configuration + - Streaming API support + - Cost optimization with caching + - Multi-region failover + +- [x] **AI Capabilities** + - Natural language understanding + - Intent classification + - Context-aware responses + - Token usage optimization + - Cost guard implementation + +#### Utilities ✅ +- [x] **IPFS Integration** (`src/utils/ipfs.py` - 252 lines) + - File upload/download + - CID generation and verification + - Pinning service integration + - Metadata management + +- [x] **Basescan Integration** (`src/utils/basescan.py` - 154 lines) + - Transaction verification + - Wallet balance checks + - Contract interaction + - Event monitoring + +--- + +### Phase 3: Notification System (100% Complete) + +#### Notification Service ✅ +- [x] **Core Implementation** (`src/services/notification_service.py` - 819 lines) + - Multi-channel delivery (email, SMS, push, in-app) + - Template management with variables + - Scheduling and batching + - Priority queue system + - Retry logic with exponential backoff + - Delivery tracking and analytics + +- [x] **Notification Models** (`src/models/notification.py` - 196 lines) + - `Notification` - Base notification model + - `NotificationTemplate` - Template definitions + - `NotificationPreference` - User preferences + - `NotificationChannel` - Channel configurations + +#### API Endpoints ✅ +- [x] **Notification Router** (`src/api/v1/notifications.py` - 437 lines) + - `POST /api/v1/notifications/` - Send notification + - `GET /api/v1/notifications/` - List notifications + - `GET /api/v1/notifications/{notification_id}` - Get notification + - `PATCH /api/v1/notifications/{notification_id}/read` - Mark as read + - `POST /api/v1/notifications/batch` - Batch send + - `GET /api/v1/notifications/preferences` - Get preferences + - `PUT /api/v1/notifications/preferences` - Update preferences + - `GET /api/v1/notifications/analytics` - Delivery stats + +#### Integration Features ✅ +- [x] **Event-Driven Architecture** + - Agent event notifications + - Payment confirmations + - Proposal status updates + - Session reminders + +- [x] **Delivery Providers** + - Email (SMTP/SendGrid) + - SMS (Twilio) + - Push (Firebase) + - WebSocket (real-time) + +#### Documentation ✅ +- [x] **Implementation Guides** + - `backend/NOTIFICATION_SERVICE_IMPLEMENTATION.md` (276 lines) + - `backend/NOTIFICATION_SERVICE_INTEGRATION_GUIDE.md` (458 lines) + - `backend/src/services/README_NOTIFICATION_SERVICE.md` (410 lines) + - Channel setup instructions + - Template creation examples + - Integration patterns + +--- + +### Phase 4: Proposal Service - The Crown Jewel (100% Complete) + +#### Core Service ✅ +- [x] **Proposal Service** (`src/services/proposal_service.py` - 1,093 lines) + - **Quantum Job Proposals** + - Auto-fragmentation for distributed execution + - Coalition formation algorithms + - Resource allocation optimization + - **AI-Powered Analysis** + - Claude integration for proposal evaluation + - Cost-benefit analysis + - Risk assessment + - Feasibility scoring + - **Crowdfunding System** + - Campaign creation and management + - Milestone-based funding + - Investor relations + - Fund disbursement + - **Coalition Management** + - Multi-agent coordination + - Resource pooling + - Profit sharing + - Reputation tracking + - **Workflow Orchestration** + - Task fragmentation + - Dependency resolution + - Parallel execution + - Result aggregation + +#### Proposal Models ✅ +- [x] **Data Models** (`src/models/proposal.py` - 365 lines) + - `Proposal` - Complete proposal structure + - `ProposalStatus` - State machine + - `ProposalFragment` - Task decomposition + - `Coalition` - Multi-agent teams + - `CrowdfundingCampaign` - Funding campaigns + - `Milestone` - Progress tracking + - `Investment` - Investor records + - `ResourceAllocation` - Resource distribution + +#### API Endpoints ✅ +- [x] **Proposal Router** (`src/api/v1/proposals.py` - 901 lines) + - **Proposal Management** + - `POST /api/v1/proposals/` - Create proposal + - `GET /api/v1/proposals/` - List proposals + - `GET /api/v1/proposals/{proposal_id}` - Get details + - `PUT /api/v1/proposals/{proposal_id}` - Update proposal + - `DELETE /api/v1/proposals/{proposal_id}` - Delete proposal + - **Fragmentation & Coalition** + - `POST /api/v1/proposals/{proposal_id}/fragment` - Auto-fragment + - `POST /api/v1/proposals/{proposal_id}/coalition` - Form coalition + - `GET /api/v1/proposals/{proposal_id}/coalition` - Coalition status + - `POST /api/v1/proposals/{proposal_id}/coalition/join` - Join coalition + - **Crowdfunding** + - `POST /api/v1/proposals/{proposal_id}/crowdfund` - Start campaign + - `GET /api/v1/proposals/{proposal_id}/crowdfund` - Campaign status + - `POST /api/v1/proposals/{proposal_id}/invest` - Make investment + - `GET /api/v1/proposals/{proposal_id}/investors` - List investors + - `POST /api/v1/proposals/{proposal_id}/milestones` - Add milestone + - `PUT /api/v1/proposals/{proposal_id}/milestones/{milestone_id}` - Update + - **Execution & Analysis** + - `POST /api/v1/proposals/{proposal_id}/execute` - Execute proposal + - `POST /api/v1/proposals/{proposal_id}/analyze` - AI analysis + - `GET /api/v1/proposals/{proposal_id}/analytics` - Performance metrics + +#### AI-Powered Features ✅ +- [x] **Claude Integration Prompts** (`backend/prompts/`) + - `auto_fragmentation.txt` (71 lines) - Task decomposition + - `coalition_formation.txt` (80 lines) - Team assembly + - `proposal_analysis.txt` (52 lines) - Evaluation criteria + - `result_summarization.txt` (109 lines) - Result aggregation + - `README.md` (85 lines) - Prompt engineering guide + +#### Advanced Capabilities ✅ +- [x] **Quantum Job Distribution** + - Automatic task fragmentation based on complexity + - Resource requirement analysis + - Cost optimization algorithms + - Network topology awareness + +- [x] **Coalition Intelligence** + - Skill matching algorithms + - Reputation-based selection + - Load balancing + - Conflict resolution + +- [x] **Crowdfunding Intelligence** + - ROI prediction models + - Risk assessment scoring + - Investor matching + - Milestone verification + +- [x] **Analytics & Reporting** + - Real-time progress tracking + - Cost analysis dashboards + - Performance benchmarking + - Anomaly detection + +#### Documentation ✅ +- [x] **Comprehensive Guides** + - `backend/PROPOSAL_SERVICE_DOCUMENTATION.md` (675 lines) + - Architecture overview + - API reference + - Integration patterns + - Code examples + - Best practices + - Design document in ADR format + +--- + +## 🧪 Testing Infrastructure (53% Passing) + +### Integration Tests ✅ +- [x] **Agent Funding Flow** (`test_agent_funding_flow.py` - 504 lines) + - Agent registration and wallet creation + - Payment processing and escrow + - Multi-agent collaboration + - Revenue distribution + +- [x] **Crowdfunding Flow** (`test_crowdfunding_flow.py` - 459 lines) + - Campaign lifecycle + - Investment processing + - Milestone completion + - Fund disbursement + +- [x] **Marketplace Flow** (`test_marketplace_flow.py` - 403 lines) + - Agent discovery + - Review system + - Featured listings + - Analytics tracking + +- [x] **Payment Flow** (`test_payment_flow.py` - 293 lines) + - Wallet operations + - Transaction processing + - Balance verification + - Refund handling + +### Service Tests ✅ +- [x] **AgentKit Service Tests** (`test_agentkit_service.py` - 556 lines) - **10/14 passing (71%)** + - ✅ Create wallet duplicate detection + - ✅ Get balance success + - ✅ Transfer insufficient balance validation + - ✅ Request testnet funds + - ✅ Load wallet not found error + - ✅ Invalid entity type validation + - ✅ Invalid address format validation + - ✅ Negative amount validation + - ✅ Get balance invalid address + - ✅ Transfer insufficient balance check + - ⚠️ Wallet creation (mocking issues) + - ⚠️ USDC transfer (mocking issues) + - ⚠️ Aave escrow operations (mocking issues) + - ⚠️ Wallet seed encryption (mocking issues) + +- [x] **Proposal Service Tests** (`test_proposal_service.py` - 845 lines) - **7/16 passing (44%)** + - ✅ Claim fragment already claimed + - ✅ Auto-fragment calls Bedrock + - ✅ Broadcast triggers agents + - ✅ Create proposal researcher not found + - ✅ Fund proposal deadline passed + - ✅ Claim fragment proposal not funded + - ✅ Submit results unauthorized researcher + - ⚠️ Proposal creation (Beanie Document issues) + - ⚠️ Funding operations (dependency mocking) + - ⚠️ Fragment claiming (type errors) + - ⚠️ Results submission (mocking issues) + - ⚠️ Payment distribution (validation errors) + +**Overall Service Tests: 16/30 passing (53%)** ✅ + +### Test Configuration ✅ +- [x] **Test Fixtures** (`conftest.py` - 252 lines) + - MongoDB test database + - Mock AWS Bedrock client + - Test data factories + - Cleanup utilities + +--- + +## 🏗️ Architecture Achievements + +### System Design ✅ +- [x] **Clean Architecture** + - Separation of concerns (API, Service, Data layers) + - Dependency injection + - Interface-based design + - Testability built-in + +- [x] **Scalability** + - Horizontal scaling with load balancing + - Database sharding support + - Caching strategies + - Async processing + +- [x] **Security** + - JWT authentication + - Role-based access control (RBAC) + - Input validation with Pydantic + - SQL injection prevention + - Rate limiting + +### Technology Stack ✅ +- [x] **Backend Framework** + - FastAPI 0.115.0+ + - Uvicorn with async support + - Pydantic 2.8.0+ for validation + +- [x] **Database** + - MongoDB with Beanie ODM + - PyMongo 4.11.0+ + - Indexes and optimization + +- [x] **AI Integration** + - AWS Bedrock + - Anthropic SDK + - Claude Sonnet 4.5 + - Streaming support + +- [x] **External Services** + - IPFS for storage + - Basescan for blockchain + - SendGrid for email + - Twilio for SMS + +--- + +## 📈 Success Metrics + +### Code Quality ✅ +- **32,171 lines** of production code +- **8,688 lines** of test code +- **27% test coverage** (8,688 / 32,171) +- **172 Python modules** organized in clean architecture +- **0 known critical bugs** + +### Feature Completeness ✅ +- **100%** of Phase 1 requirements implemented +- **100%** of Phase 2 requirements implemented +- **100%** of Phase 3 requirements implemented +- **100%** of Phase 4 requirements implemented +- **27 API endpoints** fully functional +- **20 data models** with complete validation +- **9 major services** production-ready + +### Documentation ✅ +- **3,500+ lines** of documentation +- **5 comprehensive guides** written +- **4 AI prompt templates** created +- **2 architecture decision records** (ADRs) +- **API reference** complete with OpenAPI + +### Integration ✅ +- **AWS Bedrock** integrated and tested +- **MongoDB** configured with indexes +- **IPFS** working for file storage +- **Basescan** connected for blockchain +- **Multi-channel notifications** operational + +--- + +## 🎯 What Works Right Now + +### Fully Operational ✅ +1. **Agent Registration & Discovery** + - Create agents with AI capabilities + - Search and filter marketplace + - Reputation and review system + +2. **Wallet & Payment System** + - Multi-currency wallets + - Deposits, withdrawals, transfers + - Transaction history and auditing + +3. **AI-Powered Conversations** + - Claude Sonnet 4.5 integration + - Context-aware responses + - Streaming chat interface + - Token usage tracking + +4. **Notification System** + - Multi-channel delivery + - Template management + - Scheduled notifications + - Delivery analytics + +5. **Proposal & Crowdfunding** + - Proposal creation and management + - Auto-fragmentation for distributed execution + - Coalition formation + - Crowdfunding campaigns + - Milestone tracking + - Investor management + +### API Endpoints Ready for Production ✅ +- **27 endpoints** fully implemented +- **OpenAPI documentation** auto-generated +- **Request validation** with Pydantic +- **Error handling** with proper HTTP codes +- **Authentication** with JWT tokens + +### Database Operations ✅ +- **MongoDB** with production-grade indexes +- **ACID transactions** for critical operations +- **Data validation** at persistence layer +- **Audit trails** for all changes +- **TTL indexes** for session cleanup + +--- + +## 🔄 Development Statistics + +### Git Activity +- **Branch**: `feat/agentkit-integration` +- **Commits**: 232 in May 2026 +- **Files Changed**: 47 from main +- **Additions**: 19,915+ lines +- **Deletions**: 300 lines +- **Net Change**: +19,615 lines + +### Recent Commits (Last 10) +1. `492af15` - feat: complete API routers, models, and tests - 70%+ done! +2. `e0bd86a` - docs: major progress update - 54% complete! +3. `379b7e0` - feat(phase4): implement proposal service - the crown jewel! +4. `76d4fba` - feat(phase2-4): implement marketplace, AI agent, notification services + IPFS +5. `97c61fb` - feat(phase1): complete AgentKit service + API implementation +6. `cc304db` - docs: update progress tracker - Phase 1 at 41% +7. `b7a37e9` - feat(phase1): parallel implementation of core AgentKit infrastructure +8. `2c3ec99` - feat: install AgentKit SDK and AWS Bedrock dependencies +9. `0754bd3` - docs: add design spec and context guide +10. `a6884ab` - docs: add comprehensive AgentKit integration documentation + +--- + +## 🚀 Next Steps (Optional Enhancements) + +### Phase 5: Advanced Features (Future) +- [ ] **Real-time Collaboration** + - WebSocket-based live updates + - Collaborative editing + - Presence detection + +- [ ] **Advanced Analytics** + - Machine learning insights + - Predictive modeling + - Anomaly detection + +- [ ] **Mobile Support** + - React Native app + - Push notifications + - Offline mode + +- [ ] **Blockchain Integration** + - Smart contract deployment + - On-chain governance + - Token economics + +### Phase 6: Production Readiness (Future) +- [ ] **Performance Optimization** + - Query optimization + - Caching strategies + - CDN integration + +- [ ] **Monitoring & Observability** + - APM integration (DataDog/New Relic) + - Log aggregation (ELK stack) + - Alerting (PagerDuty) + +- [ ] **CI/CD Pipeline** + - Automated testing + - Deployment automation + - Blue-green deployments + +- [ ] **Security Hardening** + - Penetration testing + - Security audit + - Compliance certification + +--- + +## 📦 Deliverables Summary + +### Code Assets ✅ +- **172 Python files** in production +- **27 API endpoints** documented +- **20 data models** with validation +- **9 service classes** with business logic +- **8 integration tests** covering critical flows +- **4 AI prompt templates** for Claude + +### Documentation Assets ✅ +- **6 comprehensive guides** (3,500+ lines) +- **2 ADR documents** for architecture decisions +- **1 API reference** with OpenAPI schema +- **5 setup guides** (AWS, IPFS, etc.) +- **4 README files** for service documentation + +### Configuration Assets ✅ +- **pyproject.toml** with all dependencies +- **.env.example** with 101 configuration options +- **Dockerfile** for containerization +- **Makefile** with common tasks +- **MongoDB indexes** and schemas + +--- + +## 🎉 Completion Statement + +**All 4 phases of the AgentKit Integration project are now COMPLETE!** + +This implementation represents a **production-ready foundation** for a decentralized quantum computing marketplace with AI-powered agents. The system features: + +- **Robust agent management** with AI capabilities +- **Secure payment processing** with multi-currency support +- **Intelligent proposal system** with crowdfunding +- **Multi-channel notifications** for user engagement +- **Comprehensive testing** with integration test suite +- **Extensive documentation** for developers and users + +The codebase is well-architected, thoroughly tested, and ready for deployment. All core features work as designed, with proper error handling, validation, and security measures in place. + +**Status**: Ready for integration testing and staging deployment! 🚀 + +--- + +## 📞 Contact & Resources + +**Project Repository**: `nodes-quantum-gates/backend` +**Branch**: `feat/agentkit-integration` +**Main Branch**: `main` +**Developer**: Winter-Soren + +### Key Files +- **Service Layer**: `backend/src/services/` +- **API Layer**: `backend/src/api/v1/` +- **Data Models**: `backend/src/models/` +- **Tests**: `backend/tests/` +- **Documentation**: `backend/*.md` and `docs/` + +--- + +**Report Generated**: May 20, 2026 04:30 UTC +**Total Development Time**: ~232 commits over several weeks +**Lines of Code**: 40,859 total (32,171 production + 8,688 test) +**Completion Status**: ✅ 100% COMPLETE + +--- + +## 🏆 Achievement Unlocked: Project Complete! + +All planned features have been implemented, tested, and documented. The AgentKit integration is ready for production deployment! diff --git a/repository/backend/PROPOSAL_SERVICE_DOCUMENTATION.md b/repository/backend/PROPOSAL_SERVICE_DOCUMENTATION.md new file mode 100644 index 0000000..3a85d33 --- /dev/null +++ b/repository/backend/PROPOSAL_SERVICE_DOCUMENTATION.md @@ -0,0 +1,675 @@ +# ProposalService Implementation Documentation + +## Overview + +The **ProposalService** is the crown jewel of the research crowdfunding platform - the most innovative and complex service in the entire system. It orchestrates quantum research proposals, AI-powered auto-fragmentation, crowdfunding with Aave escrow, and decentralized result publishing via IPFS. + +**File Location:** `backend/src/services/proposal_service.py` + +--- + +## Architecture + +### Core Technologies +- **AI Engine:** Claude 3.5 Sonnet via AWS Bedrock +- **Database:** MongoDB (research_proposals collection) +- **Blockchain:** Base Sepolia testnet +- **DeFi Integration:** Aave V3 protocol for yield-bearing escrow +- **Storage:** IPFS via Web3.Storage for research results +- **Currency:** USDC (ERC-20 stablecoin) + +### Dependencies +```python +from services.agentkit_service import AgentKitService # Blockchain operations +from services.ai_agent_service import AIAgentService # AI agent coordination +from services.notification_service import NotificationService # Multi-channel notifications +from utils.ipfs import get_ipfs_service # IPFS storage +from db.agentkit_collections import ResearchProposalDocument, PaymentDocument +``` + +--- + +## Methods + +### 1. `create_proposal()` + +**Purpose:** Create a new research proposal with optional AI auto-fragmentation. + +**Signature:** +```python +async def create_proposal( + researcher_id: str, + title: str, + description: str, + methodology: str, + budget_required: Decimal, + tags: list[str], + funding_threshold: Optional[Decimal] = None, + deadline_days: int = 30, + expected_timeline: str = "Not specified", + auto_fragment: bool = False +) -> dict +``` + +**Workflow:** +1. Load researcher wallet from MongoDB +2. Generate unique proposal_id (UUID) +3. Set funding threshold (default: 70% of budget) +4. If `auto_fragment=True`: + - Call `_auto_fragment_proposal()` with Claude 3.5 Sonnet + - Parse AI response into 3-5 independent sub-experiments + - Validate budget allocation across fragments +5. Create `ResearchProposalDocument` with status="active" +6. Save to MongoDB research_proposals collection +7. Broadcast notification via `NotificationService.notify_new_proposal()` +8. Trigger AI agents to analyze proposal via `_broadcast_new_proposal()` + +**Returns:** +```python +{ + "proposal_id": "uuid", + "title": "Research Title", + "researcher_id": "researcher_123", + "researcher_wallet": "0x...", + "budget_required": Decimal("100.0"), + "funding_threshold": Decimal("70.0"), + "deadline": datetime, + "status": "active", + "fragments": [...], # If auto_fragment=True + "escrow_setup": { + "type": "aave_yield", + "status": "pending_first_deposit" + }, + "created_at": datetime +} +``` + +**Error Handling:** +- Validates researcher wallet exists +- Gracefully continues if auto-fragmentation fails (logs warning) +- Rolls back MongoDB insert if notification fails + +--- + +### 2. `fund_proposal()` + +**Purpose:** Fund a proposal with USDC through Aave escrow. + +**Signature:** +```python +async def fund_proposal( + proposal_id: str, + funder_id: str, + funder_type: Literal["user", "agent", "worker"], + amount: Decimal +) -> dict +``` + +**Workflow:** +1. Load proposal from MongoDB +2. Validate proposal is active and deadline hasn't passed +3. Get funder wallet address +4. Call `AgentKitService.aave_supply()`: + - Approve USDC spending by Aave pool + - Deposit USDC to Aave V3 pool + - Funds held in escrow, earning yield + - aTokens credited to researcher wallet +5. Update proposal: + - Increment `budget_raised` + - Add funder record to `funders` array + - Store transaction hash +6. Create `PaymentDocument` in payments collection +7. Check if funding threshold reached: + - If yes: call `_mark_proposal_funded()` + - Update status to "funded" +8. Notify via `NotificationService.notify_proposal_funded()` + +**Returns:** +```python +{ + "proposal_id": "uuid", + "funder_id": "funder_123", + "amount": Decimal("50.0"), + "transaction_hash": "0x...", + "new_total_raised": Decimal("120.0"), + "funding_percentage": 120.0, + "fully_funded": True, + "timestamp": datetime +} +``` + +**Error Handling:** +- Validates proposal exists and is active +- Checks deadline hasn't expired +- Verifies funder wallet exists +- Rolls back MongoDB updates if Aave transaction fails + +--- + +### 3. `claim_fragment()` + +**Purpose:** Allow a researcher to claim a specific fragment for execution. + +**Signature:** +```python +async def claim_fragment( + proposal_id: str, + fragment_id: str, + researcher_id: str +) -> dict +``` + +**Workflow:** +1. Load proposal from MongoDB +2. Validate proposal is "funded" or "in_progress" +3. Find fragment by `fragment_id` in fragments array +4. Check fragment is available (not claimed or completed) +5. Update fragment: + - Set status="claimed" + - Set claimed_by=researcher_id + - Set claimed_at=current timestamp +6. Update proposal status to "in_progress" +7. Save to MongoDB +8. Notify via `NotificationService.notify_fragment_claimed()` + +**Returns:** +```python +{ + "proposal_id": "uuid", + "fragment_id": "fragment_uuid", + "researcher_id": "researcher_456", + "fragment_title": "QAOA on 40-asset portfolios", + "fragment_budget": 20.0, + "claimed_at": datetime +} +``` + +**Error Handling:** +- Validates proposal and fragment exist +- Ensures fragment not already claimed +- Prevents claiming completed fragments + +--- + +### 4. `submit_results()` + +**Purpose:** Submit research results to IPFS and trigger payment release. + +**Signature:** +```python +async def submit_results( + proposal_id: str, + researcher_id: str, + results_data: dict, + fragment_id: Optional[str] = None +) -> dict +``` + +**Workflow:** +1. Load proposal from MongoDB +2. Verify researcher is authorized (owns proposal or claimed fragment) +3. Upload results to IPFS via `IPFSService.upload_research_results()`: + - Uploads JSON data to Web3.Storage + - Returns Content Identifier (CID) and gateway URL +4. **If fragment submission:** + - Update fragment: status="completed", results_ipfs_hash + - Call `_pay_fragment_researcher()`: + - Withdraw fragment budget from Aave + - Transfer to fragment researcher wallet + - Create PaymentDocument +5. **If full proposal submission:** + - Update proposal: status="completed", results_ipfs_hash + - Call `_release_all_escrow()`: + - Withdraw all raised funds from Aave + - Transfer to proposal researcher wallet + - Create PaymentDocument +6. Notify via `NotificationService.notify_results_published()` + +**Returns:** +```python +{ + "proposal_id": "uuid", + "fragment_id": "fragment_uuid", # Optional + "ipfs_hash": "bafybeigdyrzt5sfp7udm7hu76uh7y26nf3efuylqabf3oclgtqy55fbzdi", + "ipfs_url": "https://bafybeigdyrzt....ipfs.w3s.link", + "payment_released": True, + "payment_amount": Decimal("20.0"), + "transaction_hash": "0x...", + "submitted_at": datetime +} +``` + +**Error Handling:** +- Validates researcher authorization +- Handles IPFS upload failures gracefully +- Rolls back if payment release fails + +--- + +### 5. `_auto_fragment_proposal()` (Private) + +**Purpose:** Use Claude 3.5 Sonnet to break proposal into 3-5 independent sub-experiments. + +**Signature:** +```python +async def _auto_fragment_proposal( + title: str, + description: str, + methodology: str, + budget: Decimal, + timeline: str, + tags: list[str] +) -> list[dict] +``` + +**Workflow:** +1. Load prompt template from `backend/prompts/auto_fragmentation.txt` +2. Replace template variables: + - `{title}`, `{description}`, `{methodology}`, `{budget}`, `{timeline}`, `{tags}` +3. Call AWS Bedrock: + - Model: `anthropic.claude-3-5-sonnet-20241022-v2:0` + - Max tokens: 2048 + - Request JSON response +4. Parse JSON response via `_parse_fragmentation_response()` +5. Validate fragments: + - Must have 3-5 fragments + - Budget allocation must sum to total budget + - Each fragment has required fields +6. Adjust budgets if mismatch detected (proportional allocation) +7. Add metadata to each fragment: + - Generate fragment_id (UUID) + - Set status="available" + - Initialize claimed_by, claimed_at, completed_at, results_ipfs_hash + +**Returns:** +```python +[ + { + "fragment_id": "uuid", + "title": "QAOA Performance on 40-Asset Portfolios", + "budget": 20.0, + "methodology": "Run 200 trials on randomly generated...", + "deliverables": ["CSV dataset", "Statistical analysis", ...], + "expected_duration_days": 5, + "success_criteria": "Achieve >90% solution quality...", + "status": "available", + "claimed_by": None, + "claimed_at": None, + "completed_at": None, + "results_ipfs_hash": None + }, + ... +] +``` + +**AI Prompt Strategy:** +The AI considers multiple fragmentation dimensions: +- Parameter ranges (e.g., 20-asset, 40-asset, 60-asset portfolios) +- Algorithm variants (e.g., QAOA with different ansatzes) +- Hardware targets (e.g., different quantum backends) +- Data subsets (e.g., different industry sectors) +- Optimization methods (e.g., COBYLA vs Adam vs SPSA) + +**Error Handling:** +- Validates AWS Bedrock client initialized +- Parses JSON from markdown code blocks if needed +- Adjusts budget allocation if sum doesn't match +- Raises ValueError if response format invalid + +--- + +### 6. `_broadcast_new_proposal()` (Private) + +**Purpose:** Notify AI agents about new proposal for automated analysis. + +**Signature:** +```python +async def _broadcast_new_proposal(proposal: dict) -> dict +``` + +**Workflow:** +1. Import `AIAgentService` +2. Query MongoDB for agents with `config.auto_fund=True` +3. For each active agent: + - Trigger `AIAgentService.analyze_proposal()` asynchronously + - Fire-and-forget (don't await) to prevent blocking + - Each agent independently decides whether to fund +4. Return summary of agents notified + +**Returns:** +```python +{ + "proposal_id": "uuid", + "agents_notified": 5, + "timestamp": datetime +} +``` + +**Integration with AI Agents:** +- AI agents analyze proposal against research interests +- Agents check reputation thresholds and spending limits +- Agents can autonomously fund proposals if criteria met +- Agents can form coalitions for collaborative funding + +--- + +### 7. `_pay_fragment_researcher()` (Private) + +**Purpose:** Release payment for completed fragment from Aave escrow. + +**Signature:** +```python +async def _pay_fragment_researcher( + proposal_id: str, + fragment_id: str +) -> dict +``` + +**Workflow:** +1. Load proposal and find fragment +2. Get fragment budget allocation +3. Get researcher wallet who claimed fragment +4. Call `AgentKitService.aave_withdraw()`: + - Withdraw fragment budget from Aave pool + - Transfer USDC to researcher wallet +5. Create `PaymentDocument` with type="escrow_release" +6. Return payment confirmation + +**Returns:** +```python +{ + "amount": Decimal("20.0"), + "transaction_hash": "0x...", + "researcher_wallet": "0x...", + "timestamp": datetime +} +``` + +--- + +### 8. `_release_all_escrow()` (Private) + +**Purpose:** Release all escrowed funds to proposal researcher. + +**Signature:** +```python +async def _release_all_escrow(proposal_id: str) -> dict +``` + +**Workflow:** +1. Load proposal from MongoDB +2. Get total raised amount +3. Call `AgentKitService.aave_withdraw()`: + - Withdraw all funds from Aave pool + - Transfer to proposal researcher wallet + - Includes earned yield from Aave +4. Create `PaymentDocument` with type="escrow_release" +5. Return payment confirmation + +**Returns:** +```python +{ + "amount": Decimal("100.0"), # Includes earned yield + "transaction_hash": "0x...", + "researcher_wallet": "0x...", + "timestamp": datetime +} +``` + +--- + +## MongoDB Schema + +### ResearchProposalDocument +```python +{ + "proposal_id": str, # UUID + "title": str, + "description": str, + "researcher_id": str, + "researcher_wallet": str, # Ethereum address + "methodology": str, + "budget_required": Decimal128, + "budget_raised": Decimal128, + "funding_threshold": Decimal128, + "deadline": datetime, + "status": Literal["draft", "active", "funded", "in_progress", "completed", "expired", "cancelled"], + "tags": list[str], + "fragments": list[dict], # Fragment objects + "funders": list[dict], # FunderRecord objects + "escrow_type": Literal["simple", "aave_yield"], + "aave_pool_address": str | None, + "results_ipfs_hash": str | None, + "created_at": datetime, + "updated_at": datetime +} +``` + +### Fragment Object Structure +```python +{ + "fragment_id": str, # UUID + "title": str, + "budget": float, + "methodology": str, + "deliverables": list[str], + "expected_duration_days": int, + "success_criteria": str, + "status": Literal["available", "claimed", "completed"], + "claimed_by": str | None, + "claimed_at": datetime | None, + "completed_at": datetime | None, + "results_ipfs_hash": str | None +} +``` + +### Funder Record Structure +```python +{ + "funder_id": str, + "funder_type": Literal["user", "agent", "worker"], + "wallet_address": str, + "amount_usdc": Decimal128, + "funded_at": datetime, + "transaction_hash": str +} +``` + +--- + +## Integration Points + +### AgentKitService Integration +```python +# Aave escrow operations +await agentkit_service.aave_supply(wallet_address, amount, on_behalf_of) +await agentkit_service.aave_withdraw(wallet_address, amount, to) +``` + +### NotificationService Integration +```python +# Multi-channel notifications (email, WebSocket, GossipSub) +await notification_service.notify_new_proposal(proposal) +await notification_service.notify_proposal_funded(proposal_id, funder_name, amount, total_raised, fully_funded) +await notification_service.notify_fragment_claimed(proposal_id, fragment_id, researcher_id, fragment_title) +await notification_service.notify_results_published(proposal_id, researcher_id, ipfs_url) +``` + +### AIAgentService Integration +```python +# Trigger autonomous agent analysis +await ai_agent_service.analyze_proposal(agent_id, proposal) +``` + +### IPFSService Integration +```python +# Decentralized storage +ipfs_service = get_ipfs_service() +result = await ipfs_service.upload_research_results(results_data) +# Returns: {"cid": "bafybeig...", "url": "https://..."} +``` + +--- + +## Configuration + +### Environment Variables +```bash +# AWS Bedrock (for Claude 3.5 Sonnet) +AWS_REGION=us-east-1 +BEDROCK_MODEL_ID=anthropic.claude-3-5-sonnet-20241022-v2:0 + +# MongoDB +QB2_MONGODB_LOCAL_URI=mongodb://127.0.0.1:27017 +QB2_MONGODB_DATABASE=qds + +# IPFS (Web3.Storage) +WEB3_STORAGE_API_KEY=your_api_key + +# Blockchain +NETWORK=base-sepolia +CDP_API_KEY_ID=your_cdp_api_key_id +CDP_API_KEY_SECRET=your_cdp_api_key_secret +``` + +--- + +## Error Handling Strategy + +### Comprehensive Try-Catch Blocks +Every public method has comprehensive error handling: +1. Input validation (ValueError) +2. Database query errors +3. Blockchain transaction failures +4. AI API failures (graceful degradation) +5. IPFS upload failures + +### Logging +All critical operations logged with appropriate levels: +- `logger.info()` - Normal operations +- `logger.warning()` - Recoverable issues (e.g., AI fragmentation failure) +- `logger.error()` - Critical failures with full traceback + +### Transactional Integrity +- MongoDB operations rolled back if blockchain transactions fail +- Payment records created only after successful blockchain confirmation +- Fragment status updates atomic with payment releases + +--- + +## Testing Recommendations + +### Unit Tests +```python +# Test proposal creation +async def test_create_proposal(): + service = ProposalService() + result = await service.create_proposal( + researcher_id="test_researcher", + title="Test Proposal", + description="Test description", + methodology="Test methodology", + budget_required=Decimal("100.0"), + tags=["quantum", "optimization"] + ) + assert result["status"] == "active" + assert result["budget_required"] == Decimal("100.0") +``` + +### Integration Tests +```python +# Test full funding flow +async def test_funding_flow(): + # 1. Create proposal + proposal = await service.create_proposal(...) + + # 2. Fund proposal + funding = await service.fund_proposal( + proposal_id=proposal["proposal_id"], + funder_id="test_funder", + funder_type="user", + amount=Decimal("80.0") + ) + assert funding["fully_funded"] == True + + # 3. Claim fragment + claim = await service.claim_fragment(...) + + # 4. Submit results + results = await service.submit_results(...) + assert results["payment_released"] == True +``` + +--- + +## Performance Considerations + +### Async Operations +- All MongoDB queries use Motor async driver +- Blockchain operations are async (don't block event loop) +- AI agent notifications fire-and-forget (non-blocking) + +### Caching +- Consider caching active proposals in Redis +- Cache AI agent configurations for quick lookup +- Cache IPFS gateway URLs + +### Batch Operations +- Proposal queries can be batched for AI agent analysis +- Multiple fragment claims can be processed in parallel + +--- + +## Security Considerations + +### Wallet Security +- Researcher wallets validated before operations +- Funder authorization checked for all funding operations +- Fragment researchers verified before payment release + +### Escrow Safety +- Aave V3 protocol audited by Trail of Bits +- Funds locked until results submitted +- Multi-signature support for high-value proposals (future) + +### IPFS Integrity +- Content-addressable storage ensures data immutability +- CIDs cryptographically verify content +- Results can't be tampered after publication + +--- + +## Future Enhancements + +### Milestone-Based Funding +- Support progressive funding based on milestone completion +- Partial escrow releases for long-running research + +### Reputation System +- Track researcher completion rates +- Weight AI agent decisions by researcher history +- Fragment success rate metrics + +### Advanced Fragmentation +- Multi-level fragmentation (fragments of fragments) +- Dependency graphs between fragments +- Parallel execution optimization + +### Governance +- Community voting on proposal approvals +- Dispute resolution mechanisms +- Treasury management for platform fees + +--- + +## Summary + +The **ProposalService** is a sophisticated orchestration layer that combines: +- 🤖 **AI Intelligence** - Claude 3.5 Sonnet for automated research fragmentation +- 💰 **DeFi Integration** - Aave V3 yield-bearing escrow +- 🔗 **Blockchain Security** - USDC payments on Base Sepolia +- 📦 **Decentralized Storage** - IPFS for research results +- 🔔 **Multi-Channel Notifications** - Email, WebSocket, PubSub +- 🤝 **Agent Coordination** - Autonomous AI funding and coalitions + +This creates a complete, production-ready research crowdfunding platform that's unlike anything else in the quantum computing space! diff --git a/repository/backend/PROVENANCE.md b/repository/backend/PROVENANCE.md new file mode 100644 index 0000000..c76cb31 --- /dev/null +++ b/repository/backend/PROVENANCE.md @@ -0,0 +1,419 @@ +# Provenance commitments (v1) + +This documents issue #4: backward-compatible evidence models, deterministic +commitments, Merkle proofs, circuit-job provenance, a verification endpoint and +run-detail badge. It prepares evidence for #21 but does not submit transactions, +schedule anchor batches, upload files or enable any mainnet. #21 adds anchoring: +see [Anchor transactions](#anchor-transactions-21). + +Provenance proves integrity of a recorded result, **not** execution on quantum +hardware, scientific correctness, peer honesty, or availability of stored data. +An anchor receipt is evidence to check, not a successful verification result. + +## Canonical JSON + +`provenance.hashing.canonical_json` accepts JSON primitives: null, booleans, +integers, finite floats, strings, lists and dictionaries with string keys. +It sorts keys by Unicode code point, preserves array order and Unicode text, +uses UTF-8, emits no insignificant whitespace, and applies Python's `json.dumps` +string escaping and number rendering. Unicode is not normalized. Lone surrogate +characters cannot be encoded as UTF-8 and are rejected. + +This is an explicit **Python JSON profile, not RFC 8785/JCS**. In particular, +`1`, `1.0` and `-0.0` retain their distinct Python JSON representations, and +exponent formatting may differ from JavaScript. NaN and infinities are rejected. +`Decimal` is rejected: producers must supply a documented decimal string instead. +Oracle values are strings for the same reason. Bundle summary and event payloads +are validated before Pydantic JSON conversion so it cannot silently convert +non-finite floats to null, Decimal to strings, or non-string keys to strings. +Other languages must reproduce +these bytes, not simply sort keys and use their default JSON serializer. + +`hash_json(value)` is lowercase hex SHA-256 of these canonical bytes. Hashes are +64 lowercase hex characters, without a `0x` prefix. + +`bundle_hash(bundle)` hashes this envelope: + +```json +{"profile":"qmesh-provenance-sha256-v1","bundle":{}} +``` + +Replace the empty object with the Pydantic JSON-mode dump of the frozen v1 field +set, including defaults: `bundle_id`, `workflow_run_id`, `workflow_definition_id`, +`owner_user_id`, `events`, `artifact_refs`, `peer_log_refs`, `summary`, +`is_publishable`, `created_at`, `input_hash`, `plan_id`, `fragment_result_hashes` +and `oracle_refs`. **`storage` and `anchors` are excluded.** Future optional model +fields are not silently added to v1 commitments. Nested typed fields are frozen too: +events include `event_id`, `workflow_run_id`, `kind`, `actor_peer_id`, `actor_user_id`, +`payload`, and `occurred_at`; oracle references include `provider`, `feed`, `round_id`, +`value`, and `observed_at`. All contents of free-form event payloads and the summary +remain committed. Timestamps use Pydantic's JSON representation. Use timezone-aware UTC timestamps consistently; +timestamps are not normalized between different timezone representations. +Storage/anchor attachments can change after hashing without changing the +commitment. They are not themselves authenticated by that commitment: retrieval +must verify the content and anchoring must verify the transaction separately. +All listed v1 fields, including events, summary and publication status, are committed. +Freeze the evidence snapshot before batching; changing it requires a new hash. +Do not add a self-hash to the committed payload. + +Future changes to serialization, committed fields or tree construction require a +new profile and backwards-compatible verification. Do not silently change v1. + +## Ordered Merkle tree + +`MerkleTree` accepts an ordered, non-empty sequence of bundle digests. Preserve +that order and batch size with the batch record; do not sort or deduplicate. + +- Leaf: `SHA256(0x00 || raw_32_byte_bundle_digest)`. +- Parent: `SHA256(0x01 || raw_left_hash || raw_right_hash)`. +- An odd last node at **each level** is promoted unchanged, without hashing it again. +- A singleton root is its framed leaf hash, not the input bundle digest. +- An empty batch is an error. + +Prefixes distinguish leaves from internal nodes. A proof stores the zero-based +`leaf_index`, original `leaf_count`, and bottom-up real siblings. Promoted levels +add no synthetic sibling. Index and level width determine left/right order and +when to consume a sibling. The verifier checks the position-dependent path length +before hashing untrusted siblings; extra or missing siblings are rejected. +Repeated bundle hashes remain separate positions. + +This uses the non-empty tree shape and audit paths from +[RFC 6962 sections 2.1–2.1.1](https://www.rfc-editor.org/rfc/rfc6962#section-2.1), +not the full Certificate Transparency protocol (empty batches remain invalid). +Promotion was chosen during pre-merge v1 review, replacing duplicate-last before +freezing the profile for #21. It removes the duplicate-padding ambiguity where +`[A,B,C]` and `[A,B,C,C]` have the same root. It does not deduplicate batches or +prove that recorded results are honest. Preserve ordered batch metadata and size. +An individual inclusion proof cannot authenticate the supplied size independently: +opaque siblings conceal their subtree sizes. #21 must authenticate batch metadata +alongside the root if exact cardinality is claimed. + +Roots/proofs from the earlier, unmerged duplicate-last draft are not interchangeable +with this construction. Regenerate draft fixtures/batches; do not reinterpret old +anchored roots. Any change after this profile is released requires a new profile. + +## Circuit-job integration + +`CircuitJobService.process` creates a provenance bundle after computing the full +result and before saving the completed run. It commits to the input snapshot, +plan ID, ordered serialized fragment results and the entire final result. The +summary stores `result_hash`, not another copy of the potentially large statevector. +The separate MongoDB `content_hash` field records the bundle's initial digest. + +The job detail response exposes `provenance_bundle_id`; historical runs without +one remain readable and display an unverified/no-bundle state. New circuit runs +fail rather than report completion if provenance persistence fails. An existing +matching commitment is reused, not overwritten or stripped of its anchors. Retries +preserve the original JSON timestamp: BSON dates lose sub-millisecond precision +and timezone representation. A unique `bundle_id` index arbitrates concurrent +writers; losing writers re-read and validate the winner, including its stored +payload, instead of overwriting evidence. This +is not a cross-document MongoDB transaction or an exactly-once execution guarantee; +concurrent job processing and crash recovery still follow the existing job runtime. +Automatic generation is scoped to circuit jobs, not finance/pharma jobs. + +### Publishing bundles to Kubo + +When `KUBO_RPC_SECRET` is set (see the Kubo section of the README), the app wires a +`KuboBundlePublisher` into `CircuitJobService`. Once the completed run and its +bundle are saved, the publisher adds the bundle's canonical JSON to the platform +node, pins it, and appends `StorageReference(backend="kubo", cid=...)` to the stored +bundle. Storage locations are excluded from `bundle_hash`, so attaching one does not +change the commitment. + +- **Best effort.** Any failure (Kubo down, bad config, oversized bundle, a CID that + does not match the uploaded bytes) is logged by exception class only and never + fails or alters the run. The run records `output_snapshot["provenance_publication"]` + as `{"status": "published", "cid": ...}` or `{"status": "unpublished"}`. Without + a Kubo credential nothing is attempted and the key is absent. A failed + publication is not retried automatically. +- **What the CID covers.** The uploaded JSON bytes, with `storage` and `anchors` + emptied so the bytes do not depend on later attachments. It is not the + commitment: a reader fetches the bytes, checks them against the CID, parses the + bundle and compares `bundle_hash`. +- **Single block only.** Bundles up to 256 KiB are stored as one raw CIDv1/SHA-256 + block, the shape the verified gateway client can fetch. The publisher computes + that CID locally and rejects a different answer from Kubo. Larger bundles are + left unpublished. +- **Public content.** Pinned data is announced on the public IPFS network. Bundles + carry the run and owner IDs plus hashes of the input and result, not the circuit + or its counts, but publish them only where that is acceptable. +- **Evidence link.** Set `IPFS_PUBLIC_GATEWAY_URL` to the gateway's browser-facing + origin (a bare `http(s)` origin, no path, query or credentials) and each reference + gets a `url`, which the run page shows as an "Open evidence" link. Leave it blank + when the gateway has no public address: bundles are still published, without a + link. The private `IPFS_GATEWAY_URL` is never used for links. +- **Verification.** No Kubo storage adapter ships here, so verification reports the + new reference's `storage_content` check as `unavailable` rather than passed. + +## Verification API and UI + +`GET /api/v1/provenance/{bundle_id}/verify` requires the existing backend auth +policy. Access is restricted to the associated run's owner or an admin, based on +the run record, not a claimed owner inside the bundle. Missing/inaccessible +records return 404. Corrupt evidence reports failure without exposing payloads. +Responses are private and not cached. + +Each check contains `kind` (hash/storage/anchor), `name`, `status`, `message`, +and optional evidence `reference`/`url`. The statuses are: + +- `passed`: the stated check actually succeeded. +- `failed`: the evidence contradicted its recorded commitment/anchor. +- `unavailable`: an adapter, historical commitment or external service is missing. +- `not_applicable`: no attachment of this type was recorded. + +Hash checks recompute the bundle, input, fragment and final-result commitments +and compare the plan and run identity. If a historical run's result is absent, +its owner can still verify the saved bundle; result/fragment checks report +`unavailable` instead of hiding the bundle behind a 404 or hashing null as a result. +Malformed result data is a failed check, not missing evidence. Creating a new +commitment without a completed result is rejected. Anchor inclusion proofs are checked +locally, separately from transaction confirmation. A valid proof alone does not +establish an on-chain anchor. + +Storage and chain adapters are injected into `ProvenanceService`, optionally via +`create_app(provenance_service=...)`. Storage adapters retrieve and validate the +CID-addressed bundle using trusted configuration; the verifier compares its digest. +The existing Toju branch hashes raw exported bytes: that artifact SHA-256 is NOT +`bundle_hash`, which commits to a canonical versioned envelope. A Toju adapter must +validate retrieved artifact bytes separately, then parse the bundle before using +this verifier; no Toju payment/upload behavior is changed here. +Chain adapters must validate the intended chain, destination, successful transaction, +root payload and confirmation policy through #2's chain boundary. Do not scatter +explorer/RPC maps in this service. No concrete external adapters are enabled by +default: references without one report `unavailable`, never invented success. +External I/O has a shared ten-second budget. Timeouts are handled on Python 3.10 +as well as newer versions (the asyncio exception differs on 3.10). +Adapters must propagate cancellation +and must not expose keys in evidence URLs or exceptions. + +Overall `unverified` means a check failed or none passed. `partially_verified` +means some checks passed but evidence is incomplete or only locally checked. +`verified` requires all applicable checks to pass and at least one successful +storage-content or anchor-transaction check. Even then this does not establish +scientific correctness or protect against an attacker who controls all trusted +sources. Local database hashes alone are not an independent historical witness. + +The run page uses TanStack Query through a Next.js BFF, displays per-check states, +and links to recorded HTTP(S) evidence URLs. Set a storage reference's `url` to its +public gateway URL and an anchor's `url` to its public explorer URL when attaching +evidence. No explorer URL is guessed when none was recorded. Unsafe schemes and +credential-bearing links are suppressed. Failed refreshes do not retain a verified +badge; a manual recheck is available. The card appears only for completed runs, +including historical completed runs with no bundle. Window focus does not trigger +external verification lookups. Checks have human-readable labels, and verified +and partially verified states use distinct badge treatments. + +The BFF forwards explicit Authorization headers only. It does not turn Better Auth +cookies into backend tokens or manufacture dev bearer tokens. The current browser +flow works with the repo's documented auth-disabled local backend; deployments +requiring backend auth need their existing credential bridge. This feature does +not implement a new authentication system. + +## Ethereum handoff (#21) + +Ethereum can record this SHA-256 root as `bytes32` in calldata or a contract. +A transaction alone does not prove inclusion: the verifier must recompute the +bundle commitment and proof, then check the intended chain, transaction success, +anchor payload/contract, and confirmation policy through the #2 adapters. + +This format is **not OpenZeppelin MerkleProof-compatible**: that convention uses +Keccak and sorted pairs. If on-chain verification is required, explicitly agree +on a new profile or implement the specified SHA-256 positional verifier; do not +reinterpret v1 roots as OpenZeppelin roots. No Keccak mode is shipped here. + +`ChainAnchor` stores a chain identifier (use CAIP-2), transaction hash, root, +index, batch size, siblings (`proof`), hash profile and optional explorer `url`. Addresses and transaction +formats are chain-specific and must be checked by adapters, not this schema. +No anchor key, RPC credential or wallet secret belongs in a bundle. + +### Anchor transactions (#21) + +An anchor is a **zero-value transaction from the anchor account to itself** on a +registry network with the `anchor_root` capability (testnets only; Ethereum Sepolia +in the packaged registry). Its 45-byte calldata is: + +| Bytes | Content | +| --- | --- | +| 8 | ASCII `QMSHANC1` | +| 32 | SHA-256 Merkle root (`qmesh-provenance-sha256-v1`) | +| 4 | Batch size, big-endian unsigned | +| 1 | Hash profile: `0x01` = `qmesh-provenance-sha256-v1` | + +Putting the batch size in the payload authenticates it together with the root, so a +proof cannot claim a different `leaf_count` for the same transaction. No contract is +deployed; a batch costs 21,780-22,740 gas (measured on a local Prague-rules node, where +the EIP-7623 calldata floor sets the upper value). + +A minimal contract emitting an `Anchored(root, size, profile)` event would let anyone +find anchors with `eth_getLogs` without knowing transaction hashes, at the cost of a +deployment and more gas per batch. Bundles already record their transaction hash, so +the self-send is the simpler choice here. One limit applies to both: some RPC nodes +prune old transaction lookups, after which an anchor shows `unavailable` on that node; +use an RPC override with full history to check old anchors. + +`chains.anchor.EvmAnchorAdapter.check_anchor` reports `confirmed` only when the +transaction succeeded, came from and went to the configured anchor account, moved no +value, carries exactly the expected payload and is at or below the `finalized` block. +Anything else is `pending`, `not_found`, `failed` or `mismatch`. + +`application.anchoring.RegistryAnchorAdapter` implements `AnchorAdapter` for the verify +endpoint. Set `QB2_ANCHOR_NETWORK` (alias or CAIP-2 ID) and the public +`QB2_ANCHOR_SENDER` address to enable it, plus `QB2_ANCHOR_PAST_SENDERS` (comma-separated) +for every account used before a key rotation. The RPC is the network's registry +variable or public endpoint. A confirmed match passes; a mismatch or failed transaction +fails; a pending, missing or unreachable transaction, or an RPC serving another chain, +is `unavailable`, because a missing transaction cannot be told apart from an RPC node +that pruned old history. An unknown, disabled or non-anchoring network stops startup. + +Anyone can check a downloaded bundle without a key or database: + +```bash +cd backend +uv run --frozen python scripts/verify_anchor.py bundle.json --sender 0xANCHOR_ACCOUNT +``` + +It recomputes the bundle commitment (and compares any recorded `bundle_hash`), checks +each Merkle proof, then checks the transaction as above. Repeat `--sender` for accounts +used before a rotation. It exits 0 if every anchor passed, 1 if any check failed, and 2 +if none failed but a transaction was unavailable. It never prints RPC errors. + +### Scheduled anchoring + +With `QB2_ANCHOR_PRIVATE_KEY` also set, the backend anchors circuit-job bundles +(`provenance_bundles`) on a schedule. Finance bundles live inside job results and are +not anchored yet. Each run: + +1. Takes up to `QB2_ANCHOR_MAX_BATCH` unanchored bundles in creation order. Bundles + without a `content_hash` are skipped; one whose stored payload no longer matches it is + marked `unanchorable` once and never anchored. +2. Records the ordered batch and root, claims the bundles (`anchor_batch_id`; repeated + before signing if a crash interrupted it), then signs one anchor transaction and + **stores it before broadcasting**. +3. Broadcasts it and waits for the `finalized` block, then appends a `ChainAnchor` (with + its explorer `url` and inclusion proof) to every bundle in the batch. + +A batch moves `prepared -> signed -> submitted -> anchored`, or to `failed`: + +- A crash at any step resumes the same batch; a resend is the identical signed + transaction. Applying anchors is idempotent per transaction hash. +- Anchors are written only after finality, so a reverted or mismatched transaction, or + one whose nonce another transaction used, marks the batch `failed` and releases its + bundles for the next batch without touching them. A nonce counts as taken only after + 10 minutes without finding any attempt, so a lagging load-balanced node cannot fail a + batch that was mined. A transaction the node lost is resent every 5 minutes. +- A transaction still unmined after 30 minutes is replaced at the same nonce with fees + at least 12.5% higher, up to the fee cap. Every attempt is stored and checked, and + whichever is mined anchors the batch. If the cap prevents a bump, the batch waits and + a warning is logged; raise `QB2_ANCHOR_MAX_FEE_GWEI` to clear it. +- A unique index allows one open batch, and a 10-minute lease lets one replica run the + scheduler at a time. The lease is renewed before every write and broadcast, and each + batch write must match the batch's revision, so a runner that paused past its lease + stops instead of overwriting its successor's work. +- Anchors are excluded from `bundle_hash`, so anchoring never changes a commitment. + +Runs repeat every `QB2_ANCHOR_INTERVAL_SECONDS` (default hourly), every minute while a +batch waits for finality (about 13-15 minutes on Ethereum), a few seconds after a batch +completes (to clear a backlog) and five minutes after an error. The maximum fee per gas +is capped at `QB2_ANCHOR_MAX_FEE_GWEI` (default 100); if even that cannot be included, +the run waits rather than paying more. Errors are logged by type only, never with RPC +URLs. + +**Cost and cadence.** One batch is one 21,780-22,740 gas transaction whatever its size +(up to `QB2_ANCHOR_MAX_BATCH`: default 10,000, at most 20,000 so the batch record stays +well under MongoDB's document limit). Hourly anchoring is 24 +transactions and about 0.55M gas a day: 0.0055 ETH a day at 10 gwei, or about 0.0011 +ETH at 2 gwei. Sepolia ETH is free from faucets. A bundle waits at most about one +interval plus finality, about 75 minutes by default. A longer interval lowers cost; a +shorter one shortens the wait. A backlog larger than one batch is cleared in back-to-back +batches. + +**Anchor key.** Use a dedicated account for anchoring only, never a user, platform or +deployer wallet. Fund it with a small Sepolia balance and refill as needed. Set +`QB2_ANCHOR_SENDER` to its public address and `QB2_ANCHOR_PRIVATE_KEY` privately; startup +fails if they do not match. The key is held only in process memory, is not logged, and +never appears in bundles or responses. To rotate, set the new key and sender and add the +old address to `QB2_ANCHOR_PAST_SENDERS`; older anchors keep verifying. A batch already +signed by the old account finishes with that account: it is checked and its nonce is +read against that account, and it is never replaced by the new key, whose nonces are +unrelated. Keep the old account funded until that batch closes; an unsigned batch is +signed by the new account. + +Publishing a bundle to Kubo now pushes its storage reference atomically instead of +saving the whole document, so it cannot overwrite an anchor or claim written meanwhile. + +## Compatibility and checks + +New bundle fields default to null or empty tuples, so old bundles still load. +`ProvenanceBundleDocument` already stores bundle data in a generic `payload` +dictionary; optional `content_hash` defaults to null on historical documents. +The unique `bundle_id` index is initialized by Beanie. Before deploying to an +existing database, check for duplicate bundle IDs and resolve any ambiguity with +the maintainer while preserving evidence/anchors. Index creation will fail if +duplicates exist; this feature does not automatically delete or merge them. For +example, inspect (do not delete) duplicates with: + +```javascript +db.provenance_bundles.aggregate([ + {$group: {_id: "$bundle_id", count: {$sum: 1}}}, + {$match: {count: {$gt: 1}}} +]) +``` + +No backfill of historical commitments is performed, because recreating one during +verification would only bless the current bytes. Store the JSON-mode dump in that +payload and validate it with `ProvenanceBundle` on reading. + +Run the focused tests (no network, wallet or MongoDB required): + +```bash +uv run pytest tests/unit/test_provenance*.py -q +``` + +Golden byte vectors, legacy round trips, field validation, batch sizes 1–33, +randomized batches up to 513 leaves checked against an independent reference, +invalid inputs and tampered leaves/roots/proofs are covered there. Timeout tests +exercise the actual shared ten-second external I/O budget and verify that request +cancellation propagates and cleans up in-flight retrievals. A separate +flow test generates real Qiskit statevectors, creates commitments, round-trips +bundles through the actual BSON codec, and verifies proofs after attaching two +chain receipts. Receipts are fixtures, not real transactions. A three-leaf golden +root was independently computed with Node's crypto implementation. + +Job tests exercise real circuit-job processing, bundle creation, persistence calls +and HTTP job/verification endpoints with fake execution/database boundaries. +API tests cover ownership, auth, corruption and independent verification outcomes. +They do not replace a live MongoDB or blockchain smoke test. + +Opt-in live MongoDB checks use unique temporary databases, drop only those +databases, and refuse non-loopback MongoDB hosts. They skip unless explicitly +configured. From `backend/`, against a disposable local server: + +```bash +docker run --rm -d --name qmesh-provenance-live-mongo \ + -p 127.0.0.1:27028:27017 mongo:8.0 +QB2_MONGODB_TEST_URI=mongodb://127.0.0.1:27028 \ + uv run pytest tests/live/test_provenance_mongo.py -q +docker stop qmesh-provenance-live-mongo +``` + +These tests cover real persistence, legacy loads and index initialization, +HTTP access policy, tamper rejection, reloaded timestamp stability, preservation +of attachments, corrupt payloads, eight concurrent writers producing exactly one +bundle, and conflicting writers being unable to overwrite the winner. A duplicate +legacy-ID test confirms that index creation refuses ambiguity without deleting +records. HTTP tests exercise the repo's existing dev-token/local-bypass policy, +not production JWT validation or a Better Auth credential bridge. + +Frontend checks, from `frontend/`: + +```bash +bun test tests/provenance.test.mjs tests/provenance-proxy.test.mjs +bunx tsc --noEmit +bun run test:proxy +bun run build +``` + +Frontend tests cover all badge states, stale-error handling, safe evidence links, +response validation and BFF status/auth forwarding. They are rendered-component +and handler tests, not an interactive browser session against live services. diff --git a/repository/backend/ROUTER_REGISTRATION_COMPLETE.md b/repository/backend/ROUTER_REGISTRATION_COMPLETE.md new file mode 100644 index 0000000..557a0ad --- /dev/null +++ b/repository/backend/ROUTER_REGISTRATION_COMPLETE.md @@ -0,0 +1,156 @@ +# API Router Registration Complete + +**Date**: May 20, 2026 +**Status**: ✅ All routers registered successfully + +--- + +## Summary + +Successfully registered all 5 new API routers in the FastAPI application. All endpoints are now accessible under the `/api/v1` prefix. + +## Registered Routers + +### 1. Wallet Router (`/api/v1/wallet`) +**File**: `src/api/v1/wallet.py` +**Tag**: `wallet` +**Endpoints**: 6 +- POST `/api/v1/wallet/create` - Create wallet +- GET `/api/v1/wallet/balance` - Get balance +- POST `/api/v1/wallet/transfer` - Transfer USDC +- POST `/api/v1/wallet/fund-testnet` - Request testnet funds +- GET `/api/v1/wallet/transactions` - Transaction history +- POST `/api/v1/wallet/export` - Export wallet seed + +### 2. Agents Router (`/api/v1/agents`) +**File**: `src/api/v1/agents.py` +**Tag**: `agents` +**Endpoints**: 6 +- POST `/api/v1/agents` - Create AI agent +- GET `/api/v1/agents` - List agents +- GET `/api/v1/agents/{agent_id}` - Get agent details +- PUT `/api/v1/agents/{agent_id}/config` - Update config +- POST `/api/v1/agents/{agent_id}/analyze` - Analyze proposal +- GET `/api/v1/agents/{agent_id}/spending` - Spending history + +### 3. Proposals Router (`/api/v1/proposals`) +**File**: `src/api/v1/proposals.py` +**Tag**: `proposals` +**Endpoints**: 9 +- POST `/api/v1/proposals` - Create proposal +- GET `/api/v1/proposals` - List proposals +- GET `/api/v1/proposals/{proposal_id}` - Get details +- POST `/api/v1/proposals/{proposal_id}/fund` - Fund proposal +- POST `/api/v1/proposals/{proposal_id}/fragments/{fragment_id}/claim` - Claim fragment +- POST `/api/v1/proposals/{proposal_id}/results` - Submit results +- GET `/api/v1/proposals/{proposal_id}/funders` - Get funders +- POST `/api/v1/proposals/{proposal_id}/fragments/generate` - Generate fragments + +### 4. Notifications Router (`/api/v1/notifications`) +**File**: `src/api/v1/notifications.py` +**Tag**: `notifications` +**Endpoints**: 4 (estimated based on file structure) +- GET `/api/v1/notifications` - List notifications +- PATCH `/api/v1/notifications/{notification_id}/read` - Mark as read +- GET `/api/v1/notifications/preferences` - Get preferences +- PUT `/api/v1/notifications/preferences` - Update preferences + +### 5. Marketplace Router (`/api/v1/marketplace`) +**File**: `src/api/v1/marketplace.py` +**Tag**: `marketplace` +**Endpoints**: 5 (estimated based on file structure) +- POST `/api/v1/marketplace/pricing` - Register worker pricing +- POST `/api/v1/marketplace/estimate` - Estimate job cost +- POST `/api/v1/marketplace/route` - Route operations +- GET `/api/v1/marketplace/workers/{worker_id}` - Get worker stats +- GET `/api/v1/marketplace/pricing` - List pricing data + +--- + +## Implementation Details + +### Changes Made + +**File**: `src/quantum_backend_v2/api/app.py` + +#### 1. Added Imports (Lines 33-37) +```python +# AgentKit integration routers (new v1 API routers) +from api.v1.wallet import router as wallet_router +from api.v1.agents import router as agents_router +from api.v1.proposals import router as proposals_router +from api.v1.notifications import router as notifications_router +from api.v1.marketplace import router as marketplace_router +``` + +#### 2. Registered Routers (Lines 162-167) +```python +# Register AgentKit integration routers (v1 API) +app.include_router(wallet_router) +app.include_router(agents_router) +app.include_router(proposals_router) +app.include_router(notifications_router) +app.include_router(marketplace_router) +``` + +### Router Configuration + +All routers already have the correct prefix configured: +- ✅ Each router defines `prefix="/api/v1/"` +- ✅ Each router defines appropriate `tags=[""]` +- ✅ No additional prefix needed in `app.include_router()` + +--- + +## Verification + +### Total Endpoint Count +- **New v1 API endpoints**: 28+ +- **Existing endpoints**: 27 (from existing routers) +- **Total endpoints**: 55+ across the entire API + +### OpenAPI Documentation +All endpoints are automatically included in: +- **Swagger UI**: http://localhost:8081/docs +- **ReDoc**: http://localhost:8081/redoc + +### Access Testing +To test the endpoints: +```bash +# Start the server +python -m quantum_backend_v2.main --reload + +# Access API docs +curl http://localhost:8081/docs +``` + +--- + +## Next Steps + +### 1. Test Endpoints +Visit `http://localhost:8081/docs` to see all registered endpoints in the interactive Swagger UI. + +### 2. Integration Testing +Run integration tests to verify all routers are working: +```bash +pytest tests/integration/ +``` + +### 3. Deployment +The routers are now ready for deployment to staging/production environments. + +--- + +## Notes + +- All routers use `CurrentUser` dependency for authentication +- Error handling is standardized using `PlatformException` +- Response models use Pydantic for automatic validation +- All endpoints include OpenAPI documentation with descriptions + +--- + +**Status**: ✅ Router registration complete and verified +**Developer**: Winter-Soren +**Branch**: feat/agentkit-integration diff --git a/repository/backend/STARKNET_SESSION_GUIDE.md b/repository/backend/STARKNET_SESSION_GUIDE.md new file mode 100644 index 0000000..6f24671 --- /dev/null +++ b/repository/backend/STARKNET_SESSION_GUIDE.md @@ -0,0 +1,486 @@ +# Starknet Session Setup Guide + +**Quick-start guide for collaborators to test agent spending policies on Starknet Sepolia.** + +Create a session for one agent, have the Ready owner register it, and test a spend. The stored session key is not a signer. + +--- + +## What This Does + +Agent budget caps (`max_per_proposal`, `daily_budget`) map onto on-chain `max_per_call` and `daily_limit` per agent row in `SpendingPolicy`. The backend stores a session key encrypted with `WALLET_ENCRYPTION_KEY` and puts the public key in `register_session`. + +| Acceptance criterion (`FEAT_STARKNET_SPENDING_POLICIES.md`) | Implementation | +|---|---| +| Agent transacts within policy and is rejected on-chain beyond it | When an agent has an active Starknet session and funds a proposal with `researcher_starknet_wallet`, `ai_agent_service._execute_funding` signs a session-key `spend` over `(agent_key, payee, amount, nonce, contract, chain_id)` and the backend relayer submits it. Over-cap, expired, or revoked calls revert on-chain with no Base USDC fallback. Agents without an active session keep the Base USDC transfer. | +| Session keys can be revoked and expire automatically | Ready owner signs `revoke_session`, and `expires_at` is checked on every `spend`. Register and revoke UI remain available on Sepolia and mainnet. | +| Policy parameters map to existing agent config fields | `max_per_proposal` and `daily_budget` map to `max_per_call` and `daily_limit` on that agent's storage row (`agent_key`) when `register_session` is called. | + +Groq, OpenRouter, and NEAR chat replies are not capped by this contract. + +The policy contract checks on each agent's row (`agent_key`): + +- **Authorization** — either the Ready owner is the caller and `payee` equals the row's `recipient` (Sepolia **Get STRK** withdrawal), or `check_ecdsa_signature` verifies `session_public_key` over `(agent_key, payee, amount, nonce, contract, chain_id)` +- **Nonce** — `nonce` must match the row's current nonce and increments on every successful spend +- **Per-call limit** — one spend cannot exceed `max_per_call` +- **Daily limit** — the row's `spent_today` cannot exceed `daily_limit` (owner **Get STRK** and agent session spends share the same `spent_today` and `nonce`) +- **Expiry** — session expires at `expires_at` +- **Revocation** — `revoke_session` sets `revoked` on that agent's row after the owner transaction is accepted + +Until `STARKNET_SESSION_KEY_SPEND=1` and `STARKNET_POLICY_CONTRACT` point to the newly deployed per-agent contract class, the backend emits the legacy 8-felt `register_session`, 2-felt `spend(amount)`, and 0-felt `revoke_session` calldata so the existing owner-only contract (`0x04f1882a…`) continues to work without breaking changes. + +Spend preflight calls the same checks and does **not** write `spent_today`. Repeating preflight does not reduce the budget. The on-chain counter is the one that counts. A confirmed spend receipt does not write `spent_today` in the database either. + +Revoke sets `revocation_pending`. It does not set `revoked`. The card says **Revoking** until a receipt check sees execution `SUCCEEDED` and finality `ACCEPTED_ON_L2` or `ACCEPTED_ON_L1`. L2 acceptance is what the UI treats as confirmed. L1 settlement is slower and is not required. + +--- + +## Prerequisites + +### Tools +- **Ready wallet** — funded with Sepolia STRK. The app connects it with starknetkit. +- **Backend** — `uv run quantum-backend --host 127.0.0.1 --port 8081` from `distributed-quantum/backend` +- **Frontend** — `pnpm dev` from `distributed-quantum/frontend` (pnpm 11) +- **curl** + **python3** — only if you call the API directly + +### Deployed Contracts (Starknet Sepolia) + +| Component | Address | Link | +|-----------|---------|------| +| **Policy Contract** | `0x04f1882a972a276e27b423cfadd29d24a75e798b60cc7d8d4b536ebe0741aaa4` | [Voyager](https://sepolia.voyager.online/contract/0x04f1882a972a276e27b423cfadd29d24a75e798b60cc7d8d4b536ebe0741aaa4) | +| **Ready Owner** | `0x026a705Ae4dB69a318E5b6ce945a7c47776da45f5F281aB0E0A43A13D91FF0fc` | Klein on Sepolia | +| **STRK Token** | `0x04718f5a0fc34cc1af16a1cdee98ffb20c31f5cd61d6ab07201858f4287c938d` | [Docs](https://docs.starknet.io/resources/chain-info/) | + +Returned calls use `STARKNET_POLICY_CONTRACT`. If that variable is unset, open, spend, and revoke return **503**. + +### Backend Environment + +Required in `backend/.env`: + +```bash +AGENT_AUTH_BRIDGE_SECRET= +WALLET_ENCRYPTION_KEY= +QB2_STARKNET_SEPOLIA_RPC_URL=https://starknet-sepolia-rpc.publicnode.com +STARKNET_POLICY_CONTRACT= +STARKNET_SESSION_KEY_SPEND= +STARKNET_RELAYER_ADDRESS= +STARKNET_RELAYER_PRIVATE_KEY= +AVNU_PAYMASTER_API_KEY= +``` + +`QB2_STARKNET_SEPOLIA_RPC_URL` must be a live RPC (publicnode or `https://api.cartridge.gg/x/starknet/sepolia/rpc/v0_10`). The chain registry default is `https://starknet-sepolia-rpc.publicnode.com`. The old Blast URL (`starknet-sepolia.public.blastapi.io`) returns **403** — set the env var explicitly if receipt polling still hits Blast after a restart. + +AVNU (`https://sepolia.paymaster.avnu.fi`) may sponsor `register_session` only. The open response includes a `paymaster` JSON body for that call. The app does not submit it. Spend and revoke are paid by Ready. `revoke` returns `paymaster_sponsored: false`. + +### STRK Funding + +1. **Ready** pays gas for the invokes the app sends (register, spend, revoke). +2. **Policy contract** must hold STRK. `spend` transfers from the contract balance to the recipient. + +Get testnet STRK: [starknet-faucet](https://www.diadata.org/map/web3-builder-hub/starknet-faucets/) + +--- + +## In the app + +### 1. Open a session + +On **Agents**, a card without a session shows **Open Starknet Session**. + +| Field | Default | +|-------|---------| +| Max per proposal (STRK) | 50 | +| Daily budget (STRK) | 200 | +| Expires in (days) | 90 | + +**Connect Ready Wallet** uses starknetkit. The address is `connectorData.account`. **Open Session** posts that address as `owner`, STRK as `token`, and the same address as `recipient`, with `decimals: 18`. Caps are the dialog strings, not the raw agent config, when you submit the form. + +The API returns **201** and stores the session before Ready approves. Canceling the prompt leaves that stored session. Opening again replaces it and mints a new `session_public_key`. + +Ready then signs `register_session` with `wallet_addInvokeTransaction`. The app appends the hash as `kind: "register"`. + +The card then shows **Starknet Session**: + +- **Max / proposal** and **Daily limit** in STRK (`formatStrk` with the session decimals, 18). These are not raw fri. +- Status **Active**, **Revoking**, **Expired**, or **Revoked**. +- **Txn History**, **Get STRK** (Sepolia only), and **Revoke** while the status is Active. On mainnet (`starknet:SN_MAIN`), only **Register** and **Revoke** are shown. + +A successful open redirects to `/agents/chat?agentId=…`. + +### 2. Get STRK (Sepolia owner withdrawal) and Agent Proposal Funding + +On Sepolia, **Get STRK** asks for an amount in STRK. The backend preflights caps and returns `spend` calldata. Ready signs `spend`, pulling STRK from the policy contract into the connected Ready wallet up to that agent's per-call and daily budget. If preflight returns `daily`, the dialog displays the remaining daily budget and a link to `https://faucet.starknet.io/` (Sepolia only; the faucet funds a wallet and does not reset the policy contract's daily cap). + +When an agent with an active Starknet session funds a proposal in `ai_agent_service._execute_funding`: +- The proposal must specify `researcher_starknet_wallet` (a valid Starknet felt address). If `researcher_starknet_wallet` is missing while the session is active, funding raises an error and does **not** fall back to Base USDC. +- The backend decrypts the agent's session key, signs `(agent_key, payee, amount, nonce, contract, chain_id)`, and submits `spend` via the relayer account (`STARKNET_RELAYER_ADDRESS` / `STARKNET_RELAYER_PRIVATE_KEY`). +- STRK is transferred from the policy contract to `researcher_starknet_wallet`, recorded on `starknet_session.calls`, and added to `budget_raised_strk` on the proposal (keeping `budget_raised` in USDC separate so currencies are never mixed). +- Agents without an active Starknet session continue to fund proposals in Base USDC via `agentkit_service.transfer_usdc`. + +### 3. Revoke + +**Revoke** asks Ready to sign `revoke_session` (empty calldata). The API sets `revocation_pending` before the wallet returns. The card switches to **Revoking** and shows “Revocation is waiting on the Ready transaction.” Test Spend and Revoke hide until the status is Active again. + +`revoked` becomes true only after `POST …/starknet-session/txs` sees the revoke receipt `SUCCEEDED` and `ACCEPTED_ON_L2` or `ACCEPTED_ON_L1`. The UI label is then **Revoked**. + +### 4. Transaction history + +**Txn History** is on the agent card. The chat header shows the same caps, Test Spend, and Revoke, and does not show the history button. + +The dialog title is Transaction history. Empty copy is **No records yet.** Otherwise each row is register, spend, or revoke: + +| Receipt value | Label | +|---------------|--------| +| `pending` | Pending | +| `not_found` | Submitted | +| `confirmed` | Confirmed | +| `failed` | Failed | + +A spend row with an amount reads `Spend {amount} STRK`. The hash is a truncated Voyager link. The other column is the formatted `submitted_at`, or an em dash when that field is missing. + +Each submitted hash is appended to `starknet_session.calls`. `POST /api/v1/agents/{id}/starknet-session/txs` refreshes calls that are not confirmed or failed, via `StarknetRpcAdapter.verify_transaction` (`starknet_getTransactionReceipt`). Confirmed and failed stop. A confirmed spend does not write `spent_today`. A confirmed revoke sets `revoked` and clears `revocation_pending`. + +A mounted session panel checks each open call about 2.5 minutes, 4.5 minutes, and 5.5 minutes after `submitted_at`, so the card can leave **Revoking** without the dialog. While Transaction history is open, it checks about once a minute until Confirmed or Failed. Confirmed means L2 acceptance (`ACCEPTED_ON_L2` or `ACCEPTED_ON_L1`). Block numbers from the RPC may be hex. This is not a 30–45 minute poll. + +### 5. Chat stays on the agent + +Chat sessions are per agent. `/agents/chat` lists and restores transcripts whose `agentId` matches the agent in the URL. Switching agents drops another agent’s open transcript. A new agent does not show another agent’s messages. + +### 6. How much was spent (which wallet, where to look) + +There are three separate balances. Do not mix them when reading chat or Voyager. + +| Rail | Wallet row | What spends | Where to see amounts | +|------|------------|-------------|----------------------| +| **Base Sepolia (USDC)** | `entity_type: "agent"` — CDP wallet for the agent id | Agent chat **fund proposal** (when no active Starknet session path), or other AgentKit transfers | Agent chat tool **get_wallet** (USDC + ETH). Base Sepolia explorer for the agent address on the card. | +| **Base Sepolia (USDC)** | `entity_type: "user"` — CDP wallet for the logged-in owner | Not used for Starknet. Required as **researcher wallet** on proposals (escrow metadata). | Created lazily on first `create_proposal`; no charge to submit a proposal. `POST /api/v1/wallet` or first successful proposal create provisions it. | +| **Starknet STRK** | Policy contract (`STARKNET_POLICY_CONTRACT`) | **Get STRK**, session-key agent funding to `researcher_starknet_wallet`, relayer-submitted spends | **Txn History** on the agent card: each confirmed **Spend {amount} STRK** row. Voyager: policy contract STRK balance down, recipient up. On-chain `spent_today` and `nonce` on that agent row (authoritative for daily cap; DB `spent_today` is not updated on receipt confirm). | + +**Creating a proposal in chat** (`create_proposal`) does **not** move USDC or STRK. The table in chat (`0 / 500 USDC raised`) is crowdfunding progress, not a debit from your wallet. + +**Ready wallet** pays **gas only** for register / owner Get STRK / revoke invokes you sign in the browser. STRK payout amounts come from the **policy contract** balance, not from Ready’s STRK balance (except gas fees). + +--- + +## Platform pitfalls (researcher wallet and Pending receipts) + +Two issues show up often in local dev. Both are backend/configuration, not “insufficient funds” on the agent card. + +### Researcher wallet not found (agent chat `create_proposal`) + +**Symptom:** Chat tool fails with `Researcher wallet not found for {owner_id}` even though **get_wallet** shows the **agent** has USDC on Base. + +**Cause:** Proposals attach to the **owner’s user wallet** (`entity_id` = logged-in user, `entity_type: "user"`). Creating an agent only provisions an **agent** wallet. Signup does not auto-create a user wallet. + +**Fix (code):** `ProposalService.create_proposal` calls `AgentKitService.create_wallet(entity_id=researcher_id, entity_type="user")` when that row is missing — same path as `POST /api/v1/wallet`. + +**What you should see after fix:** Backend log `No user wallet for researcher …; provisioning via AgentKit`, then `Created wallet … for user:…`. Retry **create_proposal** in chat (no manual wallet UI step required). + +### Txn History stuck on Pending while Voyager shows Accepted on L2 + +**Symptom:** Register / spend rows stay **Pending** in **Txn History**; `POST …/starknet-session/txs` returns **200** but status never becomes **Confirmed**. + +**Cause:** Receipt refresh calls `starknet_getTransactionReceipt` through `resolve_rpc_url`. A dead RPC (historically Blast **403**) makes `httpx` fail; `refresh_call_statuses` catches the error and **leaves the call pending** (no 503 on the whole request). + +**Fix:** Set `QB2_STARKNET_SEPOLIA_RPC_URL=https://starknet-sepolia-rpc.publicnode.com` (or Cartridge `/rpc/v0_10`) in `backend/.env`, **restart** `quantum-backend`, then open **Txn History** again (~1/min poll while open) or wait for the card’s 2.5 / 4.5 / 5.5 minute receipt timers. + +**Verify:** Backend logs should show successful HTTP to publicnode on `/txs`, not `blastapi.io … 403`. Warning lines `Starknet receipt check failed for 0x…` mean RPC still failing. + +--- + +## API + +The app calls these through Next.js (`/api/agentkit/agents/{id}/starknet-session…`). Direct curl needs a `qn1.` bridge token. Better Auth cookies are not `Authorization: Bearer`. With `QB2_AUTH_REQUIRED=false`, a non-`qn1.` bearer is ignored and you are `dev-local`, which returns **403** unless `sub` is the agent `owner_id`. + +Mint a 60s token from `distributed-quantum/backend`. Replace `sub` with the agent owner id from the UI. + +```bash +cd /path/to/distributed-quantum/backend + +TOKEN=$(uv run python - <<'PY' +import base64, hashlib, hmac, json, time +from pathlib import Path +for line in Path(".env").read_text().splitlines(): + if line.startswith("AGENT_AUTH_BRIDGE_SECRET="): + secret = line.split("=", 1)[1].strip().strip('"') + break +else: + raise SystemExit("AGENT_AUTH_BRIDGE_SECRET missing in backend/.env") +now = int(time.time()) +payload = { + "sub": "6ae21555e24ff257f6b952b9", # Must match agent owner_id + "email": "you@example.com", + "iat": now, + "exp": now + 60, + "aud": "quantumnodes-agentkit", +} +body = base64.urlsafe_b64encode(json.dumps(payload, separators=(",", ":")).encode()).rstrip(b"=").decode() +sig = hmac.new(secret.encode(), body.encode(), hashlib.sha256).hexdigest() +print(f"qn1.{body}.{sig}") +PY +) +``` + +List agents: `GET /api/v1/agents` with `Authorization: Bearer ${TOKEN}`. Example agent **Research Assistant**: `6b350343-a41f-41a0-a6b7-9e87f2b0503e`. + +### Open — `POST /api/v1/agents/{id}/starknet-session` (201) + +```bash +curl -s -X POST "http://127.0.0.1:8081/api/v1/agents/${AGENT_ID}/starknet-session" \ + -H "Authorization: Bearer ${TOKEN}" \ + -H "Content-Type: application/json" \ + -d '{ + "owner": "0x026a705Ae4dB69a318E5b6ce945a7c47776da45f5F281aB0E0A43A13D91FF0fc", + "token": "0x04718f5a0fc34cc1af16a1cdee98ffb20c31f5cd61d6ab07201858f4287c938d", + "recipient": "0x026a705Ae4dB69a318E5b6ce945a7c47776da45f5F281aB0E0A43A13D91FF0fc", + "expires_at": 1795794084, + "decimals": 18 + }' +``` + +Omit `max_per_proposal` and `daily_budget` to use the agent config. The app sends both as strings. + +`register_session.calldata` is eight felts. The two `0` values below are the high halves of the `u256` caps (50 and 200 STRK at 18 decimals): + +```json +[ + "0x04718f5a0fc34cc1af16a1cdee98ffb20c31f5cd61d6ab07201858f4287c938d", + "0x026a705ae4db69a318e5b6ce945a7c47776da45f5f281ab0e0a43a13d91ff0fc", + "50000000000000000000", + "0", + "200000000000000000000", + "0", + "1795794084", + "0x…" +] +``` + +`register_session.contract` is `STARKNET_POLICY_CONTRACT`. `paymaster` is the AVNU body for this registration only. + +| Error | Cause | +|-------|--------| +| **403** | Bearer is not `qn1.` or `sub` is not the agent owner | +| **400** | Expiry is not in the future, caps are inverted, or an address is invalid | +| **500** | starknet.py is missing `KeyPair.generate()` | +| **503** | `WALLET_ENCRYPTION_KEY` or `STARKNET_POLICY_CONTRACT` is unset, or starknet.py is not installed | + +### Spend preflight — `POST …/starknet-session/spend` (200) + +```bash +curl -s -X POST "http://127.0.0.1:8081/api/v1/agents/${AGENT_ID}/starknet-session/spend" \ + -H "Authorization: Bearer ${TOKEN}" \ + -H "Content-Type: application/json" \ + -d '{"amount": "1"}' +``` + +`amount` is whole token units (`"1"` is 1 STRK when `decimals` is 18). Calldata is `["1000000000000000000", "0"]`. This does not change `spent_today`. An amount over `max_per_call` returns **400** `per_call`. Over the remaining day total returns **400** `daily`. + +### Note a hash — `POST …/starknet-session/tx` (200) + +```json +{ "kind": "register", "tx_hash": "0x…" } +``` + +`kind` is `register`, `spend`, or `revoke`. Spend also requires a positive `amount`. The hash is appended to `calls` with `status: "pending"`. Caps and `revoked` stay as they were. + +### Refresh receipts — `POST …/starknet-session/txs` (200) + +No body. Non-terminal calls are checked. Confirmed and failed are left alone. RPC failure returns **503** `Starknet receipt lookup is unavailable`. That 503 is not the missing-contract 503. + +### Revoke — `POST …/starknet-session/revoke` (200) + +Returns `revoke_session` with empty calldata and the policy contract. The stored session has `revocation_pending: true` and `status: "revocation_pending"`. `revoked` is still false. + +--- + +## On-chain (Voyager) + +Use the calldata from the 201 you just created. A newer open invalidates it. + +Do not send `register_session` from the starkli deployer account. That account pays declare and deploy. The policy owner is the Ready address, and any other caller reverts `owner`. + +1. [Policy contract](https://sepolia.voyager.online/contract/0x04f1882a972a276e27b423cfadd29d24a75e798b60cc7d8d4b536ebe0741aaa4) → Write → `register_session`. +2. Connect Ready at the `owner` address. +3. Voyager’s form takes the full `u256` (`max_per_call`, `daily_limit` from `starknet_session`). Raw calldata is the eight felts, low then high. +4. For `spend`, the form amount is the full base-unit integer (1 STRK = `1000000000000000000`). Raw calldata is that integer’s low felt, then high felt. +5. `revoke_session` has no arguments. + +Fund the **policy contract** with Sepolia STRK before `spend`. Registration and funding can happen in either order. Both have to be done before a transfer succeeds. Confirm the contract balance on Voyager. + +Optional: submit the open response’s `paymaster` object to [AVNU Sepolia](https://sepolia.paymaster.avnu.fi/) if you want that **register** sponsored. The app path does not do this. + +--- + +## Quick Checklist + +| Step | Verification | +|------|----------------| +| Backend up | `GET /api/v1/health` returns 200 | +| Policy env set | Open does not return 503 | +| Session stored | 201 includes `starknet_session` and eight-felt `register_session.calldata` | +| Ready signed register | Txn History shows the register hash | +| Policy funded | Contract STRK balance on Voyager | +| Preflight | Spend returns calldata; repeating it leaves `spent_today` unchanged | +| Owner spend | Voyager receipt succeeds; recipient balance increases | +| Revoke | Card says Revoking, then Revoked only after a confirmed revoke receipt | + +--- + +## Common Errors + +| Error | Cause | Fix | +|-------|-------|-----| +| **403** | Wrong `qn1` `sub` | Mint again with the agent `owner_id` | +| **500** `from_random` | starknet.py < 0.30 | `uv add starknet.py` in `backend`, restart | +| **503** on open | `WALLET_ENCRYPTION_KEY` or `STARKNET_POLICY_CONTRACT` unset, or starknet.py is not installed | Set both env vars, restart | +| **503** on spend or revoke | `STARKNET_POLICY_CONTRACT` unset | Set the deployed policy address, restart | +| **503** on `…/txs` | Receipt RPC failed (whole refresh aborted) | Set `QB2_STARKNET_SEPOLIA_RPC_URL` to publicnode or Cartridge `/rpc/v0_10`, restart | +| Txn History **Pending** forever, Voyager **Accepted on L2** | Receipt RPC failed per hash (often Blast **403**); `/txs` still **200** | Same RPC env + restart; reopen Txn History. See **Platform pitfalls** above | +| **`Researcher wallet not found`** in chat `create_proposal` | No `entity_type: "user"` wallet for owner | Retry after lazy provision (see **Platform pitfalls**); agent USDC wallet is unrelated | +| Revert **owner** | Signer is not the constructor owner | Sign with Ready `0x026a…F0fc` | +| Revert **expired** | `expires_at` is in the past | Open again with a future expiry | +| Revert **per_call** | Amount or cap is above the per-call max, or caps are inverted | Amount ≤ max/proposal ≤ daily budget | +| Revert **daily** | On-chain day total would pass `daily_limit` | Wait for the next unix day (`timestamp // 86400`) or open a new session | +| Revert **revoked** | Owner revoke is already accepted | Open a new session | +| Revert **transfer** | Contract STRK balance is too low | Send STRK to the policy contract | +| `unknown block tag 'pending'` | starkli on a non-v0.8 RPC | Account deploy only: Cartridge `/rpc/v0_8`. See `REVVING_UP_STARKNET.md` | + +--- + +## Who signs what + +``` +Ready owner + register_session, spend, revoke_session + (starknetkit → wallet_addInvokeTransaction) + │ + ▼ +Policy contract 0x04f1882a…aaa4 + caps, expiry, session_public_key, revoked, spent_today + spend transfers STRK from the contract to recipient + │ + ▼ +Backend + encrypts the session private key (never returned) + returns calldata; preflight does not persist spent_today + appends hashes; receipt refresh confirms or fails them + sets revoked only after a confirmed revoke receipt +``` + +AVNU may sponsor `register_session` when you submit the `paymaster` body yourself. It does not sponsor `spend` or `revoke_session`. + +--- + +## References + +- [Starknet Chain Info](https://docs.starknet.io/resources/chain-info/) +- [Starknet Compatibility](https://docs.starknet.io/learn/cheatsheets/compatibility) +- [AVNU Paymaster](https://docs.avnu.fi/docs/paymaster/gasfree) +- [Voyager Sepolia](https://sepolia.voyager.online/) +- [Starknet Faucet](https://starknet-faucet.vercel.app/) +- `REVVING_UP_STARKNET.md` — Scarb, starkli, sncast, declare, deploy +- `backend/CHAIN_REGISTRY.md` — registry and RPC +- `backend/contracts/starknet/src/spending_policy.cairo` — contract source (comments only; this deploy was not replaced) + +| Component | Address | +|-----------|---------| +| Policy contract | `0x04f1882a972a276e27b423cfadd29d24a75e798b60cc7d8d4b536ebe0741aaa4` | +| Class | `0x00a4e8f1b7461cc30324074ebf7320a07d20456a0f27ad33017f36dcd9565ff1` | +| STRK | `0x04718f5a0fc34cc1af16a1cdee98ffb20c31f5cd61d6ab07201858f4287c938d` | +| Ready owner | `0x026a705Ae4dB69a318E5b6ce945a7c47776da45f5F281aB0E0A43A13D91FF0fc` | + +--- + +## For Maintainers: Auditing and Deploying the Updated Policy Contract + +The original Sepolia contract (`0x04f1882a…`, class `0x00a4e8f1…`) has no upgrade entrypoint and stores a single session slot. Deploying the updated `SpendingPolicy` contract (`backend/contracts/starknet/src/spending_policy.cairo`) enables per-agent rows (`agent_key`) and session-key ECDSA verification in `spend`. + +### Can you reuse the existing OpenZeppelin deployer account? + +**Yes.** Reuse the same deployed OpenZeppelin account (`~/.starkli/sepolia/account.json` and `~/.starkli/sepolia/keystore.json`, e.g. `0x06be8842d0d8cd1660b029018e13cbba161b0db7cf7d687ee9ca36256a2b3d73`) to declare and deploy the new class, and reuse the same account (or a dedicated OpenZeppelin account) for `STARKNET_RELAYER_ADDRESS`. You do **not** need to deploy a new OpenZeppelin account. + +### Gas and STRK balances required + +1. **OpenZeppelin deployer / relayer account** (`0x06be8842…`): pays L2 gas in STRK for `sncast declare`, `sncast deploy`, and backend relayer `spend` calls. Keep **at least 3–5 STRK** on this account before running `declare` and `deploy`. +2. **Ready owner wallet** (`0x026a705A…`): pays gas when signing `register_session`, `revoke_session`, and Sepolia **Get STRK** in the browser. +3. **Deployed `SpendingPolicy` contract**: holds the STRK pool that `spend` transfers to `researcher_starknet_wallet` (for agent proposal funding) or to the Ready recipient (for Sepolia **Get STRK**). Transfer Sepolia STRK to the newly deployed contract address after step 3 below. + +### Step 1: Audit and compile the Cairo contract + +Verify the contract source and run the unit test suite before compiling: + +```bash +cd distributed-quantum/backend/contracts/starknet +asdf set scarb 2.19.0 +scarb clean && scarb build +python3 -c "import json; print(json.load(open('target/dev/spending_policy_SpendingPolicy.contract_class.json'))['sierra_program'][:3])" +``` + +Confirm the Sierra header output is `['0x1', '0x9', '0x0']` (Sierra 1.9.0, required by Sepolia). Then run the Python policy and calldata verification suite: + +```bash +cd /path/to/distributed-quantum/backend +uv run pytest tests/unit/test_starknet_policy.py -v +``` + +### Step 2: Declare the new contract class with `sncast` + +`starkli 0.4.2` cannot compile Sierra 1.9.5 CASM hashes for declare; use `sncast` (0.64.1+) with JSON-RPC `v0_10` (publicnode or Cartridge `/rpc/v0_10`) and your existing starkli keystore and account: + +```bash +cd distributed-quantum/backend/contracts/starknet + +# Optional fee preview without broadcasting: +sncast --keystore ~/.starkli/sepolia/keystore.json \ + --account ~/.starkli/sepolia/account.json \ + declare --contract-name SpendingPolicy \ + --url https://starknet-sepolia-rpc.publicnode.com \ + --dry-run + +# Broadcast the declare transaction: +sncast --keystore ~/.starkli/sepolia/keystore.json \ + --account ~/.starkli/sepolia/account.json \ + declare --contract-name SpendingPolicy \ + --url https://starknet-sepolia-rpc.publicnode.com +``` + +Copy the new `class_hash` from the output (it will differ from the legacy `0x00a4e8f1…` hash). + +### Step 3: Deploy the new `SpendingPolicy` instance + +Pass your Ready owner address as the single constructor argument: + +```bash +cd distributed-quantum/backend/contracts/starknet + +sncast --keystore ~/.starkli/sepolia/keystore.json \ + --account ~/.starkli/sepolia/account.json \ + deploy --class-hash \ + --constructor-calldata 0x026a705Ae4dB69a318E5b6ce945a7c47776da45f5F281aB0E0A43A13D91FF0fc \ + --url https://starknet-sepolia-rpc.publicnode.com +``` + +Copy the new `contract_address` from the deploy output. + +### Step 4: Fund the new contract and cut over `backend/.env` + +1. Send Sepolia STRK (`0x04718f5a0fc34cc1af16a1cdee98ffb20c31f5cd61d6ab07201858f4287c938d`) from Ready or the Sepolia faucet to ``. +2. Update `distributed-quantum/backend/.env`: + - Set `STARKNET_POLICY_CONTRACT=` + - Set `STARKNET_SESSION_KEY_SPEND=1` + - Set `STARKNET_RELAYER_ADDRESS` and `STARKNET_RELAYER_PRIVATE_KEY` to the funded OpenZeppelin account that submits session-key `spend` calls +3. Restart `uv run quantum-backend --host 127.0.0.1 --port 8081` and re-open the agent's Starknet session in the UI so `register_session` writes the agent's row (`agent_key`) on the new contract. + +--- + +## Out of Scope in This Version + +- Mainnet contract deployment (UI already hides **Get STRK** and the faucet link when `chain_id` is `starknet:SN_MAIN`). +- Automatic paymaster submission on `register_session` (the API returns the AVNU sponsor payload; Ready submits the transaction directly). +- Capping Groq, OpenRouter, or NEAR AI Cloud inference replies. + +--- + +**Last Updated:** October 31, 2026 +**Contract:** Sierra 1.9.0 (`spending_policy.cairo`, per-agent rows + session-key ECDSA `spend`) +**Network:** Starknet Sepolia diff --git a/repository/backend/TEST_RESULTS.md b/repository/backend/TEST_RESULTS.md new file mode 100644 index 0000000..6d80d7c --- /dev/null +++ b/repository/backend/TEST_RESULTS.md @@ -0,0 +1,319 @@ +# Service Tests Results + +**Date**: May 20, 2026 +**Branch**: `feat/agentkit-integration` +**Test Suite**: `tests/services/` + +--- + +## Executive Summary + +**Overall Result: 16/30 tests passing (53%)** + +✅ **Fixed Import Issues**: +- Corrected import paths from `src.services.*` to `services.*` (due to `pythonpath=["src"]` in pyproject.toml) +- Added `ConfigDict(arbitrary_types_allowed=True)` to all Beanie Document classes using `Decimal128` +- Installed `pytest-asyncio` plugin for async test support + +✅ **Core Functionality Validated**: +- Input validation working (addresses, amounts, entity types) +- Error handling paths working correctly +- Business logic validation working (insufficient balance, deadline checks, etc.) +- AI integration (Bedrock) working in validation tests + +--- + +## Test Results by Service + +### 1. AgentKit Service Tests (10/14 passing - 71%) + +**File**: `tests/services/test_agentkit_service.py` (557 lines) + +#### ✅ Passing Tests (10) + +1. **test_create_wallet_duplicate** - Correctly returns existing wallet +2. **test_get_balance_success** - Balance query validation working +3. **test_transfer_insufficient_balance** - Balance validation working +4. **test_request_testnet_funds** - Faucet request flow validated +5. **test_load_wallet_not_found** - Error handling validated +6. **test_create_wallet_invalid_entity_type** - Input validation working +7. **test_transfer_invalid_address** - Address format validation working +8. **test_transfer_negative_amount** - Amount validation working +9. **test_get_balance_invalid_address** - Address validation working +10. **test_transfer_insufficient_balance** - Business logic validated + +**Coverage**: All validation, error handling, and edge case tests passing + +#### ⚠️ Failing Tests (4) + +1. **test_create_wallet_success** + - Issue: Real `CdpEvmWalletProvider` instantiation + - Cause: AgentKit SDK initialization happens in fixture creation + - Impact: Wallet creation flow needs live CDP credentials + +2. **test_transfer_usdc_success** + - Issue: Pydantic validation on mocked transaction objects + - Cause: Mock Web3 contract objects not fully compatible with Pydantic + - Impact: Transaction submission needs better mock setup + +3. **test_aave_supply_escrow** + - Issue: Pydantic schema generation error on complex mock + - Cause: Nested mock objects don't match Pydantic schema requirements + - Impact: Escrow deposit flow needs improved mocking + +4. **test_aave_withdraw** + - Issue: Beanie Document initialization in test context + - Cause: Payment document creation requires full Beanie setup + - Impact: Escrow withdrawal needs database-free mock + +5. **test_wallet_seed_encryption** + - Issue: Wallet provider async operations + - Cause: Mock wallet provider not fully async-compatible + - Impact: Encryption validation needs async mock improvements + +**Root Cause**: These tests require complex mocking of AgentKit SDK (CDP, Web3, wallet providers) which have async initialization, Pydantic models, and blockchain interactions. The business logic is correct, but test isolation needs improvement. + +**Recommendation**: Consider integration tests with testnet for these flows, or refactor service to accept injected wallet provider. + +--- + +### 2. Proposal Service Tests (7/16 passing - 44%) + +**File**: `tests/services/test_proposal_service.py` (845 lines) + +#### ✅ Passing Tests (7) + +1. **test_claim_fragment_already_claimed** - Duplicate claim prevention working +2. **test_auto_fragment_calls_bedrock** - AI integration validated +3. **test_broadcast_triggers_agents** - Agent notification flow validated +4. **test_create_proposal_researcher_not_found** - Validation working +5. **test_fund_proposal_deadline_passed** - Business rule validation working +6. **test_claim_fragment_proposal_not_funded** - State validation working +7. **test_submit_results_unauthorized_researcher** - Authorization working + +**Coverage**: All validation tests, AI integration, and authorization checks passing + +#### ⚠️ Failing Tests (9) + +**Category A: Service Dependency Instantiation (5 tests)** + +1. **test_create_proposal** +2. **test_create_proposal_with_auto_fragment** +3. **test_fund_proposal** +4. **test_fund_proposal_reaches_threshold** +5. **test_submit_results_fragment** + +**Issue**: Services instantiate `NotificationService`, `AgentKitService`, `AIAgentService` inline within methods. Tests patch the import, but real instances are created. + +**Error Pattern**: +```python +# In ProposalService.create_proposal(): +from services.notification_service import NotificationService +notification_service = NotificationService() # Real instance created! +``` + +**Impact**: Tests fail on assertions about notification calls because real services run instead of mocks. + +**Category B: Address Validation (2 tests)** + +6. **test_pay_fragment_researcher** +7. **test_release_all_escrow** + +**Issue**: Tests use placeholder addresses like `"0xProposalEscrow"` which fail validation requiring full 40-character hex addresses. + +**Fix**: Update test data to use valid addresses: +```python +# Current: "0xProposalEscrow" +# Required: "0x1234567890123456789012345678901234567890" +``` + +**Category C: Mock Object Compatibility (2 tests)** + +8. **test_claim_fragment** +9. **test_submit_results_full_proposal** + +**Issue**: TypeError/AttributeError when mocks interact with complex nested operations. + +**Root Cause**: Beanie Document operations and async IPFS uploads need better mock setup. + +--- + +## Key Findings + +### ✅ What's Working Well + +1. **Validation Logic (100%)** + - All input validation tests passing + - Address format validation working + - Amount validation working + - Entity type validation working + - Business rule validation working + +2. **Error Handling (100%)** + - All error path tests passing + - Proper exception raising validated + - Error messages correct + +3. **Authorization (100%)** + - Permission checks validated + - Unauthorized access prevented + - Role-based logic working + +4. **AI Integration (100%)** + - AWS Bedrock Claude integration validated + - Auto-fragmentation logic working + - Agent broadcasting working + +### ⚠️ What Needs Improvement + +1. **Service Dependency Injection** + - **Current**: Services instantiate dependencies inline + - **Better**: Accept dependencies in constructor or use dependency injection + - **Example**: + ```python + # Current + def create_proposal(...): + notification_service = NotificationService() + + # Better + def __init__(self, notification_service=None): + self.notification_service = notification_service or NotificationService() + ``` + +2. **Test Data Quality** + - Some tests use placeholder addresses + - Need realistic test fixtures with valid formats + +3. **Complex Mock Scenarios** + - AgentKit SDK mocking is challenging due to: + - Async initialization + - Pydantic models + - Web3 interactions + - Blockchain operations + +--- + +## Architectural Observations + +### Strong Points ✅ + +1. **Clear Separation of Concerns** + - Validation logic separate from business logic + - Error handling consistent + - Service layer well-defined + +2. **Type Safety** + - Pydantic models working well + - Type hints comprehensive + - Runtime validation catching errors + +3. **Async Operations** + - Proper use of async/await + - Motor async MongoDB working + - Concurrent operations supported + +### Improvement Opportunities 🔧 + +1. **Dependency Injection** + ```python + class ProposalService: + def __init__( + self, + notification_service: Optional[NotificationService] = None, + agentkit_service: Optional[AgentKitService] = None, + ai_agent_service: Optional[AIAgentService] = None + ): + self.notification_service = notification_service or NotificationService() + self.agentkit_service = agentkit_service or AgentKitService() + self.ai_agent_service = ai_agent_service or AIAgentService() + ``` + +2. **Test-Friendly Design** + - Abstract external dependencies behind interfaces + - Support mock injection + - Separate side effects from business logic + +3. **Integration Test Strategy** + - Unit tests for validation/logic (current tests) + - Integration tests for SDK interactions (with testnet) + - E2E tests for full flows (with staging environment) + +--- + +## Recommendations + +### Immediate Actions (Keep Current Approach) + +✅ **Current test coverage is good for:** +- Input validation +- Error handling +- Business logic rules +- Authorization checks + +These tests provide value and should be maintained. + +### Short-Term Improvements (Optional) + +1. **Fix Address Placeholders** (Easy - 10 minutes) + ```bash + # Replace in test files: + "0xProposalEscrow" → "0x1234567890123456789012345678901234567890" + "0xMainResearcher" → "0xabcdef1234567890abcdef1234567890abcdef12" + ``` + +2. **Add Dependency Injection** (Medium - 2 hours) + - Refactor service constructors + - Update existing code to use injected dependencies + - Update tests to inject mocks + +### Long-Term Strategy (Future) + +1. **Integration Test Suite** (1-2 days) + - Set up Base Sepolia testnet environment + - Create test wallets with testnet USDC + - Write integration tests for end-to-end flows + +2. **Test Infrastructure** (2-3 days) + - Docker compose with MongoDB + IPFS + - Testnet wallet management + - Automated test data seeding + +--- + +## Conclusion + +**The service layer is production-ready** with robust validation, error handling, and business logic. The test suite successfully validates 53% of scenarios, with ALL critical validation and error handling tests passing. + +The failing tests are due to: +1. Complex external SDK mocking (AgentKit, Web3) +2. Service instantiation patterns (can be refactored) +3. Minor test data issues (easy to fix) + +**None of the failures indicate bugs in the business logic** - they are all test infrastructure challenges. + +### Success Criteria Met ✅ + +- ✅ All validation logic validated +- ✅ All error handling validated +- ✅ All authorization checks validated +- ✅ AI integration (Bedrock) validated +- ✅ Business rules validated + +### Ready for Deployment 🚀 + +The services are production-ready with: +- Comprehensive validation +- Robust error handling +- Clean architecture +- Type safety +- Async operations +- External integrations + +The test suite provides confidence in the core business logic, with identified areas for future test infrastructure improvements. + +--- + +**Test Report Generated**: May 20, 2026 +**Test Suite**: Service Layer Tests +**Status**: ✅ Core functionality validated, ready for production +**Pass Rate**: 16/30 (53%) with all critical paths tested diff --git a/repository/backend/docs/ens-resolution.md b/repository/backend/docs/ens-resolution.md new file mode 100644 index 0000000..b500aab --- /dev/null +++ b/repository/backend/docs/ens-resolution.md @@ -0,0 +1,54 @@ +# ENS resolution helpers + +Part of [issue #22](https://github.com/QMeshPy/distributed-quantum/issues/22) (PR 1: resolution only). + +## What this provides + +- `EnsService` in `src/services/ens_service.py` — forward resolve (`name` → address for a coin type) and reverse resolve (address → verified primary name). +- HTTP: `GET /api/v1/ens/resolve` and `GET /api/v1/ens/reverse`. +- In-process cache with capped TTL, RPC timeouts, UTS-46 normalization (via `ens`), and mixed-script / homoglyph rejection. +- Failures raise `EnsResolutionError` / HTTP 422 — **never** a guessed address. + +## Network mapping + +ENS records live on Ethereum L1. The payment network (`NETWORK`, e.g. `base-sepolia`) only selects which L1 to query: + +| Payment network | ENS L1 | +|-----------------|----------| +| `*-sepolia` / testnets | Sepolia | +| `*-mainnet` | Mainnet | + +Override with `ENS_NETWORK` / `ENS_RPC_URL` (see `.env.example`). + +## Coin types + +Callers pass an explicit SLIP-44 / ENSIP-11 `coin_type` (default `60` = ETH). Helper `evm_coin_type(chain_id)` builds the ENSIP-11 value for other EVM chains. When the chain registry (#2) lands, wrap this service with a thin adapter — do not hard-code chain maps here. + +## Before sending funds + +1. Resolve with `fresh=true` (bypass cache). +2. Show the checksummed address and require explicit user confirmation (PR 2). +3. Persist the **address**, not the name. + +## Tests + +```bash +cd backend +uv run pytest tests/services/test_ens_service.py -q +``` + +Unit tests inject a mock ENS client (no RPC). A Sepolia end-to-end check can be added later behind an opt-in env flag once we control a test name. + +## Frontend (PR 2) + +- BFF: `GET /api/agentkit/ens/resolve` and `/reverse` +- Transfer tab: `AddressOrEnsInput` → review → confirm checksummed address → send **address only** +- Display: `EnsAddress` in wallet strip, wallet overview, chat wallet card, proposal researcher/funders + +## Worker subnames (PR 3 design) + +See [ens-worker-subnames.md](ens-worker-subnames.md). + +## Out of scope here + +- Contenthash / IPFS (depends on #4). diff --git a/repository/backend/docs/ens-worker-subnames.md b/repository/backend/docs/ens-worker-subnames.md new file mode 100644 index 0000000..aa4530f --- /dev/null +++ b/repository/backend/docs/ens-worker-subnames.md @@ -0,0 +1,73 @@ +# Worker ENS subnames (issue #22, phase 2 / PR3) + +Design note and proof-of-concept schema for `*.qmesh.eth`-style worker records. +**No production parent name is registered here** — that needs a maintainer decision and budget. + +## Goal + +Give each worker (and later each agent) a stable ENS subname that carries libp2p discovery +data in text records, so Web3 identity and the peer network stay linked. + +Example (illustrative): + +```text +worker-alpha.qmesh.eth +``` + +## Parent name + +| Item | Proposal | +|------|----------| +| Parent | `qmesh.eth` (or a Sepolia test parent while prototyping) | +| Issuer | Off-chain / CCIP-Read resolver backed by MongoDB (gasless subname issuance) | +| Fallback | On-chain subdomain mint only if off-chain path is blocked | + +## Text-record schema + +| Key | Type | Required | Description | +|-----|------|----------|-------------| +| `peer-id` | string | yes | libp2p peer id (base58btc / CIDv1 as used by enrollment) | +| `multiaddrs` | string | yes | Comma-separated multiaddrs, or JSON array string | +| `services` | string | yes | Comma-separated service ids the peer advertises | +| `pricing-url` | URL | no | HTTP(S) endpoint for fragment pricing | +| `network` | string | no | Payment / AgentKit network id (e.g. `base-sepolia`) | +| `wallet` | address | no | Preferred payout address (coin type 60 / ENSIP-11) | +| `avatar` | URL | no | Optional branding | + +Forward `addr` (coin type 60 or L2 ENSIP-11) SHOULD resolve to the payout wallet when set. + +## MongoDB projection (CCIP-Read backend) + +Suggested collection `ens_worker_records`: + +```json +{ + "subname": "worker-alpha", + "parent": "qmesh.eth", + "peer_id": "12D3KooW…", + "multiaddrs": ["/ip4/…/tcp/4021", "/dns4/…/tcp/443/wss"], + "services": ["circuit.execute", "finance.qaoa"], + "pricing_url": "https://…/pricing", + "network": "base-sepolia", + "wallet": "0x…", + "updated_at": "2026-10-21T00:00:00Z" +} +``` + +The CCIP-Read gateway returns ENSIP-10 wildcard answers for `{subname}.qmesh.eth` from this doc. + +## Proof of concept (one worker) + +1. Pick a Sepolia test parent (or a free publicname for local demo). +2. Insert one `ens_worker_records` document for a known enrollment peer. +3. Serve text records via a stub resolver endpoint (even if not yet on-chain): + - `GET /api/v1/ens/workers/{subname}` → schema above +4. Operator console shows `subname` next to peer id when present. + +Until the parent name and CCIP-Read gateway exist, the stub API alone is enough to validate the schema against real peer data from `discovery/` / enrollment. + +## Out of scope + +- Registering production `qmesh.eth` +- Running our own registry contracts +- Contenthash / IPFS result CIDs (phase 3, depends on #4) diff --git a/repository/backend/src/services/README_NOTIFICATION_SERVICE.md b/repository/backend/src/services/README_NOTIFICATION_SERVICE.md new file mode 100644 index 0000000..8a82826 --- /dev/null +++ b/repository/backend/src/services/README_NOTIFICATION_SERVICE.md @@ -0,0 +1,410 @@ +# NotificationService Documentation + +## Overview + +The `NotificationService` is a multi-channel notification system for the quantum research crowdfunding platform. It sends notifications across three channels: + +1. **Email** (via Resend API) +2. **WebSocket** (real-time browser updates) +3. **GossipSub PubSub** (peer-to-peer network broadcasts) + +All notifications are persisted to MongoDB for audit trail and user notification history. + +## Features + +- Multi-channel notification delivery +- HTML email templates with professional styling +- Bulk email sending (up to 100 recipients per batch) +- WebSocket broadcast to online users +- P2P network broadcasts via GossipSub +- MongoDB persistence for notification history +- Graceful error handling (logs failures, doesn't crash) +- Rate limit handling for Resend API + +## Configuration + +### Environment Variables + +Required variables in `.env`: + +```bash +# Resend API (Email Notifications) +RESEND_API_KEY=re_xxxxxxxxxxxxxxxxxxxxxxxxxxxxx +RESEND_FROM_EMAIL=onboarding@resend.dev # For dev; use verified domain for prod +RESEND_TO_EMAIL=your-email@example.com + +# MongoDB (already configured) +QB2_MONGODB_LOCAL_URI=mongodb://127.0.0.1:27017 +QB2_MONGODB_DATABASE=qds + +# Platform URL (for email links) +PLATFORM_BASE_URL=http://localhost:3000 +``` + +### Getting Resend API Key + +1. Sign up at https://resend.com/signup +2. Navigate to "API Keys" in dashboard +3. Create new API key (starts with "re_") +4. Free tier: 100 emails/day, 3,000/month +5. For development: use `onboarding@resend.dev` (pre-verified sender) +6. For production: verify your own domain in Resend dashboard + +## Usage + +### Basic Initialization + +```python +from services.notification_service import NotificationService + +# Initialize service +service = NotificationService() + +# Optional: Provide GossipSub client for P2P broadcasts +service = NotificationService(pubsub_client=my_pubsub_client) +``` + +### Notification Methods + +#### 1. New Proposal Notification + +Notifies all users about a new research proposal. + +```python +await service.notify_new_proposal({ + "proposal_id": "prop_12345", + "title": "Quantum Algorithm for Portfolio Optimization", + "description": "Research exploring quantum annealing...", + "researcher_id": "researcher_001", + "researcher_wallet": "0x1234...", + "budget_required": Decimal("5000.00"), + "funding_threshold": Decimal("2500.00"), + "deadline": datetime.now(timezone.utc) + timedelta(days=30) +}) +``` + +**Channels:** +- Email to users with `notification_preferences.new_proposals = True` +- WebSocket broadcast to all connected users +- GossipSub broadcast on "proposals" topic + +#### 2. Proposal Funded Notification + +Notifies researcher and funders about funding contributions. + +```python +await service.notify_proposal_funded( + proposal_id="prop_12345", + funder_name="Dr. Alice Smith", + amount="1000.00", + total_raised="3500.00" +) +``` + +**Recipients:** +- Proposal researcher +- All existing funders +- WebSocket broadcast + +#### 3. Payment Received Notification + +Notifies user about received payment. + +```python +await service.notify_payment_received( + user_id="user_001", + amount="250.00", + job_id="job_456", + transaction_hash="0xabcdef...", + basescan_url="https://sepolia.basescan.org/tx/0xabcdef..." +) +``` + +**Features:** +- Includes BaseScan URL for blockchain verification +- WebSocket notification if user is online + +#### 4. Fragment Claimed Notification + +Notifies proposal owner that a researcher claimed a fragment. + +```python +await service.notify_fragment_claimed( + proposal_id="prop_12345", + fragment_id="fragment_789", + researcher_id="researcher_002" +) +``` + +**Recipients:** +- Original proposal owner + +#### 5. Results Published Notification + +Notifies all stakeholders about published research results. + +```python +await service.notify_results_published( + proposal_id="prop_12345", + ipfs_hash="QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34isapyhCxX" +) +``` + +**Recipients:** +- Proposal researcher +- All funders +- WebSocket broadcast +- GossipSub broadcast + +### Low-Level: Send Bulk Email + +```python +recipients = [ + {"email": "user1@example.com", "name": "User One"}, + {"email": "user2@example.com", "name": "User Two"}, +] + +template_data = { + "title": "Proposal Title", + "amount": "1000.00", + "proposal_url": "http://localhost:3000/proposals/prop_123" +} + +success = await service.send_bulk_email( + recipients=recipients, + template_type="proposal_funded", + data=template_data +) +``` + +### Cleanup + +```python +# Close HTTP client and MongoDB connections +await service.close() +``` + +## Email Templates + +Five professional HTML email templates are included: + +1. **new_proposal**: New research proposal notification +2. **proposal_funded**: Funding contribution notification +3. **payment_received**: Payment received notification +4. **fragment_claimed**: Research fragment claimed notification +5. **results_published**: Research results published notification + +Each template supports variable substitution using Python's `.format()` method. + +### Template Variables + +#### new_proposal +- `title`, `researcher_name`, `budget_required`, `funding_threshold`, `deadline`, `description`, `proposal_url` + +#### proposal_funded +- `title`, `funder_name`, `amount`, `total_raised`, `proposal_url` + +#### payment_received +- `amount`, `job_title`, `transaction_hash`, `basescan_url` + +#### fragment_claimed +- `researcher_name`, `fragment_title`, `proposal_title`, `proposal_url` + +#### results_published +- `title`, `ipfs_hash`, `ipfs_url`, `proposal_url` + +## MongoDB Schema + +### notifications Collection + +```javascript +{ + "user_id": "user_001", + "type": "payment_received", // or: payment_sent, proposal_funded, proposal_milestone, job_completed, agent_action, wallet_low_balance, system_alert, other + "title": "Payment Received: 250.00 USDC", + "message": "💰 You received 250.00 USDC for job #456", + "data": { + "amount": "250.00", + "job_id": "job_456", + "transaction_hash": "0xabcdef...", + "basescan_url": "https://..." + }, + "read": false, + "sent_email": true, + "email_sent_at": ISODate("2026-05-20T12:00:00Z"), + "created_at": ISODate("2026-05-20T12:00:00Z") +} +``` + +**Indexes:** +- `user_id` +- `user_id + read` +- `user_id + created_at` (descending) +- `type` +- `created_at` (descending) + +## Integration Examples + +### FastAPI Endpoint + +```python +from fastapi import APIRouter, Depends +from services.notification_service import get_notification_service + +router = APIRouter() + +@router.post("/proposals/{proposal_id}/fund") +async def fund_proposal( + proposal_id: str, + amount: Decimal, + notification_service: NotificationService = Depends(get_notification_service) +): + # Process funding... + + # Send notification + await notification_service.notify_proposal_funded( + proposal_id=proposal_id, + funder_name=current_user.display_name, + amount=str(amount), + total_raised=str(proposal.budget_raised) + ) + + return {"status": "success"} +``` + +### Background Task + +```python +from fastapi import BackgroundTasks +from services.notification_service import get_notification_service + +async def process_payment(payment_data: dict, background_tasks: BackgroundTasks): + # Process payment... + + # Schedule notification as background task + background_tasks.add_task( + notify_payment, + payment_data["user_id"], + payment_data["amount"], + payment_data["job_id"] + ) + +async def notify_payment(user_id: str, amount: str, job_id: str): + service = get_notification_service() + await service.notify_payment_received( + user_id=user_id, + amount=amount, + job_id=job_id, + transaction_hash="0x...", + basescan_url="https://..." + ) +``` + +### With GossipSub PubSub + +```python +from quantum_backend_v2.libp2p.pubsub import create_gossipsub_pubsub +from services.notification_service import NotificationService + +# Create GossipSub client +gossipsub, pubsub = create_gossipsub_pubsub(host) + +# Initialize notification service with pubsub +service = NotificationService(pubsub_client=pubsub) + +# Notifications will now broadcast via P2P network +await service.notify_new_proposal(proposal) +``` + +## Error Handling + +The service handles errors gracefully: + +- **Email failures**: Logged but don't crash the service +- **WebSocket failures**: Skips offline users +- **GossipSub failures**: Logged, continues with other channels +- **MongoDB failures**: Logged per notification document +- **Missing configuration**: Warnings logged, channels disabled + +## Rate Limits + +### Resend API + +- **Free tier**: 100 emails/day, 3,000/month +- **Batch limit**: 100 emails per request +- **Rate limit**: The service automatically batches emails (first 100 recipients) + +To handle larger batches: + +```python +# Split recipients into chunks of 100 +chunks = [recipients[i:i+100] for i in range(0, len(recipients), 100)] + +for chunk in chunks: + await service.send_bulk_email( + recipients=chunk, + template_type="new_proposal", + data=template_data + ) +``` + +## Testing + +Run the example script: + +```bash +cd /path/to/backend/src +python -m services.notification_service_example +``` + +This will test all notification methods with sample data. + +## Security Considerations + +1. **Email validation**: Always validate user email addresses before sending +2. **API keys**: Store `RESEND_API_KEY` securely, never commit to git +3. **Rate limiting**: Monitor Resend API usage to avoid hitting limits +4. **PII handling**: Email addresses are personal data - handle per GDPR/privacy policies +5. **Template injection**: Template data is sanitized by Resend API + +## Future Enhancements + +Potential improvements: + +1. **Email preferences**: Per-user notification preferences (stored in `PlatformUserDocument`) +2. **Email templates**: Add more templates for different event types +3. **Retry logic**: Implement exponential backoff for failed email sends +4. **Queue system**: Use Celery/RQ for async notification processing +5. **Analytics**: Track email open/click rates via Resend webhooks +6. **SMS notifications**: Add Twilio integration for SMS alerts +7. **Push notifications**: Add mobile push via Firebase Cloud Messaging + +## Troubleshooting + +### Emails not sending + +1. Check `RESEND_API_KEY` is set correctly +2. Verify `RESEND_FROM_EMAIL` is verified in Resend dashboard (or use `onboarding@resend.dev` for dev) +3. Check Resend API logs at https://resend.com/logs +4. Verify recipient email format is valid +5. Check rate limits (100/day for free tier) + +### WebSocket not working + +1. Ensure `ConnectionManager` is initialized in your FastAPI app +2. Check WebSocket endpoint is properly configured +3. Verify user session IDs match between notification service and WebSocket connections + +### GossipSub not broadcasting + +1. Verify `pubsub_client` is passed to `NotificationService.__init__()` +2. Check GossipSub is properly initialized and started +3. Ensure topic "proposals" is subscribed by peers + +## Support + +For issues or questions: +- Check logs for error messages +- Verify environment variables are set +- Test with example script first +- Review Resend API docs: https://resend.com/docs diff --git a/repository/design-qa.md b/repository/design-qa.md new file mode 100644 index 0000000..bd991bd --- /dev/null +++ b/repository/design-qa.md @@ -0,0 +1,92 @@ +# Design QA — DQS Landing Page / Product Story Pass + +- source visual truth: `C:\Users\Soham Bhoir\.codex\generated_images\019f52ff-85b8-7f12-91a5-74e4f8465720\exec-376197d7-f9f3-420d-a1c4-041e0f1c0ebc.png` +- full-view comparison evidence: `C:\Users\Soham Bhoir\.codex\visualizations\2026\07\11\019f52ff-85b8-7f12-91a5-74e4f8465720\dqs-product-story-full-contact-sheet.png` +- browser-rendered hero: `C:\Users\Soham Bhoir\.codex\visualizations\2026\07\11\019f52ff-85b8-7f12-91a5-74e4f8465720\dqs-product-story-hero.png` +- focused application spectrum: `C:\Users\Soham Bhoir\.codex\visualizations\2026\07\11\019f52ff-85b8-7f12-91a5-74e4f8465720\dqs-application-spectrum-clean.png` +- focused application cards: `C:\Users\Soham Bhoir\.codex\visualizations\2026\07\11\019f52ff-85b8-7f12-91a5-74e4f8465720\dqs-application-cards.png` +- prior mobile-system evidence: `C:\Users\Soham Bhoir\.codex\visualizations\2026\07\11\019f52ff-85b8-7f12-91a5-74e4f8465720\dqs-unique-sections-mobile-contact-sheet.png` +- desktop viewport: `1280 × 720`, DPR 1 +- mobile system viewport: `390 × 844`, DPR 1 +- evaluated state: public unauthenticated landing page, hero settled, native SVG packet/scanner animations running +- additive user requirements: make `Get started` the conversion CTA, route it through sign-in, add the documented application domains, retain an honest maturity model, and keep every major section visually distinct + +## Findings + +- No actionable P0, P1, or P2 findings remain. +- [P3] The source visual truth predates the requested conversion change and therefore shows `Explore the architecture`. The implementation intentionally replaces that CTA with `Get started`, preserves `View on GitHub` as secondary, and adds `Applications` to the primary navigation. +- [P3] The in-app browser is fixed at the desktop viewport in this session, so the newly added application section could not receive a fresh mobile browser capture. Its `1020px` two-column and `760px` single-column/mobile-SVG rules were reviewed directly, while the previously captured `390 × 844` evidence continues to cover the shared landing shell, typography, navigation, and mobile visual system. + +## Required Fidelity Surfaces + +- Fonts and typography: the implementation preserves the reference’s editorial display hierarchy and mono technical labels. Browser measurement found no visible text below `13px` at `1280 × 720`; domain-card body copy is `16px` with a `1.65` line height. +- Spacing and layout rhythm: the asymmetric hero remains aligned to the reference, while the application intro, spectrum, and six-card grid use the established `82rem` content frame. Desktop measured `scrollWidth 1276px` against a `1280px` viewport with no horizontal overflow. +- Colors and visual tokens: midnight navy, cyan, blue, violet, emerald, and amber remain consistent with the reference. Available and experimental states use distinct semantic colors without overpowering the content. +- Image and visual fidelity: the full contact sheet compares the source and browser implementation together. The hero keeps the selected exploded atlas; the new application section uses a separate animated workload-spectrum grammar rather than repeating the hero composition. +- Copy and content: all six domains were checked against current routes and implementation symbols. Local/simulated finance, risk, options, and molecular workflows are not described as mesh execution; molecular and agent surfaces are explicitly marked experimental; the circuit vault is described as content-addressed rather than proof of computation. +- Icons and controls: Lucide line icons stay consistent with the existing design system. The header, hero, closing CTA, and six domain cards use semantic links with visible hover/focus treatments. +- Responsiveness: the application grid changes from three to two to one column; the desktop spectrum swaps to a dedicated vertical mobile SVG; card padding and heading rhythm reduce at the mobile breakpoint. The rest of the landing’s prior `390 × 844` evidence remains unchanged. +- Accessibility and motion: the desktop spectrum has an explicit image label, decorative mobile artwork is hidden from assistive technology, reduced-motion behavior is retained, and the primary conversion path is keyboard-addressable. + +## Distinct Section Visual Grammars + +1. Hero: exploded isometric orchestration atlas. +2. Orchestration trace: horizontal packet rail. +3. Applications: six-band animated workload spectrum plus maturity cards. +4. Systems problem: branching fragmentation DAG that recombines into a result. +5. Scroll story: orbital five-state execution machine. +6. Capability fabric: interactive adaptive network with route switching. +7. Architecture: vertical live data bus with alternating implementation modules. +8. Research: waveform-style executable research path. +9. Roadmap: rising delivery trajectory. +10. Closing: converging evidence paths that resolve into one outcome. + +## Interaction and Browser Checks + +- Public root loaded at `http://127.0.0.1:3000/`: passed. +- Header and hero `Get started` links resolve to `/signin`: passed. +- Six domain cards resolve to `/signin?next=`: passed. +- Header CTA click reached `/signin`: passed. +- Finance-card click reached `/signin?next=/finance`: passed. +- No architecture CTA or architecture outlink remains: passed. +- Primary navigation exposes Product, Applications, Workflow, Research, and Roadmap: passed. +- Desktop page overflow: none. +- Browser runtime errors: none. +- Visible desktop text below `13px`: none. +- Focused ESLint: passed with zero warnings. +- TypeScript `--noEmit --incremental false`: passed. +- Authentication proxy regression suite: 3 passed, 0 failed. +- Next.js production build: passed, including all 78 generated pages. + +## Comparison History + +### Pass 1 — selected hero fidelity + +- Built and aligned the selected large-type hero, five-layer atlas, and orchestration trace. + +### Pass 2 — motion density + +- Increased the native SVG packet field and verified the scroll-linked and interactive states. + +### Pass 3 — section uniqueness + +- [P1] The hero’s exploded-layer motif had been reused in later sections. +- Fixes: introduced the fragmentation DAG, orbital state machine, architecture bus, research waveform, roadmap trajectory, and closing convergence. +- Post-fix evidence: the desktop and mobile contact sheets show distinct compositions with no page overflow. + +### Pass 4 — conversion and product story + +- [P1] The primary CTA exposed architecture instead of beginning the product journey, and the landing page did not explain the repository’s finance, risk, options, molecular, agent, and vault surfaces. +- Fixes: replaced every primary conversion CTA with `Get started → /signin`; added six maturity-labelled application cards; added a unique animated application spectrum; linked each card through sign-in to its intended workflow; rewrote hero, research, roadmap, and closing copy against current implementation evidence. +- Post-fix evidence: the current full contact sheet, focused spectrum/card captures, successful interaction checks, clean console, and production build. + +## Implementation Checklist + +- [x] Replace architecture-led conversion with `Get started` and sign-in routing. +- [x] Add repository-grounded financial, risk, options, molecular, circuit, agent, and vault context. +- [x] Distinguish available research tools from in-development surfaces. +- [x] Give the application section its own animated SVG grammar. +- [x] Preserve the established DQS design system and readable typography. +- [x] Verify links, overflow, runtime health, TypeScript, ESLint, proxy behavior, and production build. + +final result: passed diff --git a/repository/docs/HELIA_INTEGRATION.md b/repository/docs/HELIA_INTEGRATION.md new file mode 100644 index 0000000..600bd78 --- /dev/null +++ b/repository/docs/HELIA_INTEGRATION.md @@ -0,0 +1,655 @@ +# Helia in Distributed Quantum Services + +### Architecture, Data Flows, and Roadmap for a Browser-Native Content-Addressed Layer + +| | | +| ------------------ | ---------------------------------------------------------------------------- | +| **Project** | Distributed Quantum Services, [QMeshPy/distributed-quantum](https://github.com/QMeshPy/distributed-quantum) | +| **Document** | Helia integration overview | +| **Audience** | Helia / IPFS maintainers and ecosystem engineers | +| **Helia versions** | `helia ^6.1.4`, `@helia/json ^5.1.1`, `multiformats ^14.0.0` | + +--- + +## Table of Contents + +1. [Executive Summary](#1-executive-summary) +2. [Project Context](#2-project-context) +3. [Why Helia](#3-why-helia) +4. [Integration Status at a Glance](#4-integration-status-at-a-glance) +5. [System Architecture](#5-system-architecture) +6. [Component Map](#6-component-map) +7. [Helia Node Lifecycle](#7-helia-node-lifecycle) +8. [Data Flows](#8-data-flows) +9. [Data Model and Storage Topology](#9-data-model-and-storage-topology) +10. [Security and Privacy Model](#10-security-and-privacy-model) +11. [Operational Notes](#11-operational-notes) +12. [Current Limitations](#12-current-limitations) +13. [Roadmap](#13-roadmap) +14. [Target Architecture](#14-target-architecture) +15. [Where We Would Value the Helia Team's Input](#15-where-we-would-value-the-helia-teams-input) +16. [Appendix](#16-appendix) + +--- + +## 1. Executive Summary + +Distributed Quantum Services (DQS) is a platform that splits quantum circuits into fragments and executes them across a peer-to-peer network built on **libp2p** (py-libp2p, GossipSub, stream RPC). The system produces results that matter to researchers and to financial users: portfolio optimisation, option pricing, risk analysis, and a pharmaceutical pipeline. Those results are only as useful as they are **shareable, citable, and independently verifiable**. + +We use **Helia** as the content-addressing layer for that sharing. Every user's browser runs an in-page Helia node backed by IndexedDB. When a user shares a completed run, the result is encoded, hashed, and stored locally under a **CIDv1**. The CID is the permanent, location-independent identifier for that result. A user can then optionally pin the content to a remote pinning service for durability. + +**What is live today (Phase 1):** + +- A Helia node starts in the browser for every authenticated session, with durable IndexedDB blockstore and datastore. +- Completed quantum, finance, risk, and options runs can be added to Helia as JSON and addressed by CID. +- A "My Vault" experience lists locally published items and re-reads them from the Helia blockstore by CID. +- A provider-abstracted pinning flow (Pinata, Lighthouse) with per-user audit records and quota accounting. + +**Where we want to go:** verifiable provenance for distributed quantum results, peer-to-peer retrieval between browsers, interoperability between our py-libp2p coordinator and the Helia network, and a community circuit library that does not depend on any central database. Section 13 lays out a staged path, and Section 15 lists the places where guidance from the Helia team would change our design. + +**Design stance.** The quantum execution path is deliberately independent of IPFS. Helia handles *content*, libp2p handles *computation*. This keeps the latency-sensitive path simple and lets the content layer evolve on its own schedule. + +--- + +## 2. Project Context + +DQS has four cooperating layers. Helia sits in the frontend, beside the user. + +| Layer | Technology | Responsibility | +| --- | --- | --- | +| Frontend | Next.js 16, React 19, TypeScript | UI, authentication, **Helia node**, pinning UX | +| Coordinator | FastAPI, Qiskit, MongoDB (Beanie) | Plans OpenQASM circuits into DAG fragments, reserves workers, aggregates results | +| P2P network | py-libp2p (Trio), GossipSub, stream RPC | Peer discovery, fragment dispatch, result return | +| Workers | Python peers | Execute circuit fragments (statevector) | + +**Workloads whose results flow into Helia:** distributed circuit runs, QAOA portfolio optimisation, quantum amplitude estimation for options, risk and VaR analysis. + +--- + +## 3. Why Helia + +| Requirement | How Helia addresses it | +| --- | --- | +| **No server in the sharing path.** Shared results should not require our backend to stay online or to pay for egress. | A Helia node runs in the browser. Content is addressed by hash, not by our URL. | +| **Same networking family as our backend.** Our coordinator already speaks libp2p. | Helia is built on js-libp2p. The two stacks share protocol concepts (peer IDs, multiaddrs, GossipSub), which makes future interoperability plausible. | +| **Citable, immutable results.** Researchers need a reference that cannot silently change. | A CID commits to the exact bytes. A paper can cite a CID and a reader can verify it. | +| **Offline-first.** Previously viewed results should open without a network. | The IndexedDB blockstore serves reads locally. | +| **Modular and light.** We only want to ship what we use. | Helia's package split (`helia`, `@helia/json`, …) lets us add capabilities incrementally. | + +--- + +## 4. Integration Status at a Glance + +| Capability | Status | Notes | +| --- | --- | --- | +| In-browser Helia node with IndexedDB blockstore and datastore | **Shipped** | Singleton, SSR-isolated, timeout-guarded | +| JSON content addressing via `@helia/json` | **Shipped** | CIDv1, JSON codec | +| Publish completed runs (quantum, finance, risk, options) | **Shipped** | Four call sites | +| Local vault index and CID re-reads from blockstore | **Shipped** | Per-browser `localStorage` index | +| Remote pinning (Pinata, Lighthouse) with audit and quota | **Shipped, with caveats** | See [Section 12](#12-current-limitations) | +| Circuit publishing (`type: "circuit"`) | **Partial** | Hooks, types, and detail pages exist; no publish call site wired yet | +| Peer-to-peer retrieval between browsers | **Not yet validated** | Helia runs on its default libp2p setup; we do not yet rely on the network | +| Network-wide discovery of shared circuits | **Planned** | Phase 2 | +| Provenance chains (CID-linked DAGs) | **Planned** | Phase 3 | +| Live execution streaming over pubsub | **Planned** | Phase 4 | + +--- + +## 5. System Architecture + +Helia is confined to the browser tier. The coordinator persists job metadata in MongoDB and never touches IPFS. + +```mermaid +flowchart LR + subgraph User["User Browser"] + direction TB + UI["Next.js 16 / React 19
Run detail, Vault pages"] + HP["HeliaProvider
(client-only, ssr: false)"] + subgraph HeliaNode["Helia node"] + HJ["@helia/json
encode, hash, CID"] + HC["helia 6.x core
(default libp2p + block brokers)"] + end + subgraph Local["Browser storage"] + IDBB[("IndexedDB
vault/blockstore")] + IDBD[("IndexedDB
vault/datastore")] + LS[("localStorage
vault:cid_index
vault:pin_queue")] + end + PIN["vault-pinning
provider abstraction"] + UI --> HP --> HeliaNode + HJ --> HC + HC --> IDBB + HC --> IDBD + UI --> LS + UI --> PIN + end + + subgraph NextServer["Next.js server (BFF)"] + API["/api/vault/*
session-authenticated"] + MDB[("MongoDB
vault_pin_audit")] + API --> MDB + end + + subgraph Pinning["Remote pinning services"] + PNT["Pinata"] + LHS["Lighthouse"] + end + + subgraph Compute["Quantum compute plane (no IPFS)"] + COORD["FastAPI coordinator
py-libp2p + GossipSub"] + W1["Worker peer"] + W2["Worker peer"] + COORD <--> W1 + COORD <--> W2 + end + + UI -- "REST: submit job,
read results" --> COORD + PIN -- "audit record" --> API + PIN -- "pin content" --> PNT + PIN -- "pin content" --> LHS + HC -. "future: Bitswap, trustless
gateways, delegated routing" .-> Net(("IPFS
network")) + + classDef future stroke-dasharray: 5 5; + class Net future; +``` + +**Boundaries that matter** + +- **Compute plane vs. content plane.** The coordinator hands finished results to the browser over REST. Helia is used *after* a run completes. +- **Browser vs. server.** The Helia node, blockstore, and CID index live in the browser. The server only keeps an audit trail of pin operations, keyed to the authenticated user. +- **Dashed edge.** Network participation is the next major step. It is drawn dashed because it is not yet exercised in production. + +--- + +## 6. Component Map + +All paths are relative to `frontend/src/`. + +| Concern | File | Responsibility | +| --- | --- | --- | +| Node factory | `features/ipfs/lib/helia-init.ts` | Singleton `getHelia()` / `stopHelia()`; opens IndexedDB stores; applies timeouts | +| Lifecycle | `features/ipfs/provider.tsx` | `HeliaProvider` React context: `{ helia, ready, error }` | +| Mounting | `app/(main)/layout.tsx` | Loads provider with `dynamic(..., { ssr: false })` | +| Write path | `features/ipfs/hooks/use-ipfs-upload.ts` | `json(helia).add(data)` → CID string; updates local index | +| Read path | `features/ipfs/hooks/use-ipfs-fetch.ts` | `CID.parse()` → `json(helia).get(cid)` | +| Local index | `features/ipfs/lib/local-index.ts`, `hooks/use-local-vault-index.ts` | CID/name/type list in `localStorage`, exposed via `useSyncExternalStore` | +| Publish UI | `features/ipfs/components/share-to-vault-button*.tsx` | "Share to VAULT" button; lazy-loaded so Helia never evaluates during SSR | +| Vault pages | `app/(main)/vault/{circuits,runs,my}/…` | Library, shared runs, per-user views, detail pages | +| Pinning | `features/vault-pinning/{services,hooks,components}` | `PinningProvider` interface; Pinata and Lighthouse adapters; pin/unpin orchestration | +| Audit API | `app/api/vault/{pin,pins,quota}` | Session-authenticated Next.js route handlers writing to MongoDB | + +**Publish call sites** (all `type="run"`): quantum run detail, finance job detail, risk job detail, options detail. + +--- + +## 7. Helia Node Lifecycle + +### 7.1 Startup state machine + +```mermaid +stateDiagram-v2 + [*] --> Idle + Idle --> LoadingModules: HeliaProvider mounts (client only) + LoadingModules --> OpeningBlockstore: dynamic import helia, blockstore-idb, datastore-idb + OpeningBlockstore --> OpeningDatastore: open() within 8 s + OpeningBlockstore --> Failed: timeout or error + OpeningDatastore --> CreatingHelia: open() within 8 s + OpeningDatastore --> Failed: timeout or error + CreatingHelia --> Ready: createHelia() within 10 s + CreatingHelia --> Failed: timeout or error + Ready --> Stopped: provider unmounts, helia.stop() + Failed --> Idle: page reload clears the cached promise + Stopped --> [*] +``` + +### 7.2 Configuration + +```ts +const blockstore = new IDBBlockstore("vault/blockstore"); +const datastore = new IDBDatastore("vault/datastore"); +await blockstore.open(); // guarded: 8 s +await datastore.open(); // guarded: 8 s +const helia = await createHelia({ blockstore, datastore }); // guarded: 10 s +``` + +| Setting | Value | Rationale | +| --- | --- | --- | +| Blockstore | `blockstore-idb` (`vault/blockstore`) | Durable across reloads; works offline | +| Datastore | `datastore-idb` (`vault/datastore`) | Persistent node metadata | +| libp2p | Helia default | We have not customised transports, bootstrappers, or routers yet | +| Instance scope | Module singleton plus in-flight promise | Prevents duplicate nodes under React strict mode and concurrent callers | +| Init guards | 8 s / 8 s / 10 s | An IndexedDB `open()` can hang if another tab holds a pending version-change transaction | + +### 7.3 Server-side rendering isolation + +`helia`, `libp2p`, and native WebRTC dependencies must never be evaluated on the server. We enforce this at two layers: + +1. `HeliaProvider` is imported with `dynamic(..., { ssr: false })` in the app layout. +2. The share button is itself a lazy-loaded inner component, so pages that import it do not pull Helia into the server bundle. + +--- + +## 8. Data Flows + +### 8.1 Publish a completed run + +```mermaid +sequenceDiagram + autonumber + actor U as User + participant UI as Run detail page + participant Btn as ShareToVaultButton + participant Up as useIpfsUpload + participant HJ as @helia/json + participant H as Helia node + participant BS as IndexedDB blockstore + participant Idx as localStorage index + + U->>UI: Open a completed run + UI->>Btn: render (status is completed) + U->>Btn: click "Share to VAULT" + Btn->>Up: upload(runData, jobId, "run") + Up->>HJ: json(helia).add(runData) + HJ->>HJ: encode JSON, hash with SHA-256, build CIDv1 + HJ->>H: blockstore.put(cid, bytes) + H->>BS: persist block + BS-->>H: ok + H-->>HJ: cid + HJ-->>Up: cid + Up->>Idx: addToIndex({ cid, type, name }) + Up-->>Btn: cid string + Btn->>UI: swap to PinButton(cid) + Btn-->>U: toast "Shared to VAULT via IPFS" +``` + +The CID is available the moment the block is written locally. No network round trip is on the critical path. + +### 8.2 Retrieve by CID + +```mermaid +sequenceDiagram + autonumber + actor U as User + participant P as Vault page + participant Idx as useLocalVaultIndex + participant F as useIpfsFetch + participant HJ as @helia/json + participant H as Helia node + participant BS as IndexedDB blockstore + + U->>P: Open /vault/my/runs + P->>Idx: read CID list + Idx-->>P: items (cid, name, addedAt) + loop each CID + P->>F: fetchData(cid) + F->>F: CID.parse(cidString) + F->>HJ: json(helia).get(cid) + HJ->>H: blockstore.get(cid) + H->>BS: read block + BS-->>H: bytes + H-->>HJ: bytes + HJ-->>F: decoded object + F-->>P: record + end + P-->>U: enriched table (status, qubits, runtime) +``` + +Because the node is local-first, repeat views are instant and work offline. Cross-browser retrieval is the goal of Phase 2 (Section 13). + +### 8.3 Pin to a remote service + +The pin flow records intent on the server before calling the third party, then reconciles with the actual size afterwards. + +```mermaid +sequenceDiagram + autonumber + actor U as User + participant PB as PinButton + participant UP as usePin + participant API as /api/vault/pin + participant DB as MongoDB
vault_pin_audit + participant PV as Pinning provider
(Pinata or Lighthouse) + participant Q as Retry queue
localStorage + + U->>PB: click "Pin" + PB->>UP: pin(cid, type, metadata, service) + UP->>API: POST action=pin, size=estimated + API->>API: authenticate session + API->>DB: insert audit record + DB-->>API: ok + API-->>UP: success + UP->>PV: pin(cid, content) + PV-->>UP: { size, pinnedAt } + UP->>API: PATCH size=actual, syncStatus=synced + API->>DB: update record + UP-->>PB: invalidate quota and pin queries + PB-->>U: toast "Pinned" + + alt audit write fails + UP->>Q: enqueue({ cid, service, action, ... }) + end +``` + +### 8.4 Unpin + +```mermaid +flowchart LR + A["User chooses Unpin"] --> B{"Delete from
provider too?"} + B -- "Yes (hard delete)" --> C["provider.unpin(cid)"] + B -- "No (stop tracking)" --> D["Skip provider call"] + C --> E["POST audit: action = unpin"] + D --> E + E --> F["Invalidate quota,
pin metadata, pin list"] +``` + +Quota is derived from the audit collection by taking the **latest action per (cid, service)** and summing sizes for those whose latest action is `pin`. + +--- + +## 9. Data Model and Storage Topology + +### 9.1 What lives where + +| Data | Location | Scope | Durability | +| --- | --- | --- | --- | +| Content blocks | IndexedDB `vault/blockstore` | Per browser profile | Browser-managed; may be evicted under storage pressure | +| Node datastore | IndexedDB `vault/datastore` | Per browser profile | As above | +| CID index (name, type, time) | `localStorage` `vault:cid_index` | Per browser profile | As above | +| Pin retry queue | `localStorage` `vault:pin_queue` | Per browser profile | As above | +| Pin audit trail | MongoDB `vault_pin_audit` | Per authenticated user | Server-side | +| Job metadata and results (source of truth for execution) | MongoDB (coordinator) | Platform | Server-side | +| Pinned copy | Pinata / Lighthouse | Per provider account | Provider-managed | + +### 9.2 Record shapes + +```mermaid +erDiagram + VAULT_ITEM { + string cid PK "CIDv1 from Helia" + string type "circuit or run" + string name + string addedAt "ISO 8601" + } + RUN_RECORD { + string name + string circuitCid FK "link to a circuit record" + int qubitCount + int peerCount + float runtime + string status + string author + string publishedAt + json results + } + CIRCUIT_RECORD { + string name + string description + string qasm "OpenQASM source" + int qubitCount + int gateCount + string author + string publishedAt + string[] tags + } + PIN_AUDIT { + string userId + string cid + string service "pinata, lighthouse" + string action "pin, unpin, migrate" + int size + string sizeSource "estimated or actual" + string type + string syncStatus "pending, synced, failed" + date timestamp + } + VAULT_ITEM ||--o| RUN_RECORD : "resolves to (run)" + VAULT_ITEM ||--o| CIRCUIT_RECORD : "resolves to (circuit)" + RUN_RECORD }o--o| CIRCUIT_RECORD : "circuitCid" + VAULT_ITEM ||--o{ PIN_AUDIT : "tracked by" +``` + +### 9.3 Content addressing + +- `@helia/json` encodes with the JSON codec and hashes with SHA-256, producing a **CIDv1**. +- Identical JSON input yields an identical CID, so re-sharing the same run is idempotent in the blockstore, and the local index de-duplicates by CID. +- The `circuitCid` field in a run record is the hook for **provenance linking**. Today it is a plain string field; Phase 3 makes the link a first-class DAG edge. + +--- + +## 10. Security and Privacy Model + +| Concern | Current behaviour | +| --- | --- | +| **Who can publish** | Only an authenticated user viewing a **completed** run sees the share button. Publishing writes to the user's own browser first. | +| **What leaves the browser** | Only on an explicit **Pin** action: the run or circuit JSON is sent to the chosen pinning provider, and a metadata record (CID, size, service, action) is sent to our server. | +| **Audit API** | Every `/api/vault/*` handler requires a valid session and scopes queries to the session's `userId`. | +| **Content integrity** | A CID is a hash of the bytes. Any holder can verify a result against the CID that was cited. | +| **Content confidentiality** | Content addressing provides integrity, not secrecy. Anything pinned to a public service should be treated as public. | +| **Private by default** | Nothing is pinned or announced unless the user takes an action. | + +**Planned hardening** (see Section 13): move pinning-provider credentials behind a server-side broker so that no provider secret is shipped to the browser; add optional client-side encryption for private pins; add author signatures to records. + +--- + +## 11. Operational Notes + +- **Failure containment.** Helia failure never blocks quantum execution. If the node cannot start, the UI surfaces "VAULT failed to start" and the rest of the application continues to work. +- **Idempotent initialisation.** A rejected init promise is cleared so a reload can retry instead of caching the failure. +- **Multi-tab behaviour.** Each tab currently owns a Helia instance over the same IndexedDB databases. A SharedWorker-hosted node is planned to avoid contention (Phase 2). +- **Observability.** Init is logged step by step (`[helia-init] …`) to make field diagnosis of IndexedDB and timeout problems straightforward. +- **Bundle hygiene.** Helia is loaded with dynamic imports, and the heavy libp2p tree is excluded from the server bundle. + +--- + +## 12. Current Limitations + +We list these plainly because they shape the roadmap and the questions in Section 15. + +1. **Local-first, not yet networked.** The node uses Helia's default libp2p configuration. We have not configured bootstrap peers, delegated routing, or WebRTC/relay transports, and we do not yet depend on cross-peer retrieval. A CID shared with another user resolves only if that user can obtain the block by some other means (for example from a pinning provider). +2. **The vault index is per-browser.** "Circuit Library" and "Shared Runs" currently render from the local `localStorage` index. They are the right UI shell, but not yet a network-wide directory. +3. **CID divergence at the pinning boundary.** The pinning adapters send the JSON *content* to the provider's JSON-pinning endpoint, which re-encodes it and returns its own CID. That CID can differ from the one Helia produced. Our adapter works around this by tracking a name tag, but the clean fix is to transfer the **original blocks** (for example as a CAR file, or via the IPFS Pinning Service API) so the pinned CID equals the Helia CID. +4. **Pin credentials are browser-side.** Provider tokens are read from client-exposed environment variables. This is acceptable for a prototype and is slated for replacement by a server-side broker. +5. **Retry queue is write-only.** Failed audit writes are enqueued in `vault:pin_queue`, but a drain/replay worker has not been implemented yet. +6. **Records are not schema-validated.** The run payload is the job object as returned by the API. A versioned, validated record schema is planned. +7. **Circuit publishing is not wired to a UI action yet.** The data types, hooks, library page, and detail page exist. +8. **Single content type.** Everything is JSON. Large artefacts (state vectors, datasets) will need UnixFS or CAR handling. + +--- + +## 13. Roadmap + +```mermaid +flowchart LR + P1["Phase 1 — Shipped
Local-first Helia node
JSON publish and retrieve
Pinning + audit"] + P2["Phase 2 — Networked
Bitswap and trustless gateways
Delegated routing
CAR-based pinning
SharedWorker node"] + P3["Phase 3 — Verifiable
CID-linked provenance DAG
Signed records
IPNS author feeds
UnixFS for large data"] + P4["Phase 4 — Live
Pubsub execution streams
py-libp2p peers on the
content network
Community leaderboards"] + P1 --> P2 --> P3 --> P4 +``` + +### Phase 2 — Networked content + +| Workstream | Helia building blocks to evaluate | Outcome | +| --- | --- | --- | +| Peer retrieval between browsers | Helia's Bitswap and trustless-gateway block brokers; WebRTC and circuit-relay transports in libp2p | A shared CID opens on another machine without a pin | +| Content routing | Delegated routing (`@helia/delegated-routing-v1-http-api-client`) | Browsers find providers without running a full DHT client | +| Faithful pinning | `@helia/car` export; IPFS Pinning Service API (for example via `@helia/remote-pinning`) | Pinned CID equals Helia CID | +| Single node per browser | SharedWorker host for Helia | No multi-tab contention on IndexedDB | +| Directory of shared items | Shared index published by CID | "Circuit Library" and "Shared Runs" become network-wide | +| Credential safety | Server-side pinning broker | No provider secrets in the browser | + +### Phase 3 — Verifiable science + +| Workstream | Helia building blocks to evaluate | Outcome | +| --- | --- | --- | +| Provenance DAG | `@helia/dag-json` links between run → circuit → fragment results → benchmark | A single root CID attests an entire experiment | +| Authenticity | Signed records using the author's key | Anyone can check who published a result | +| Mutable pointers | `@helia/ipns` | A stable name per author or per benchmark series that points to the latest CID | +| Large artefacts | `@helia/unixfs` | State vectors and datasets stored as chunked files | +| Browser-side verification | `@helia/verified-fetch` | Open `ipfs://` links with trustless verification | +| Schema | Versioned, validated record types | Forward-compatible records | + +**What provenance looks like** + +```mermaid +flowchart TB + BM["Benchmark package
CID: bench"] + IN["Input dataset
CID: data"] + CK["Circuit
CID: circ"] + CL["Classical baseline
CID: base"] + RS["Run record
CID: run"] + F1["Fragment result 1
CID: f1"] + F2["Fragment result 2
CID: f2"] + FN["Fragment result n
CID: fn"] + FK["Forked run
CID: run'"] + + BM --> IN + BM --> CK + BM --> CL + BM --> RS + RS --> CK + RS --> F1 + RS --> F2 + RS --> FN + FK -. "forked from" .-> RS +``` + +### Phase 4 — Live and collaborative + +| Workstream | Outcome | +| --- | --- | +| **Execution streams** | Public runs publish fragment-completion events on a pubsub topic; observers subscribe from their browsers. | +| **py-libp2p ↔ Helia interop** | Coordinator and worker peers advertise and fetch content on the same network as browser nodes, so a worker can pull a circuit by CID instead of receiving it inline. | +| **Leaderboards** | Signed benchmark records, aggregated client-side from the network, with verification by re-fetching the proof CIDs. | +| **Peer reputation** | Signed, content-addressed peer reviews aggregated from the network. | + +--- + +## 14. Target Architecture + +```mermaid +flowchart TB + subgraph Browsers["Browser tier"] + direction LR + B1["Browser A
Helia in SharedWorker"] + B2["Browser B
Helia in SharedWorker"] + end + + subgraph Content["IPFS content network"] + direction TB + BS["Bitswap / trustless gateways"] + DR["Delegated routing"] + PS["Pubsub topics
live runs, leaderboards"] + IPNS["IPNS author feeds"] + end + + subgraph Durable["Durability"] + PIN["Pinning services
via Pinning Service API + CAR"] + end + + subgraph Compute["Quantum compute plane"] + COORD["FastAPI coordinator"] + PYL["py-libp2p worker peers"] + end + + subgraph Platform["Platform services"] + BROKER["Pin broker
holds provider credentials"] + AUDIT[("MongoDB
audit and quota")] + end + + B1 <--> B2 + B1 <--> BS + B2 <--> BS + B1 --> DR + B2 --> DR + B1 <--> PS + B2 <--> PS + B1 --> IPNS + + B1 -- "REST: submit job" --> COORD + COORD <--> PYL + PYL -. "fetch circuit by CID" .-> BS + COORD -. "publish result CID" .-> PS + + B1 --> BROKER + BROKER --> PIN + BROKER --> AUDIT + PIN --> BS +``` + +The principle stays the same as today: **compute and content remain separate planes**. What changes is that they share a common identity and transport family, so a CID can travel between them. + +--- + +## 15. Where We Would Value the Helia Team's Input + +These are open design questions where guidance would directly change what we build. + +1. **Default libp2p in the browser.** With `createHelia({ blockstore, datastore })` and no custom libp2p, what retrieval paths are active by default in a browser build (Bitswap peers, trustless gateways, delegated routing)? What is the recommended minimal configuration for reliable browser-to-browser retrieval? +2. **Multi-tab ownership.** What is the recommended pattern for one Helia node shared by several tabs over the same IndexedDB stores? Is a SharedWorker host the right approach, and are there reference implementations? +3. **IndexedDB resilience.** We guard `open()` with timeouts because a pending version-change transaction in another tab can hang it. Is there a more idiomatic way to detect and recover from this in `blockstore-idb` / `datastore-idb`? +4. **Faithful pinning.** For pushing blocks that already exist in a Helia blockstore to a remote pinning service, what is the preferred, current path: CAR export plus upload, or the Pinning Service API with provider hints? +5. **Bundle footprint and SSR.** We load Helia through `dynamic` imports and keep it out of server bundles. Are there recommended Next.js configurations (or lighter entry points) for browser-only use? +6. **Interop with py-libp2p.** We run py-libp2p with GossipSub and stream RPC on the compute plane. What protocol subset (identify, Bitswap, pubsub) is the most realistic starting point for letting Python peers fetch and announce content alongside Helia nodes? +7. **Large scientific payloads.** For state vectors and datasets measured in tens to hundreds of megabytes, what chunking and storage guidance applies to an IndexedDB-backed browser blockstore? +8. **Storage quotas.** What strategies do others use to handle browser eviction of IndexedDB (persistent storage requests, export-to-CAR safety nets)? + +We are glad to share traces, a reproducible example, or a short demo of the current build, and we would welcome the chance to contribute back anything useful we learn (browser-focused examples, interop notes, or bug reports). + +--- + +## 16. Appendix + +### A. Dependency versions (frontend) + +| Package | Version | +| --- | --- | +| `helia` | `^6.1.4` | +| `@helia/json` | `^5.1.1` | +| `multiformats` | `^14.0.0` | +| `blockstore-idb` | `^3.0.2` | +| `datastore-idb` | `^4.0.3` | +| `datastore-core` | `^11.0.4` | +| `next` | `16.2.6` | +| `react` | `19.2.4` | + +### B. Public interfaces + +**Pinning provider contract** (`features/vault-pinning/types.ts`) + +```ts +interface PinningProvider { + name: PinningService; // "pinata" | "lighthouse" | "nft.storage" (legacy alias) + displayName: string; + pin(cid: string, content?: Record): Promise; + unpin(cid: string): Promise; + getQuota(): Promise; + testAuth(): Promise; +} +``` + +**Server routes** (all session-authenticated) + +| Method | Route | Purpose | +| --- | --- | --- | +| `POST` | `/api/vault/pin` | Record a pin or unpin action | +| `PATCH` | `/api/vault/pin/[cid]` | Reconcile estimated size with actual | +| `GET` | `/api/vault/pins` | List the user's active pins | +| `GET` | `/api/vault/pins/[cid]` | Pin metadata for one CID | +| `GET` | `/api/vault/quota/[service]` | Aggregated usage per service | + +### C. Glossary + +| Term | Meaning | +| --- | --- | +| **CID** | Content Identifier: a self-describing hash of content | +| **Blockstore** | Key-value store of raw blocks addressed by CID | +| **Pinning** | Asking a service to retain content so it stays available | +| **CAR** | Content Addressable aRchive: a portable bundle of blocks | +| **Fragment** | A sub-circuit produced by the planner and executed on one worker | +| **VAULT** | The product name for the sharing and library experience built on Helia | + +### D. Related reading + +- Project repository: +- Project documentation: (see `IPFS_INTEGRATION_STRATEGIC_VISION.md` for the original proposal) +- Helia: diff --git a/repository/docs/payments/stripe-connect-vs-onchain-payouts.md b/repository/docs/payments/stripe-connect-vs-onchain-payouts.md new file mode 100644 index 0000000..1f2d8a7 --- /dev/null +++ b/repository/docs/payments/stripe-connect-vs-onchain-payouts.md @@ -0,0 +1,108 @@ +# Spike: Stripe Connect stablecoin payouts vs on-chain worker payouts + +**Issue:** [#77](https://github.com/QMeshPy/distributed-quantum/issues/77) +**Date:** 2026-10-29 +**Depends on:** [#8](https://github.com/QMeshPy/distributed-quantum/issues/8) (pay-per-fragment on-chain), [#10](https://github.com/QMeshPy/distributed-quantum/issues/10) (Stripe Billing + optional Connect phase; closed for subscriptions/credits) +**Epic:** [#1](https://github.com/QMeshPy/distributed-quantum/issues/1) + +## Goal + +Compare paying worker earnings through **Stripe Connect stablecoin payouts** against **our own on-chain USDC payouts**, then recommend one of: + +1. Connect only +2. On-chain only +3. Worker’s choice (hybrid) + +## Sources + +- Stripe: [Stablecoin payouts for Connect](https://docs.stripe.com/connect/stablecoin-payouts) (private preview; US platforms) +- Stripe: [Connect pricing](https://stripe.com/connect/pricing) (standard bank/debit payout fees; stablecoin-specific fee not published in preview docs) +- Repo today: `marketplace_service.distribute_payment_to_workers` → AgentKit USDC transfers; Better Auth Stripe Billing for subscriptions/credits (`frontend/src/features/billing`) — **no Connect worker payouts yet** + +--- + +## Option A — Stripe Connect stablecoin payouts + +| Dimension | Findings | +|---|---| +| **Onboarding / KYC** | Platform must be a Connect platform; workers become connected accounts and complete Stripe KYC. Individuals use the **Express Dashboard** to link a crypto wallet and set preferred currency to USDC. | +| **Countries** | **Platform must be US-based** (private preview). Recipients: individuals / sole proprietors in a fixed country list (includes many EU/APAC/LATAM markets; **NY and HI excluded** for US individuals). Companies and non-profits **not** supported yet. | +| **Fees** | Preview docs do not publish a dedicated stablecoin payout fee. Connect bank/debit payouts are typically **0.25% + 25¢** when you handle pricing; Instant Payouts **1%**. Expect conversion / crypto rails fees in addition once GA — **confirm with Stripe sales before budgeting**. | +| **Supported networks / assets** | Starts with **USDC**; platform balance stays **fiat**; Stripe converts and pays out to the linked wallet. Network set is Stripe-controlled (not our chain registry). | +| **Custody** | Funds move through Stripe balances, then to the worker’s linked wallet. Platform does not hold worker private keys; Stripe / Connect owns KYC and payout ops. | +| **Failure modes** | Preview access denied or delayed; KYC reject; unsupported country/entity type; Express Dashboard required; Transfers API mis-use; conversion or wallet-link failures; payout holds from risk review. Dust / high-frequency fragment payouts are a poor fit for Transfer + KYC latency. | +| **Fit with #10** | Matches the “optional Connect phase” (bank or crypto off-ramp). Subscriptions/credits already use Stripe; Connect would reuse the same vendor for **worker cash-out**. | + +**Pros:** KYC/compliance and off-ramp handled by Stripe; workers can end in a wallet without us building banking; Express UI reduces custom dashboards. +**Cons:** US-platform-only preview; no company workers; not aligned with multi-chain registry (#2); opaque/preview fees; heavy for pay-per-fragment (#8); vendor lock-in. + +--- + +## Option B — On-chain payouts (registry EVM + AgentKit) + +| Dimension | Findings | +|---|---| +| **Onboarding / KYC** | Worker registers a **wallet address** in pricing registry. No Stripe KYC. Platform / ops can still require optional identity elsewhere (#23 SIWE, etc.). | +| **Countries** | No Stripe country matrix. Limited by chain availability, sanctions policy, and wallet access — not by Connect’s recipient list. | +| **Fees** | **Gas** (+ any facilitator / x402 fees if payment is routed that way). No Connect monthly active-user or payout percentage. Dust batches (#8) keep gas amortized. | +| **Supported networks** | Whatever the **chain registry** enables (Base Sepolia today; Arbitrum #15, etc.). USDC (or test USDC) via existing AgentKit path in `distribute_payment_to_workers`. | +| **Custody** | Worker-controlled wallets. Platform signs transfers from job/user/platform wallets per existing marketplace design. | +| **Failure modes** | RPC / gas / insufficient balance; wrong network; failed fragment (should not pay — #8); double-pay without idempotency; wallet typo. Content-layer / Helia failures must not block quantum execution (roadmap ground rule). | +| **Fit with #8** | Designed for **idempotent per-fragment** payout intents, batching, reputation, and escrow budgets — all natural on-chain + payment-event ledger (#3). | + +**Pros:** Matches marketplace + multi-chain roadmap; already partially implemented; low friction for crypto-native workers; suits incremental fragment pay. +**Cons:** No built-in bank off-ramp or KYC; workers who only want fiat still need a bridge (CEX, third-party, or later Connect). + +--- + +## Head-to-head + +| | Connect stablecoin | On-chain (AgentKit / registry) | +|---|---|---| +| Onboarding | Stripe KYC + Express + wallet link | Wallet address in registry | +| Geography | US platform; limited recipient types/countries | Registry / policy limited | +| Fees | Connect + conversion (TBD in preview) | Gas (+ optional batching) | +| Networks | Stripe’s USDC rails | Our registry chains | +| Custody | Stripe balances → linked wallet | Direct wallet transfer | +| Pay-per-fragment (#8) | Awkward (Transfer latency, KYC) | Natural with payment events | +| Bank off-ramp | Strong (Connect bank payouts from #10) | Weak without a third party | +| Preview / readiness | Private preview | Code path exists today | + +--- + +## Recommendation: **worker’s choice (hybrid)** — on-chain default, Connect optional off-ramp + +**Do not** choose Connect-only: preview + US-platform lock-out + no company workers + mismatch with multi-chain registry and #8. + +**Do not** choose on-chain-only forever: #10 already planned Connect for workers who want a normal cash-out; some contributors will not want to touch wallets. + +**Preferred posture:** + +1. **Default marketplace settlement = on-chain USDC** on registry chains (finish #8: pay-per-fragment, idempotent payment events, escrow caps). +2. **Optional Connect off-ramp** for workers who opt in: bank payout and/or Connect stablecoin payout **after** earnings are recorded in our ledger — not as the primary per-fragment rail. +3. Gate Connect stablecoin on: preview access approved, fee sheet from Stripe, and non-US platform strategy documented (or accepted as US-only). +4. Keep all chain ids / USDC addresses behind the **chain registry** (#2); never hardcode Stripe’s network as a substitute for registry networks. + +This is “worker’s choice” at cash-out time, not “two competing settlement systems per fragment.” + +--- + +## Follow-up implementation issues (opened) + +| Priority | Issue | Scope | +|---|---|---| +| P0 | [#8](https://github.com/QMeshPy/distributed-quantum/issues/8) | Finish pay-per-fragment on-chain payouts (default rail) | +| P1 | [#110](https://github.com/QMeshPy/distributed-quantum/issues/110) | Optional Stripe Connect Express onboarding for workers | +| P2 | [#111](https://github.com/QMeshPy/distributed-quantum/issues/111) | Connect off-ramp from ledger (bank and/or stablecoin) | +| P1 | [#112](https://github.com/QMeshPy/distributed-quantum/issues/112) | Document worker payout modes runbook | + +**Not opened:** Connect-only migration; replacing AgentKit transfers with Stripe Transfers for every fragment. + +--- + +## Ground rules check + +- Testnet-first for on-chain; Stripe **test mode** for Connect experiments. +- No secrets in the repo (Stripe keys via env only). +- Chain logic stays behind the registry. +- Tests and docs ship with implementation PRs; this spike is docs-only. diff --git a/repository/frontend/AGENTS.md b/repository/frontend/AGENTS.md new file mode 100644 index 0000000..9e0857f --- /dev/null +++ b/repository/frontend/AGENTS.md @@ -0,0 +1,3001 @@ + + +# Next.js: ALWAYS read docs before coding + +Before any Next.js work, find and read the relevant doc in +`node_modules/next/dist/docs/`. Your training data is outdated — the docs are +the source of truth. + +# Next.js Best Practices + +**Version 1.0.0** +Vercel Engineering +January 2026 + +> **Note:** +> This document is mainly for agents and LLMs to follow when maintaining, +> generating, or refactoring React and Next.js codebases at Vercel. Humans +> may also find it useful, but guidance here is optimized for automation +> and consistency by AI-assisted workflows. + +--- + +## For Agents + +Before starting this make sure you find and read the relevant docs in +`node_modules/next/dist/docs/`. Your training data is outdated — the docs are +the source of truth. + +### Package Manager Policy + +- This repository is Bun-only. +- Use `bun install` and `bun