docs: record EtherCalc design, implementation plan, and progress

superpowers/plans/2026-10-29-ethercalc.md

@@ -0,0 +1,57 @@
+# EtherCalc Integration Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. Native execution is authorized; do not repeat design approval.
+
+**Goal:** Persistent authenticated collaborative workbooks for completed QMesh experiments, with native-format IPFS backups.
+**Architecture:** Next feature services use existing Better Auth/MongoDB. A same-origin Bun gateway bounds EtherCalc HTTP/WebSocket access. Pinned upstream workerd persists rooms independently. App-managed tabs represent workbook sheets.
+**Tech Stack:** Next 16, React 19, Bun, MongoDB, workerd, Kubo.
+**Spec:** ../specs/2026-10-29-ethercalc.md
+
+## Global Constraints
+- Feature branches never contain codex; exactly one issue, #103.
+- Preserve source JSON, units, provenance and existing collaborator edits.
+- Authenticate all workbook data surfaces. Do not trust browser-submitted ownership.
+- Manual public IPFS consent; actual bytes and returned CID; restore to new identity.
+- Bun-only project operations. Reuse installed dependencies.
+
+## Review Focus
+- Concurrent creation/add-run and partial EtherCalc import failures cannot overwrite edits (Task 2).
+- Alternate room paths, frame payloads and revoked sessions cannot bypass membership (Task 3).
+- Unowned historical pharma runs cannot be claimed by guessing identifiers (Task 1).
+- Malformed or huge snapshots/ranges cannot escape room mappings or exhaust resources (Tasks 2/4).
+- Snapshot failure never appears pinned; restored formulas cannot fetch private sheets outside membership (Tasks 3/4).
+
+### Task 1: Owned source runs and mapping
+**Files:** backend pharma router and tests; frontend pharma routes; shared spreadsheet types/adapters and tests.
+**Interfaces:** `mapRun(kind, run): SheetSeed[]`, `SheetSeed {name, save, source}`; server source fetch requires authenticated backend assertion.
+- [ ] Write failing tests for pharma owner isolation, mapping units/SMILES/confidence/counts, literal formula-like input and JSON fidelity.
+- [ ] Run focused pytest and Bun tests; expect failure before implementation.
+- [ ] Add ownership at pharma submit/read/list/live/cancel and propagate existing auth assertion. Map exact completed server results to native typed cells and immutable source sheets.
+- [ ] Run focused tests; expect pass. Commit Task 1.
+
+### Task 2: Workbook and membership services
+**Files:** frontend features/spreadsheets server/types; app/api/spreadsheets thin handlers; tests.
+**Interfaces:** create/reopen, list, detail, invite, join, add-run services; workbook records carry owner/members/sheets/source runs/status.
+- [ ] Test idempotent concurrent creation, membership checks, explicit invite acceptance and failed import recovery.
+- [ ] Implement unique run/version mapping, atomic state transitions, server-owned run fetch, room imports and explicit share consent.
+- [ ] Run Bun tests; expect pass. Commit Task 2.
+
+### Task 3: Embedded engine gateway and UI
+**Files:** frontend scripts/ethercalc-gateway; next config; spreadsheet pages/hooks/components; result-page buttons; constants; deployment configuration.
+**Interfaces:** same-origin `/sheets-engine/` proxy; internal permission check returns authorized room/write flag; bounded WS forwarding.
+- [ ] Test deny-by-default path parsing and source/viewer write denial, frame room confinement and origin/session checks.
+- [ ] Implement pinned runtime setup, persistent storage configuration, gateway, app-managed tabs, shared list/invites and completed-result entry points.
+- [ ] Run boundary tests and frontend build; expect pass. Commit Task 3.
+
+### Task 4: Snapshots and agent context
+**Files:** spreadsheet snapshot/range services, UI, tests; existing agent context entry.
+**Interfaces:** snapshot schema version 1 contains engine pin, ordered sheet saves, exact source JSON and timestamps; Kubo returns verified CID; restore creates fresh workbook/room IDs.
+- [ ] Test native multi-sheet roundtrip, formula/format preservation, malformed snapshot limits, provider failure/returned CID and bounded range parsing.
+- [ ] Implement explicit public consent, server Kubo upload/retrieval verification and new-workbook restore; range context delivered through existing agent flow without automatic job execution.
+- [ ] Run focused suite; expect pass. Commit Task 4.
+
+### Task 5: End-to-end verification and documentation
+**Files:** documentation feature/setup/demo/validation and README links.
+- [ ] Run project checks and real Chrome owner/collaborator/viewer walkthrough, restart/persistence, snapshot retrieval/restore; record exact evidence and blocked checks.
+- [ ] Complete setup, privacy, collaboration and demo documentation. Run whole-branch fresh review per execution skill, address important findings with regression tests.
+- [ ] Commit and push documentation feature branch; record commit and push result. Keep application branch unmerged.

superpowers/specs/2026-10-29-ethercalc.md

@@ -0,0 +1,24 @@
+# EtherCalc integration — accepted design
+
+Feature issue: https://github.com/QMeshPy/distributed-quantum/issues/103
+Implementation base: QMeshPy/distributed-quantum main 530121280.
+Design was approved before implementation; no new brainstorming approval is required.
+
+## Decisions and reasoning
+Embed current audreyt/EtherCalc inside QMesh navigation. Pin 11d8a49540634249094b2c92c3ee1a3aa3d3b4ca, verified October 29, 2026. Its bundled SocialCalc provides formulas and formatting; its standalone workerd service owns collaborative room state. Docker is an optional deployment wrapper, not a Cloudflare dependency. Persist /data; disable expiration and room indexes. This is service-coordinated collaboration, not browser P2P.
+
+One application workbook consists of an ordered collection of EtherCalc rooms (sheets), database membership, original run JSON and saved versions. Immutable source sheets and editable comparison/notes are separate. App-managed sheet tabs keep the complete room allowlist explicit, preventing native multi-sheet endpoints from bypassing workbook authorization. Reopen the existing workbook for the same owner/run/result hash; never reimport over edits. A new run/version creates a separate workbook.
+
+Use Better Auth identity, MongoDB membership, authenticated invitations and explicit accept/join. Stable URLs identify documents; they grant no access. A same-origin gateway checks HTTP and WebSocket access, denies unsupported routes and enforces viewers/source-sheet immutability. Direct EtherCalc is internal-only. Do not expose service HMAC credentials to browsers. Ownership of imported runs is checked server-side; pharma needs ownership added because its current routes are anonymous. Existing unowned runs must not be silently claimed.
+
+Adapters preserve pharma SMILES, scores, confidence intervals, ADMET, VQE and metadata; finance inputs/output/context and units; circuit counts/probabilities and provenance. Exact original JSON remains available, large nested values remain referenced in JSON rather than pretending lossy flattened cells are the scientific record. Generated/simulated candidates are not validated drugs.
+
+An editor explicitly adds their owned experiment and confirms sharing with all workbook members. Never modify source runs or share future runs implicitly. Agent context uses an explicit bounded range and existing conversation/budget controls. No automatic jobs on sheet open/recalculation, and no invented iframe selection API.
+
+Manual Save version to IPFS requires deliberate public-data consent. Use server-side persistent Kubo, upload actual manifest/native saves/original JSON, record returned CID and verify retrieval. Failed uploads remain failed. Restore all sheets into a NEW workbook, preserving the active one. A CID without successful retrieval is not a verified backup. Keep provider secrets server-side.
+
+## Deferred
+Browser libp2p replication, offline CRDT merge, automatic backups, arbitrary agent spreadsheet commands, unrelated wallet changes and additional provider integrations. No main merge or video publication/contact is authorized by this implementation.
+
+## Validation target
+Automated mapping, idempotency, membership/invitations, HTTP/WS boundary, native snapshot roundtrip and provider-failure tests. Real Chrome: authenticated embedded import/edit/reload, two independent sessions, viewer write rejection, deliberate collaborator run import, service restart, IPFS retrieval with browser-local storage unavailable and multi-sheet restore. Record blocked checks precisely. Demo October 31, 2026; November 1 buffer, Asia/Kolkata.

technical/ethercalc-progress.md

@@ -0,0 +1,47 @@
+# EtherCalc integration — draft progress, October 29, 2026
+
+Tracking issue: https://github.com/QMeshPy/distributed-quantum/issues/103
+Application branch: `feat/ethercalc-integration`, based on QMeshPy main `530121280`.
+Documentation branch: `feat/ethercalc-integration`.
+
+This is an unfinished implementation submitted for review at the user's request. It is not demo-ready or production-ready. Spreadsheet APIs and gateway require `SPREADSHEETS_ENABLED=true`; entry links require `NEXT_PUBLIC_SPREADSHEETS_ENABLED=true`. Both default off.
+
+## Implemented progress
+- Pharma submit/list/read/live/cancel ownership checks and frontend signed identity forwarding.
+- Native SocialCalc import preserving literal text, source JSON pointers/units, circuit bitstrings/counts/probabilities, and deterministic owner/run/version identity.
+- MongoDB workbook/member/invitation records, idempotent creation, explicit join, owned-run addition with sharing consent, failed-initial-import retry.
+- Same-origin gateway with room authorization, viewer/source write restrictions, server-side credentials, per-frame session checks, authorized cross-sheet recalculation and filtered outbound credentials.
+- Pinned upstream standalone workerd build recipe with workbook-isolated internal formula lookups. No Docker or Cloudflare account required.
+- Native snapshot manifest, public publication consent, Kubo upload and returned-CID retrieval verification, restoration into a fresh workbook and formula-only room-reference remapping.
+- Workbook pages, source/notes tabs, invitations, version controls, completed-run entry points and explicit bounded range context for copying into an existing agent conversation.
+
+The agent integration currently prepares context for manual copy; it does not provide an automatic selection bridge, integrated conversation panel or approved spreadsheet edits. Result mapping is primarily a typed path/value source table, not polished domain-specific comparison tables. These are remaining scope, not verified features.
+
+## Setup for continued development
+1. Install frontend dependencies with Bun, configure Better Auth/MongoDB and `AGENT_AUTH_BRIDGE_SECRET` consistently with the backend. Use an isolated demo/test database; no credentials belong in Git.
+2. Run `services/ethercalc/prepare.sh /absolute/new-build-directory`. The exact upstream revision is `11d8a49540634249094b2c92c3ee1a3aa3d3b4ca`. The checked patch restricts internal formula lookups to room names sharing the workbook prefix. Do not substitute unpatched upstream.
+3. Run standalone workerd using the prepared tree's `bin/workerd-entrypoint.sh`, with `APP_ROOT`, a persistent `ETHERCALC_DATA_DIR`, `ETHERCALC_HOST=127.0.0.1`, `ETHERCALC_PORT=8000` and a random 32+ character `ETHERCALC_KEY`. Keep expiration unset. Back up durable storage and MongoDB together.
+4. Set the same engine key and `ETHERCALC_INTERNAL_URL` in the frontend/gateway. Set `BETTER_AUTH_URL` to the public app origin, `SPREADSHEET_APP_INTERNAL_URL` to Next, and run `bun --env-file=.env.local scripts/ethercalc-gateway.mjs`. Next rewrites `/sheets-engine/*` to port 8002. This topology needs a long-running gateway; deployment alongside a Vercel frontend still needs an explicit reachable gateway topology.
+5. Set server-only `KUBO_RPC_URL` and optional `KUBO_RPC_SECRET` to the persistent existing Kubo service. Public snapshot consent covers all sheet and source JSON contents. Native room storage is primary persistence; IPFS is manual version backup.
+6. Enable both preview flags only in a controlled development environment until validation below is complete.
+
+Run the optional MongoDB integration suite explicitly with `RUN_SPREADSHEET_DB_TESTS=1 bun --env-file=.env.local test --timeout 30000 tests/spreadsheet-service.test.mjs` from frontend. It uses a randomly named temporary test database and removes only that database afterward.
+
+## Evidence and blockers
+- Pharma ownership/unit/integration suite: **66 passed**.
+- Pure spreadsheet import/access/snapshot regression suite: **15 passed**, including removal of outbound engine credentials, bounded sibling recalculation and formula-only restore remapping.
+- MongoDB service integration suite: **blocked**. Frontend configured local MongoDB refused connection; backend configured remote MongoDB SRV lookup returned ENOTFOUND, including an unrestricted retry. No claim of successful DB integration is made.
+- TypeScript: one error remains in existing `frontend/src/features/ipfs/lib/cid-resolve.ts:66`, involving incompatible transitive multiformats/Helia types. Full build is not claimed passing.
+- EtherCalc client, multi-client and static asset build succeeded. Worker bundling stopped on Wrangler cache/log filesystem permission; escalation was declined. No Cloudflare deployment occurred.
+- Real Chrome walkthrough, multi-session collaboration/viewer rejection, restart persistence, real Kubo retrieval/restore and end-to-end gateway behavior have **not been verified**.
+- Full repository lint/test/build suite has not been completed.
+
+## Review findings and remaining risk
+The fresh review found outbound HMAC exposure, rejected cross-sheet recalculation, unretryable failed imports and replacement of literal cells during restore. Fixes are included. Pure regressions pass; failed-import retry is in the DB integration test but remains unexecuted against a reachable MongoDB.
+
+Still validate mutation/snapshot synchronization under in-flight WebSocket commands, stale creating-state recovery after process crash, aggregate workbook/snapshot limits, source-run ownership across backend configurations, and native formula/format roundtrip on the actual engine. Existing pharma storage remains in-memory, so source-run availability across backend restarts remains limited even though workbook copies are intended to persist.
+
+## Demo walkthrough to validate before recording
+Use two independently authenticated accounts plus a viewer. Export one completed pharma run; verify exact source JSON and literal cells. Edit notes/formulas, reload, and reopen the same run without overwrite. Invite the other account, accept in Shared with me, and demonstrate actual service-coordinated edits. Add the collaborator's owned finance/circuit run with explicit consent. Verify viewer writes and unauthorized direct UI/API/export/WS requests fail. Restart the engine and reopen. Publish a non-sensitive workbook manually, verify retrieved bytes without browser-local storage, restore into a new workbook, and compare all native sheets/formulas/source JSON. Read an explicit range and review its agent context. Record actual results before claiming completion.
+
+Demo target remains Saturday October 31, with Sunday November 1 buffer (Asia/Kolkata). No video publication or external contact has occurred.