Skip to content

Preview an import (spec 025) - #113

Merged
plainprogrammer merged 20 commits into
mainfrom
025-import-preview
Oct 15, 2026
Merged

plainprogrammer merged 20 commits into
mainfrom
025-import-preview

Conversation

@plainprogrammer

Copy link
Copy Markdown
Owner

Closes #39.

A collector uploads an export file at /collection/import_preview/new (account menu and More, after "Export collection") and sees what applying it would do. Nothing is written. The page shows lots added, changed, unchanged, held by a problem row, and removed, with their copies, plus profile notes. Each row that can't be applied gets its own problem with a rule and a place. A file that can't be read at all is refused with a 422, giving the reason. Applying is #40; this preview exposes each row's outcome and the held lot it names, so #40 isn't blocked.

  • Spec: docs/specs/025-import-preview/spec.md (v1.5.0, approved)
  • Plan: docs/specs/025-import-preview/plan.md
  • ADR: 0021, json_schemer and an in-app copy of the schema
  • Verification: docs/specs/025-import-preview/verification.md, which includes 12 screenshots in screenshots/

Maintainer rulings (2026-10-15)

  • Replace semantics. A problem row blocks applying. A file in another currency is refused whole. Applying never changes the collector's name or email; it takes collection_view (decisions 1–4).

  • Where the rules live. json_schemer is in the app bundle with a copy of the schema kept byte-identical, and the rules moved to ExportFormat::Rules (decision 5, ADR 0021).

  • The upload. It is read in the request and never stored. It is judged by its content and refused over 32 MB. Request bodies over 40 MB are refused by Puma, Thruster (MAX_REQUEST_BODY) and kamal-proxy (max_request_body) (decision 6).

  • Lists. Each shows its first 100 entries, then "and N more" (decision 11).

  • Performance bars. A 50,000-row preview must finish in under 30 s (decision 12) and grow memory by under 500 MB (decision 13). The first measurement missed the memory bar at about 1 GB. The maintainer ruled the plan's fallback: check the header with lots: [], then each lot against $defs/lot (f8ca320, 861f076, ADR 0021 amended). Re-measured against bin/dev with a fresh server for each file:

    File Time Memory growth
    Resolvable 10.6 s +316 MB
    Unresolvable 8.3 s +139 MB
    Bad price 8.6 s +120 MB
  • Spec 022 isn't edited. Spec 025 records that it supersedes spec 022 decision 6's menu order, and docs/specs/README.md records it in both directions (decision 14).

Checks

  • bin/ci at 861f076, the last commit that changes code, passed: 1583 examples, 0 failures, 1 pending (pre-existing), seed 22429. Later commits change documents only, plus a merge of origin/main, which was also documents only.
  • A 41 MB POST gets a plain 413 from Puma (AC-3.4, run by hand).
  • The implementation review (Mode B, Opus, read-only) found the code SPEC-ALIGNED, with no Critical findings.

Deviations from the plan

All are listed in verification.md:

  • Phase 2: Row#text is parenthesised, and read_rows leaves duplicate_lot pointers out of its failed set.
  • Phase 3: the AC-1.2 request example builds its held lots in a loop, to stay within RuboCop's 15-line limit. The AC-1.8 example builds the file before taking its snapshot.
  • Phase 6: the per-lot schema check described above.

Follow-ups (from the review, not fixed here)

  • Problem counts aren't capped. The list of problem counts has one entry per distinct bad value, such as each unknown finish, so it can grow long. The spec asks for this as written, so changing it needs a spec change.
  • Prices too large to store. A price_paid too large for a 64-bit count of cents previews as applicable. Adding a row problem for it belongs with Apply an import #40.
  • Plan retirement. The plan is owed retirement after this PR merges (distill-feature). Spec 025's areas, data portability and release and deploy, have no area doc yet; Area doc: data portability #108 and Area doc: release and deploy #109 will write them.

🤖 Generated with Claude Code

plainprogrammer and others added 20 commits October 15, 2026 10:58
Spec 025 (issue #39): a collector uploads an export file and sees what
applying it would do, with every row that can't be applied, writing
nothing. Reviewed read-only on Opus and on Fable.

Rulings (maintainer, 2026-10-15): decisions 1-7 as proposed (replace,
a problem row blocks applying, another currency is refused, only the
collection view is applied, json_schemer and the rules move into the
app, the upload is never stored, Import collection beside Export
collection), decision 11 (lists show their first 100 rows) and
decision 12 (50,000 rows in under 30 s).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Plan 025 for previewing an import, reviewed read-only on Opus with its
findings applied. ADR 0021 records where the format's rules live in the
app: json_schemer and a byte-identical copy of the schema.

Rulings (maintainer, 2026-10-15, approving the plan): the memory bar is
500 MB for a 50,000-row preview (spec decision 13); spec 022 isn't
edited, and spec 025 records that Import collection supersedes its
decision 6's menu order (spec decision 14). Spec 1.5.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ExportFormat::Rules applies v1's schema, through an in-app copy kept byte-identical to docs/, and the five further rules; the suite's ExportFormatRules passes well-formedness and lot identity through to it (ADR 0021).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CollectionImport refuses a file as a whole (decision 10) or gives each row's problems (decision 9) and what applying would do: added, changed, unchanged, held by a problem row, removed (decision 1). Nothing is written.

Deviation: Row#text is written `def text(value) = (value if value.is_a?(String))` — without the parentheses Ruby parses the modifier if as guarding the endless def itself, so loading the class raised NameError (undefined local variable 'value').

Deviation: read_rows leaves duplicate_lot out of the failed pointers it passes to identity_readable? — duplicate_lot is reported at /lots/<i> itself, so counting it made a duplicate row's identity unreadable, and its reason ("as row %d") got a nil row number (TypeError). ExportFormat::Rules finds duplicates with only the schema's pointers failed; the reader now judges identity the same way.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Deviation: the AC-1.2 example creates its three held lots in one loop instead of three create lines — RSpec/ExampleLength (16/15); same lots, same assertions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GET /collection/import_preview/new uploads an export file; POST /collection/import_preview shows what applying it would do, or why it is refused (422). Import collection follows Export collection in the account menu and on More (decision 7).

Deviation: the AC-1.8 example builds its file before taking the snapshot — the plan's block called `printing` inside the measured block, so the spec's own setup (a catalog entry, identity, set and printing) showed up as writes; the assertion is unchanged, and a mutation that touches a lot in the controller still fails it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Puma's http_content_length_limit, Thruster's MAX_REQUEST_BODY and kamal-proxy's buffering.max_request_body (decision 6; the maintainer approved the Thruster and kamal-proxy settings on 2026-10-15, spec 1.4.0).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pins the maintainer's ruling of 2026-10-15 (spec 025 decision 13): the
per-lot pass gives the same problems, in the same order, as a reference
whole-file pass on every example and on crafted variants; a lot checked
against $defs/lot gives the whole schema's errors once prefixed; and the
root schema never sees a lot.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ExportFormat::Rules validated the whole file in one json_schemer call,
which holds a result tree for every lot. It now validates the file with
lots: [] and then each lot against $defs/lot (ExportFormat::Rules.lot_schemer),
prefixing pointers with /lots/<i> and mapping each lot's errors before the
next. Root-level errors (a missing key) follow the lots' errors, as in the
whole-file pass, so problems come in the same order. Data that isn't an
object, or whose lots aren't an array, is still validated whole. ADR 0021
is amended to say so.

Ruling (maintainer, 2026-10-15): Option A — header validated with lots: [], then each lot against $defs/lot with pointers prefixed /lots/<i>; whole-file pass grew RSS ~1 GB on 50,000 lots against a 500 MB bar.

Runner check (indication only): problems_in on the 50,000-lot resolvable
file grows RSS 203 -> 269 MB in 8.1 s; bad-price file 203 -> 273 MB in 8.7 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each acceptance criterion with its evidence, AC-1.10's two measurements (the
whole-file pass missing 500 MB, the ruled per-lot pass meeting both bars),
AC-3.4 by hand, the screenshots, the deviations from the plan and the
documents read. The final bin/ci line is left to fill in.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Records both directions of decision 14: spec 025 supersedes spec 022
decision 6's menu order (docs/specs/README.md, .claude/rules/documentation.md).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Mode B (Opus, read-only): SPEC-ALIGNED. Fills in the bin/ci line, the
commit list and the diff base after the merge of origin/main, and lists
the review's minor findings as follow-ups.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Data portability and release and deploy have no area doc, so spec 025 is
listed among the specs each covers (#108, #109 distil them). Its index
row's plan reads retirement owed; verification.md records the step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@plainprogrammer plainprogrammer self-assigned this Oct 15, 2026
@plainprogrammer
plainprogrammer marked this pull request as ready for review October 15, 2026 23:33
@plainprogrammer
plainprogrammer merged commit ec938d1 into main Oct 15, 2026
4 checks passed
@plainprogrammer
plainprogrammer deleted the 025-import-preview branch October 15, 2026 23:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Preview an import

1 participant