Skip to content

docs(export-format): define the versioned export format (spec 017) - #68

Merged
plainprogrammer merged 20 commits into
mainfrom
017-export-format
Oct 12, 2026
Merged

plainprogrammer merged 20 commits into
mainfrom
017-export-format

Conversation

@plainprogrammer

@plainprogrammer plainprogrammer commented Oct 12, 2026 •

Copy link
Copy Markdown
Owner

Defines the format a collector's data is exported in, so that export (#38), import preview (#39) and import apply (#40) are built against one definition. It is an enabling slice: a format document, a schema, examples and an ADR. No page, job, setting or application code is added.

Closes #37

Why

Foundation principle 2 says collectors own their data in open, documented formats, and the 1.0 test's second check ("an export of that instance has been restored into a fresh instance and matched") had no definition of "matched". This PR gives it one.

What the format is

  • One JSON document per account, UTF-8, with a header (format, format_version, exported_at, optional generator), the collector's profile, one currency and the lots.
  • Keyed by external ids. A lot's entry is identified by collectible_type and source_id (for Magic, the Scryfall card id). No internal primary key appears anywhere. The name, set code, collector number and language ride along as information only, so a person can read the file without a catalog.
  • A lot carries its quantity, finish, condition, price paid per copy (a two-place decimal string) and when it was added and last changed. Finish, condition and price are always present, and null when not recorded.
  • Well-formed and resolvable are kept apart. Whether a file is well-formed never depends on a catalog; whether its entries exist is a question about one instance.
  • "The same collection" is defined by the profile, the currency and the lots as sets. That is what "matched" means.
  • format_version is a whole number, 1 here. It rises when an older reader would misread a file or lose the collector's data from it, so the spec that puts a lot in a location (Put a lot in a location #45) will make version 2.

What changed

  • docs/export-format/README.md: the format document.
  • docs/export-format/v1.schema.json: a JSON Schema (draft 2020-12).
  • docs/export-format/examples/: 33 example files (valid, invalid, and pairs for the comparison rule) and problems.json, which names why each invalid file is invalid.
  • spec/export_format_examples_spec.rb, spec/export_format_document_spec.rb, spec/support/export_format_rules.rb: 27 examples that check the document, the schema, the examples and the application against each other.
  • docs/adr/0020-export-format.md: the decision and the alternatives rejected (CSV, an id-only reference, descriptive fields as a fallback match, whole-number minor units for price).
  • Gemfile: json_schemer in the :test group, require: false. The published image installs the test group already, so the gem is present there and never loaded.
  • README.md, docs/adr/README.md, docs/roadmap.md (item 2 done, decision log 32), CLAUDE.md.

The guard for "export covers it"

The format document has a coverage table of every database table that belongs to an account or a user: exported, left out, or nothing to export, with each column of an exported table accounted for. The suite compares it with the database, so a later spec that adds a table with account_id or user_id, or a column to accounts, lots or users, fails until the document says what the format does about it. It compares names; it can't tell whether a mapping is right.

Verification

  • bin/ci passed locally on 37e7778: 1146 examples, 0 failures, 1 pending (pending before this branch too; the baseline was 1119). The head commit adds only the verification record after that run. The pre-push hook ran bin/ci again on the head commit and let the push through.
  • Each phase was written test first; its red and green counts are in the verification record.
  • A read-only review of the implementation against the spec found all 20 acceptance criteria met: 13 by passing examples and 7 by reading the format document.

Spec: spec.md · Plan: plan.md · Verification: verification.md · Format: docs/export-format/README.md · ADR 0020

There are no screenshots: nothing a collector or operator can see changed.

Worth a look

  • The field names are frozen once this merges. The meaning of version 1 never changes after release.
  • The spec moved after approval, to 1.2.1. Version 1.2.0 records six additions the maintainer approved with the plan (a repeated JSON key and a time that is no real moment are not well-formed, no byte order mark, collection_view is required, accounts is in the coverage table, generator may name any program). Version 1.2.1 fixes four sentences that were left stale.
  • The rules module is test support. Import (Preview an import #39) decides where a copy inside the app lives, and must look at format_version before it validates: a file of a higher version fails version 1's schema, and is "newer", not malformed.
  • Sibling branches (specs 018 and 019) may also touch CLAUDE.md and the roadmap's decision log. ADR 0020 and log row 32 were free on main when this was pushed.

Owed from the maintainer

  • Read docs/export-format/README.md as rendered on GitHub, including its relative links. That rendering is unchecked.
  • Confirm the GitHub checks pass (the ci job and both image builds with their smoke test). They run only on a pull request.
  • Session wrap for memories and lessons, before this is marked ready.
  • At merge, re-check ADR 0020 and decision-log row 32 against whichever sibling branch merged first.

No phone or in-person check is needed for this spec.

🤖 Generated with Claude Code

@plainprogrammer
plainprogrammer marked this pull request as ready for review October 12, 2026 14:16
@plainprogrammer plainprogrammer self-assigned this Oct 12, 2026
plainprogrammer and others added 20 commits October 12, 2026 15:49
Outline agreed by the maintainer on 2026-10-11: one JSON document, entries
by external id with descriptive fields, profile and currency and lot times
in the file, and a schema with examples checked in CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fable review (Mode A) returned NEEDS REVISION. Finish is the printing's own
list, not a per-type vocabulary; not recorded is null; price paid is per
copy with one written form; times have one written form; the coverage guard
gets a unit (tables that belong to an account or user). Proposes that a new
collector-owned field raises the format version, pending the maintainer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The maintainer ruled, through the coordinator, that a new field holding
collector-owned data raises format_version; only information-only fields
are free (decision 14, FR-8).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Approved by the maintainer as it stands, relayed by the coordinator; the
former open questions become decisions 15 to 19.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Five phases: the plan's additions into the spec, then schema, rules and
examples, the format document, ADR 0020 with the pointers to it, and
verification. Every code block was run against scratch copies before it
was written down (26 examples green; red states as stated).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fable plan review: READY TO EXECUTE, every block run and every count
reproduced. A higher version is no longer called malformed, generator is
any string, and two unlisted additions (no byte order mark, a required
collection view) are now listed for the maintainer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Approved in full by the maintainer, relayed by the coordinator: the field
names, the six additions beyond spec v1.1.2, the test-only json_schemer gem
and the format living under docs/.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Phase 0: Record the plan's additions in the spec. Spec 1.2.0: a repeated
key and a time that is no real moment are not well-formed, accounts join
the coverage table, the writer's name may be any program's, no byte order
mark, and the collection view is required (decision 20).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Phase 1: Schema, rules and examples. Checks the export format's examples
against its JSON Schema (spec 017); require: false, never loaded by the
running app.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Phase 1: Schema, rules and examples. The JSON Schema, the rules a schema
can't state, FR-7's comparison, and 32 example files checked against them
(spec 017 AC-1.3, AC-1.4, AC-2.1, AC-2.2, AC-2.3, AC-2.5).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Phase 2: The format document. The prose definition of version 1, checked
against the schema, the examples and the application: field names, the
version, each type's conditions, the currencies, and a coverage table of
every table that belongs to an account or a user (spec 017 AC-1.1, AC-3.4,
AC-3.5, AC-4.1, AC-4.2, AC-4.3).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Phase 3: ADR 0020 and the pointers to the format. The decision and the
alternatives rejected, the ADR index entry, and a README section that
links the format document (spec 017 AC-4.4).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Phase 3: ADR 0020 and the pointers to the format. Roadmap item 2 is done,
item 10 notes the version it raises, decision log 32 records the ruling,
and CLAUDE.md says where the format lives and what guards it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Phase 4: Integration Verification. bin/ci green on e663f80 (1146 examples,
0 failures, 1 pending), the phases' red and green states, and the seven
criteria checked by reading the format document.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Spec 1.2.1, after the implementation review (Fable, Mode B, SPEC-ALIGNED).
No requirement changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…cases

Review follow-up. Ruling: one invalid example beyond the plan's 32
(invalid/byte-order-mark.json), two malformed variants for a missing or
blank collection view, and an unknown condition as a well-formed edge —
the spec's Error Scenarios name each and nothing asserted them — no rule
or meaning changes; if wrong, the example and three table rows come out.
The document's rule table says not_json covers a byte order mark, and its
currency section says the version doesn't change with issue 64.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bin/ci green on 37e7778: 1146 examples, 0 failures, 1 pending. The
implementation review (Fable, Mode B) returned SPEC-ALIGNED; its follow-ups
and what was left as it is are listed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
New: orca-supervised-stages, spec-017-followups. Updated:
sdd-execute-exact-plans (a scratch-drafted plan without a prototype
branch) and precommit-hook-staging (the pre-push hook shows output only on
failure). Approved by the maintainer through the coordinator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A plan that adds to its spec lists every addition; ask where a vocabulary
lives before specifying it; ignoring unknown fields needs a rule for new
ones. Approved by the maintainer through the coordinator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@plainprogrammer
plainprogrammer merged commit 3ab9864 into main Oct 12, 2026
4 checks passed
@plainprogrammer
plainprogrammer deleted the 017-export-format branch October 12, 2026 22:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Define the export format

1 participant