Scriptorium

Turn scanned documents into a verified, readable, chapter-split corpus that both people and language models can cite.

Point it at a directory of PDFs. It reads every page with two independent OCR engines, records every place they disagree instead of picking one, splits the result into chapters and numbered passages, and emits Markdown plus a full-text index where every passage carries its page number and a confidence. Runs entirely on your machine: no API key, nothing sent anywhere.

Why two engines. Tesseract read a stacked fraction as 32. The true value is 11/20, confirmed by the page's own arithmetic. 32 is a perfectly well-formed number, so no dictionary, spellchecker or language model can object to it — only a second engine reading the same marks differently can. The second engine has to be architecturally independent; another Tesseract wrapper would repeat Tesseract's mistakes and agree, which is worse than one engine because it manufactures confidence.

Install

git clone https://github.com/lohiarahul/scriptorium.git
cd scriptorium
scripts/setup.sh

Needs Python 3.12, plus poppler and tesseract on the system. scripts/setup.sh checks for both and tells you what's missing; scriptorium doctor catches anything it couldn't, including a missing language pack.

Run the app

From the repository root:

make web       # once: build the browser bundle (needs Node)
make serve     # http://127.0.0.1:8765

There is no collection open at the root, which is a state the app starts in rather than an error — it lists what it found under collections/ and you open one from the start screen. To open one directly:

.venv/bin/scriptorium serve collections/mycorpus

Run the pipeline

scripts/new-collection.sh mycorpus
cd mycorpus && cp ~/scans/*.pdf sources/

scriptorium add sources/mybook.pdf    # probe the PDF, draft its config
scriptorium run                       # everything, with the gates enforced
open corpus/mybook/full.md

Everything is resumable per (document, page, stage); an interruption costs only the page in flight. No configuration is required — add drafts it by measuring the PDF, and leaves a field null rather than guessing.

What you get

corpus/<slug>/
├── full.md            the whole document, readable, chapters and units
├── full.txt           same text without markup, for grep and diffing
├── chapters/NN-*.md   one file per chapter
├── INDEX.md           chapter → file → page range → unit numbers → confidence
├── verses.jsonl       one record per unit, with provenance and confidence
├── provenance.jsonl   every unit mapped back to its source page and spans
└── document.json      manifest: coverage, chapter reliability, how to cite it

full.md opens with the two numbers that say how far it can be trusted: what fraction of the body text reached a unit, and whether the chapter split is credible. Where it is not, the file says so in a banner — a failed chapter split leaves every character present, so nothing looks wrong until you try to navigate.

Collection-wide there is an SQLite FTS5 index, and every search hit carries its citation, its confidence and its flags:

scriptorium search 'NEAR(alpha beta, 5)'

mybook §14 14, printed p.144  [low_confidence]
    conf 0.72  … the <<alpha>> of <<beta>> as described above, one will …

Documentation

docs/map/	How this repository is built. Short files, one diagram each — start here to change the code
docs/	How the pipeline works. A numbered manual, read in order

Two levels, deliberately separate. Level 1 is raw page-faithful extraction, every engine's output verbatim and append-only. Level 2 is derived structure, rebuilt from Level 1 alone. So a structural bug is never a reason to re-OCR. That is enforced, not just documented: tests/test_levels.py breaks subprocess and socket, deletes Level 2, rebuilds it, and asserts the result is byte-identical.

What is not claimed

Cross-engine agreement cannot catch correlated errors, where both engines misread the same degraded mark identically. That is the irreducible floor, and those spans carry both variants and a confidence rather than a silent correction. A document with no second engine is reported uncertifiable rather than passing: zero conflicts because nothing was compared is not zero conflicts.

Chapter detection falls short on 3 of the 10 reference documents; two cannot be cited to a printed page; there is no ground truth anywhere in the corpus; and the three-engine path has never been run against a real third engine.

Every figure in this repository was measured on a reference collection of ten scanned documents — 4,430 pages, 5,462 units, 231 chapters. That collection is not distributed and no part of it is in git, so the measurements are reported and cannot be re-derived from a clone; documents are named document A … document J. The tool ships no collection, no domain vocabulary and no subject-specific configuration.

Development

make check     # lint, format, types, tests, links, vocabulary — everything that must pass

All of it must pass. CONTRIBUTING.md has the rules that are not negotiable; the short version is that a comment here records a measurement, not an intention — name the page you measured on, and quote the output verbatim, errors included.

Also CODE_OF_CONDUCT.md and SECURITY.md.

Licence

MIT. This covers the software, not the documents in any collection: scanned sources are the copyright of their respective publishers and translators, and none are distributed with this repository.