extract

Reads a PDF page back: the text on it, with where each piece of it sits, and the images it places.

text, err := extract.Text(doc, 1)     // the page as words, lines put back together
runs, err := extract.Runs(doc, 1)     // every piece, with its place and its size
imgs, err := extract.Images(doc, 1)   // every picture, and where it lands
tabs, err := extract.Tables(doc, 1)   // the grids its text lines up into

lines := extract.Lines(runs)          // the runs gathered into their lines
s     := extract.Join(lines[0])       // one line's pieces, spaced as words

Lines and Join are exported because the rule has to be the same rule: Text uses them to put a page's text back together, and anything reading structure out of a page — where a paragraph ends, which line is a heading, as document does — has to agree with it line for line, or the two disagree about what the page says. A test asserts that agreement rather than trusting it.

Pure Go, no C. It builds for GOOS=js/wasm and every architecture the fleet targets.

What a page actually holds

Not text. A page holds instructions for drawing glyphs, and the text is what those glyphs were meant to say — which the document says only if it was asked to. Where it does not, this says so rather than guessing: a run whose codes could not be worked out comes back marked Unreadable, so a caller can tell this page says nothing from this page could not be read.

Three things are tried, in order:

1. the font's /ToUnicode map, which is the document's own word on what its text says;
2. the name its encoding gives the code — but only a name the document chose itself, since a symbolic font read through an assumed encoding gives the wrong letter with nothing to say so;
3. what the embedded font program calls the code, read with go-opentype — which is the honest answer for a symbolic font, and the reason a page of mathematics comes back as mathematics.

Text drawn in the mode that puts no ink on the page — how a scanner writes what it read underneath the picture it read it from — is returned and marked Invisible.

Where a run came from, so a page can be edited

Run.Op is the byte range of the operation that drew the run, in the content stream it was scanned from. It is there so a caller can change a page and not only read it: a redaction has to cut the glyphs inside a region out of the stream, and which glyphs are inside is precisely what this package already works out — the text matrix, Td/TD/Tm/T*, Tc/Tw/Tz, the font metrics. The only thing missing was where to cut.

The alternative was for every caller to pair runs with operators by counting them, which does not survive a form, invisible text, or an operator that draws nothing — and gets the cut wrong in a way that no test catches, because both sides of the comparison move together.

Two things it says that a caller must act on:

• ⛔ Several runs can share one span. A TJ array shows several strings with a single operator, so being rid of one piece means rewriting the span, not deleting it. Two runs with equal Op are two pieces of one operation, and deleting the span would take the other away too.
• ⛔ InForm means the span is not the page's. A run drawn from inside a Form XObject has its offsets in that stream. Cutting the page there would cut somewhere else entirely, and an XObject is shared, so editing it to change one page changes every page that draws it. A caller editing a page must refuse such a run rather than guess — the shape ops.Doc.RecolourText already has, where UnreachedText counts what it could not reach.

The span covers the operands as well as the operator, so removing one is one cut, and the text that stays does not move:

runs, _ := extract.Runs(doc, 1)
cut := content[:runs[1].Op.From] + content[runs[1].Op.To:]
// the page still draws runs[0], in the same place, and no longer draws runs[1]

Images

A picture comes back as the file holds it, with the box it covers on the page. A JPEG comes out as a JPEG: re-encoding it would lose something for nothing. Turning the other kinds into pixels means reading their colour space, which is render's work rather than this package's.

⛔ Matrix is the transform it was drawn with, and the box is not enough for every caller. The box a turned picture covers looks exactly like the box an upright one covers, so a caller that must map a region of the page into the picture's own samples — a redaction destroying the pixels under it, which ISO 32000-1 §12.5.6.23 requires rather than a mask over them — would destroy more of a turned picture than was asked for, and removing too much is its own betrayal. b and c are zero for an upright picture (a flip keeps them zero), so the matrix is also how such a caller refuses the ones it cannot do by name. For a picture inside a form it is the whole chain: the page's transform, the form's /Matrix, and the one in force where the Do is.

Tables

A page holds no table. It holds text put at places, and a table is what a reader sees when those places line up — so Tables reports an inference, and there is nothing in the file to check it against.

A column boundary is a stretch of the page that every row of the table leaves empty. The first line of a block proposes its own gaps, each line after it takes away whatever it draws in them, and a line that leaves nothing closes the block and begins the next. That is the one decision everything else follows from, and it is subtraction rather than the intersection of the lines' gaps, which is where this started:

a cell whose text wraps	leaves a line with one piece of text on it and so no gaps of its own. Intersecting with nothing ends the table, and a four-row table with one wrapped cell came back as two tables of two rows. A wrapped line draws nothing in the boundary beside it, so subtracting keeps it.

Three things are not columns, and each one was found by running this over real documents rather than over a fixture:

a bullet	a bulleted list is a two-column grid. What tells them apart is that one column says the same thing wherever it says anything — and a column like that distinguishes nothing. A mark of one or two characters, repeated, is dropped; CNRS repeated down a column is data and is kept.
a column nothing was drawn in	that is a stretch of page nothing was drawn in, which is what a boundary already is. A run of nothing but spaces splits a boundary in two and leaves an empty column between the halves.
a line far below the row above	a page number sits in the left-hand column and draws nothing in any boundary, which is exactly what a row of a table does. What says it is not one is the distance: more than three times the row's own text size.

Measured on one 213-page document, as each of those went in:

588 tables, 1841 rows   gaps the rows share
213 tables, 1631 rows   + boundaries the rows leave empty (wrapped cells)
 83 tables,  268 rows    + a repeated mark is a bullet
 40 tables,  120 rows     + an empty column is not a column, and a page
                            number is not a row

The forty are the document's Code / Intitulé pairs, one per job description, with their second cells wrapped. A 3-page prose report in the same run reports none, which is the control: a detector that cannot say no to a paragraph says yes to every page there is.

⛔ What it still gets wrong. A page laid out in two columns of prose is a grid too — the gutter is a stretch every line leaves empty. What saves the ordinary case is that prose does not stay out of its gutter for long, since one full-width line closes the block. And ruled lines are not read at all: this is the text strategy, the one a bordered table does not need, and a table whose columns are held apart by rules rather than by whitespace is not found by it.

How it is checked

100% of statements, including every branch that handles a page saying something it should not.

Then on the corpus — 118 833 real PDFs, 121 946 pages:

pages read         121946
  with text        94865 (77.8%)
  saying nothing   27081, of which 4812 did draw glyphs
characters         33976414 (17272226 of them letters)
runs               10797045
  unreadable       203607 (1.89%)
  invisible        29917
images             300685
panics             0

The 22% of pages that say nothing are figures — this corpus is arXiv's, and most of it is plots. Of the pages that did draw glyphs and still said nothing, nearly all are Type 3 fonts with no /ToUnicode: dvips bitmap fonts whose glyphs are called a1 and s32, which no reader can turn back into letters.

The rest of the family

reader reads and writes the format · ops is the verbs and the pdfops command · render turns a page into pixels · pdffont is what a document says about a font · extract reads a page back · coedit is a PDF several people edit at once · app is the browser workbench.

Documentation for all of it: https://go-pdfkit.github.io/docs/

Licence

BSD-3-Clause.