ops

The verbs of go-pdfkit: what people actually do to a PDF they already have. Merge two files, pull out pages three to seven, turn a page the right way up, drop the metadata, split a report into chapters.

Built on reader, which takes a file apart and puts one back together; nothing outside the Go standard library is used, so this builds for GOOS=js/wasm and every architecture the fleet targets.

The model

A document here is an ordered list of pages, each borrowed from a source file, plus the document-level pieces. Every operation rearranges that list or annotates its entries; nothing is applied until the document is written out. An operation therefore costs nothing until it has to, several files can be mixed freely, and — the reason the model is shaped this way — the list is a plain value that two people can edit at the same time.

What it does

go install github.com/go-pdfkit/ops/cmd/pdfops@latest

pdfops merge whole.pdf part1.pdf part2.pdf
pdfops select -pages 3-7,last report.pdf extract.pdf
pdfops delete -pages even scan.pdf fronts.pdf
pdfops rotate -pages all -by 90 sideways.pdf upright.pdf
pdfops crop -box 20,20,575,820 wide.pdf trimmed.pdf
pdfops resize -to a4 odd-sizes.pdf a4.pdf
pdfops resize -to a4 -landscape slides.pdf wide.pdf
pdfops move -from 1 -to 3 misplaced.pdf ordered.pdf
pdfops split -every 10 book.pdf chapters/
pdfops reverse back-to-front.pdf right-way-round.pdf
pdfops nup -n 4 slides.pdf handout.pdf
pdfops booklet chapter.pdf to-fold.pdf
pdfops interleave both.pdf fronts.pdf backs.pdf
pdfops duplex scan.pdf document.pdf
pdfops onepage strip.pdf one-long-page.pdf
pdfops poster -across 2 -down 2 plan.pdf tiles.pdf
pdfops divide -across 2 spread.pdf pages.pdf
pdfops overlay -with letterhead.pdf plain.pdf headed.pdf
pdfops underlay -with watermark.pdf report.pdf behind.pdf
pdfops blank -before 3 report.pdf report-with-a-gap.pdf
pdfops watermark -text CONFIDENTIAL contract.pdf marked.pdf
pdfops number -format "{page} of {pages}" report.pdf numbered.pdf
pdfops bates -prefix ACME- -start 1 exhibits.pdf stamped.pdf
pdfops stamp -text "seen 25 August" -at top-right -bold notes.pdf seen.pdf
pdfops place -image signature.png -at bottom-right -width 120 contract.pdf signed.pdf
pdfops place -image logo.png -x 40 -y 740 -dpi 300 -opacity 0.3 plain.pdf headed.pdf
pdfops sanitize downloaded.pdf safe.pdf
pdfops flatten filled-form.pdf final.pdf
pdfops strip -annotations -bookmarks private.pdf clean.pdf
pdfops compress fat.pdf lean.pdf
pdfops encrypt -user letmein -allow print,copy plain.pdf locked.pdf
pdfops -password letmein decrypt locked.pdf plain.pdf
pdfops -password letmein permissions locked.pdf
pdfops text paper.pdf
pdfops text -layout -pages 1 paper.pdf
pdfops text -layout -json paper.pdf
pdfops tables accounts.pdf
pdfops tables -as md -pages 3 accounts.pdf
pdfops tables -as json -to tables/ accounts.pdf
pdfops markdown report.pdf > report.md
pdfops markdown -json report.pdf > report-for-indexing.json
pdfops word report.pdf report.docx
pdfops excel accounts.pdf accounts.xlsx
pdfops objects broken.pdf | jq .
pdfops images paper.pdf pictures/
pdfops links report.pdf
pdfops links -pages 1-3 report.pdf
pdfops attachments report.pdf
pdfops attachments -to files/ report.pdf
pdfops attach -file figures.csv -description "the figures" report.pdf with.pdf
pdfops detach -name figures.csv with.pdf without.pdf
pdfops outline -from toc.txt book.pdf bookmarked.pdf
pdfops outline -drop bookmarked.pdf plain.pdf
pdfops contents -from toc.txt -bookmarks book.pdf with-contents.pdf
pdfops info file.pdf
pdfops metadata -set 'Title=The Report' -set Author=Someone draft.pdf titled.pdf
pdfops metadata -clear draft.pdf anonymous.pdf
pdfops version -set 1.7 old.pdf declared.pdf

A page range is written 1-3,7,10- and may say all, even, odd or last. It keeps its own order and its own repeats, so select -pages 3-1 reverses three pages and select -pages 1,1 gives you two copies.

-password opens an encrypted file.

links lists what the document's pages link to — a page of the same document, or an address — with the rectangle each link occupies, in the document's own page numbering. Reorder the pages or drop one and the list says where the links lead now, not where they led in the file they came from.

1    72,648,232,660    page 14
1    72,600,180,612    https://example.org/
3    90,120,300,132    nowhere

(the columns are tab-separated)

⛔ A link that leads nowhere this can follow is listed, and says so. It is a link all the same: a page the document no longer has, a jump into another file, a destination name nothing defines, or an action that runs something instead of going somewhere. Printing only the ones that work would answer "eleven" about a file holding twelve, and a listing that under-reports is worse than none — whoever is auditing a document for what it points at would never learn that the twelfth exists.

⛔ /A is read before /Dest. The link-annotation clause (ISO 32000-1 §12.5.6.5, Table 173) does not permit /Dest beside an /A, so a file carrying both is malformed — and the action is what a reader acts on, so it is what this reports. A rectangle written from either pair of corners is normalised, because a caller hit-testing a back-to-front one finds nothing at all, which looks exactly like a page with no links on it.

interleave takes one page from each file in turn, which is how two single-sided scans — the fronts and the backs — become one double-sided document. onepage stacks every page onto a single long one. poster cuts each page into tiles so a plan can be printed larger than the paper and assembled.

deblank finds the pages that hold nothing and drops them. It draws each page and counts the pixels that are not paper, because that is the only test that agrees with a reader: a page whose content stream is full of operators painting white, or drawing outside the media box, looks blank and is blank, and a page carrying a picture of nothing is blank too.

pdfops deblank -list scan.pdf          # say which, write nothing
pdfops deblank scan.pdf tidied.pdf

⛔ The default resolution is 150 DPI, and it was raised from 50 because of a measurement. On A4:

	50 DPI	150 DPI
the page	414 × 585	1240 × 1755
one pixel	4.1e-06 of it	4.6e-07
an 8-point full stop	1 pixel	4 pixels
an 8-point comma	2 pixels	8 pixels

At 50 DPI a full stop is exactly the mark a speck of dust makes, so no threshold at that resolution can tell punctuation from dirt. At 150 there is a factor of four to set -max-ink from, and TestMeasureTheInkLadder prints the whole ladder so that a renderer change which moved either end fails there instead of silently reclassifying somebody's pages.

⛔ The zero value is the strictest setting, not a middle one: a page with a single inked pixel is kept. This is the input to deleting pages — a page wrongly called blank destroys content, while a blank page wrongly kept merely leaves a blank page. For scans, -max-ink 1e-6 is about two pixels: it forgives a speck and still keeps a page holding a full stop. It is not the default because whether that margin holds depends on the scanner, which this package cannot see.

Three more things it will not do:

• it will not call a page of solid colour blank. The test is distance from white, not uniformity; a measure of variance would delete a deliberate black separator sheet;
• it will not empty the document. Every page blank is an error, because a file of no pages still opens and still has a size, and a caller handed one would be told nothing was wrong;
• it does not count annotations as ink unless asked: a page whose only mark is a sticky note is blank paper somebody wrote on.

And -list writes nothing and takes no output file, because the first thing anybody should do with a tool that deletes pages is ask it which ones it means.

deskew straightens the tilted pages of a scan. It draws each page to measure it, and reports every page's angle and whether it was turned.

pdfops deskew -list scan.pdf              # just the angles, write nothing
pdfops deskew scan.pdf straight.pdf
pdfops deskew -threshold 0.1 -dpi 300 scan.pdf straight.pdf

The angle comes from Baird's projection profile (The skew angle of printed documents, Proc. SPSE 1987): the page is sheared rather than rotated — which keeps the scan-line counts exact, where resampling would blur the very line structure being measured — and the angle taken is the one that maximises the variance of the marked-pixel count per scan line. The survey that compares the methods (Mascaro, Cavalcanti et al., Pattern Recognition Letters 31 (2010) 1403–1411) ranks it first on documents with clear separation between text lines, which is what a page of text is.

⛔ The correction is not a rasterisation. The angle is measured on a render, but it is applied as a transformation of the page's own content, so text stays text and a vector drawing stays vector. A deskew that flattened the page would quietly undo the thing a scan is usually deskewed for, which is OCR.

Judged by tesseract, which knows nothing of this code: a rendered page it reads at 0.968 similarity straight, and at 0.895 tilted by 7°, it reads back at 0.968 once deskewed — the straight page's own figure.

A page under -threshold is reported and left alone, byte for byte: a scan that is already level should not come back rewritten. A page with nothing on it has no skew, and is reported as such rather than counted as level — one blank separator sheet does not stop the other three hundred being straightened.

⛔ -dpi is a measurement setting only, since nothing of the render reaches the output: a higher value buys accuracy and costs time, and never quality. Too low a value does silent damage, though — the projection has to resolve the text lines, and below about two samples per line pitch it returns an angle that is simply wrong. See SkewOptions.MaxPixels in go-images for the measured table.

⛔ divide makes the same grid as poster and is the opposite operation. poster keeps every piece the size of the page it came from, so it enlarges: the pieces taped together make one page the size of a wall. divide keeps the content at its own size and makes the pages smaller, so the pieces laid side by side are the page again at 1:1 — which is what a two-page spread scanned as one wide sheet, or two tickets printed on one page, needs.

Confusing them gives a result that looks right in a thumbnail and is wrong by a factor of across and down. The tests assert the media box, not the page count, and TestDivideIsNotPoster compares the two directly: without it, every assertion about divide could be satisfied by poster.

duplex is the same job when the scanner left one file rather than two: a feeder that cannot turn a sheet over gives every front, then every back. It cuts the file in the middle and interleaves the halves.

⛔ The backs are reversed by default, and the flag is -keep-backs rather than -reverse-backs so that the default is the common case: the stack is turned over as a stack, so the last sheet's back is scanned first. A tool whose default assumed otherwise would quietly put every back on the wrong front — which reads as "the scan is fine, the order is odd" rather than as a reversal.

The default split is ceil(n/2), not n/2: an odd stack has its unpaired sheet at the end of the fronts, so thirteen pages split at seven. ⛔ The test for that had to be rewritten. The first version used a reversed back block — and with the backs reversed the output is the same for any split in a range, because the reversal absorbs the difference. It passed at the ceiling and at the floor, so it proved nothing; the negative control is what said so, and the assertion now uses -keep-backs, where the split is load-bearing.

place draws a picture where you say it goes: a scanned or typed signature, a logo, a seal. It is stamp with an image instead of a line of text, and it deliberately takes the same words for where — the nine positions and a margin — so that -at bottom-right means the same thing for both.

pdfops place -image signature.png -at bottom-right -width 120 contract.pdf signed.pdf
pdfops place -image logo.png -pages 1 -x 40 -y 740 -dpi 300 plain.pdf headed.pdf
pdfops place -image seal.png -at center -opacity 0.3 draft.pdf sealed.pdf

Three ways of saying how big, and they are tried in that order: -width or -height alone keeps the picture's proportions (a signature scaled into a box is a signature nobody recognises), both together may distort it, and neither falls back to -dpi — the picture's own pixels, where 300 means a 300-pixel-wide picture is an inch across. The default is 72, one point per pixel.

⛔ A picture above 40 megapixels is refused, and the ceiling is yours to move. The size a picture declares comes from the file, and a compressed format describes a huge image in a few bytes — a PNG saying 100000 by 100000 is tens of kilobytes that become tens of gigabytes — so go-gfx/gfx v0.37.0 reads the header first and refuses what is over codec.DefaultMaxPixels (40 million) before allocating anything. A 50 MP camera photo is therefore refused by default, with an error wrapping codec.ErrTooLarge. In Go, (*Doc).SetMaxPixels(n) sets the ceiling for Doc.Picture and Doc.Place (width times height; zero means the default, and a picture exactly at the ceiling is accepted); on the command line pdfops place -max-pixels 60000000 …. Raise it for photographs you trust; lower it to what you need for files from strangers.

⛔ -x -y is an anchor, not a hint: it is taken as given and not clamped into the page, and -at is then ignored rather than combined with it. A signature deliberately bleeding off the edge of a scanned form is a real request, and a viewer clips what falls outside the media box perfectly well. The nine positions clamp; an explicit corner is an instruction.

⛔ This is what "sign a PDF" means in every tool that offers it — a picture of a signature, placed. It is not cryptography, and nothing here pretends it is: a drawn squiggle proves nothing about who drew it. The cryptographic kind is a different job under a different name, and conflating the two is how a scanned squiggle comes to be called a digital signature.

⛔ The drawing itself was already here, and nothing could ask for it. The machinery that puts an image at a rectangle on a page was written for the XFA renderer, and only XFA fed it: the capability was present, tested through one caller, and absent from the outside. Worse, a borrowed page did not honour it at all, because the conditions deciding whether a source page is wrapped or cached counted marks and not pictures. Wiring it up found a second defect of the same shape in nup: a blank page that carries anything emitted an empty form, because the switch asked "is it blank" before it asked "is anything on it" — so blank then number then nup dropped the page numbers off the sheet, silently, and had always done.

repair writes a file back out with its structure rebuilt, and says whether rebuilding was needed. -check answers that question and writes nothing.

pdfops repair -check suspect.pdf
pdfops repair broken.pdf fixed.pdf

⛔ The rebuilding is not this verb's doing. go-pdfkit/reader repairs a damaged file on its own and quietly, which is the right behaviour for a reader — a caller asking for page three wants page three, not a lecture. What was missing was a way to say it happened and to keep the result, and that is the whole of this verb. Doc.Repaired() is the same answer in Go.

It reports on any source, not all: a merged document's pages come from several files, and one damaged source is the answer to "did this need repairing". A true that depended on every source being broken would be false for every real merge.

⛔ And the scope is reader's, not QPDF's. What it recovers is a cross-reference table that is wrong, missing, or points into the middle of something else — the commonest damage, and the kind where the objects are all still there. It is not a general salvage engine, and a file whose objects are themselves truncated may still be unreadable. A file that did not need repairing is said out loud and is not an error, so that "repair everything in this directory" can be scripted.

labels sets what a viewer shows in its toolbar — "iv of 320" — which is never what is printed on the paper. For a number on the page, that is number above.

pdfops labels -list book.pdf
pdfops labels -rule 'pages=1-12;style=r' -rule 'pages=13-;style=D' book.pdf out.pdf
pdfops labels -rule 'pages=all;style=D;prefix=A-;start=1' appendix.pdf out.pdf
pdfops labels -clear book.pdf plain.pdf

The six styles are the format's own: D (1, 2, 3), r (i, ii, iii), R (I, II, III), a (a, b, c), A (A, B, C), and an empty style, which is the prefix alone — how a run of pages is marked "Cover" or "Insert".

⛔ A page-label range in a PDF is CONTIGUOUS. The file holds the first page of each run and the viewer counts forward from there, so a rule over 1-9,30-40 has to be compiled into two runs. Without continue the second run starts again at start; with it, it carries on. pages=odd is the extreme of the same thing — fifty runs of one page over a hundred — and both answers are expressible only because a rule is compiled rather than stored as written.

⛔ A page-label tree has no gaps either. An entry labels from its own key until the next entry, so a document labelled 1, 2 and then meant to stop comes back 1, 2, 3, 4 unless the stop is said out loud: an entry with neither a style nor a prefix. The first draft skipped unlabelled pages and the round-trip caught it.

⛔ start must be at least 1, and that is the format's rule rather than a choice here: ISO 32000-1 table 159 says /St "shall be greater than or equal to 1". A zero-based numbering scheme is not expressible as a numeric style, so a start below one is refused rather than written out of specification — a file that opens in one viewer and renumbers itself in another is worse than a refusal. BentoPDF's own tool offers "set to 0 for zero-based numbering", and what that writes is a value the standard forbids.

⛔ A page label's letters are not a spreadsheet's columns. Table 159 says the letters run "A to Z for the first 26 pages, AA to ZZ for the next 26" — so 27 is AA and 28 is BB, where a spreadsheet's 28 is AB. Borrowing the spreadsheet rule, which is the one everybody has written before, relabels every appendix past the twenty-sixth.

Roman numerals stop at 3999, where the additive notation runs out — 4000 needs a bar over the M, which neither a PDF string nor a page number has — and past it the number is written in decimal. A label a viewer shows wrongly is worse than one it shows in the wrong style.

⛔ Every verb used to drop the page labels

A /PageLabels tree is keyed by page index, so it cannot be carried across a rebuild the way the other document keys are: after a select or a reverse every label would be on the wrong page. So this package dropped it — and a document whose front matter was numbered i, ii, iii came back from a watermark numbered 1, 2, 3.

The labels are now asked for per page and rebuilt, which is right in both directions: a verb that keeps the pages keeps their labels, and a verb that moves them moves the labels with them. A merge keeps each source's labels on its own pages.

The runs are compressed on the way out: three hundred decimal pages are one entry, not three hundred. A label no style can explain becomes a run of one with the whole label as a literal prefix — the file then says exactly what the viewer must show, at the cost of an entry.

layers is the optional content a viewer calls layers: a drawing's dimensions, its markup, its alternatives.

pdfops layers -list plan.pdf
pdfops layers -hide 'Internal pricing' plan.pdf out.pdf
pdfops layers -add 'Markup/Mine' plan.pdf out.pdf

⛔ Every verb used to lose them. Measured before this: a one-page drawing with a layer switched off, run through pdfops watermark, came back with no /OCProperties at all — the content still marked as belonging to a layer, the layer object still reachable through the page's own resources, and nothing left saying which layer was hidden. So the hidden layer showed. A drawing distributed with an "internal pricing" layer turned off revealed it to any verb at all.

The layers are now constructed on the way out rather than carried, which is what lets them survive a reorder and a merge: keepCatalogue refuses to merge two catalogues, because two forms whose fields are named the same cannot honestly become one — but two drawings' layers can, since a layer is its object and two layers named "Markup" are two layers.

What the file says about its own layers, and is obeyed

/BaseState OFF	inverts the question: everything is hidden unless /ON names it. A reader that only looked at /OFF would report every layer of such a document as visible — and a drawing that ships with all its layers off but one is exactly how an alternatives document is built
/Locked	the document asking that a layer not be switched. -hide on one is refused: a drawing's dimension layer is what everything else is positioned against, and switching it anyway ignores the only thing the file says
/D /Order	the nesting, which is written back nested. A string inside it is the group's label rather than a layer

⛔ Dropping a hidden layer is refused, because what was drawn on it does not go with it: the content stays in the page and stops being optional, so the hidden drawing becomes permanently visible — the opposite of what somebody deleting a layer called "internal pricing" expects. Show it first if that is what you mean.

⛔ A name is not an identifier. A drawing assembled from two others often has two "Markup" layers, and switching one of them at random is a change nobody asked for and nobody can see, so an ambiguous name is refused with the count.

tables finds the grids a page's text lines up into, and writes them as CSV (RFC 4180, a blank line between one table and the next), as JSON (with the page and the box, so a caller knows where to look), or as GFM pipe tables.

pdfops tables accounts.pdf
pdfops tables -as md -pages 3 accounts.pdf
pdfops tables -as json -to tables/ accounts.pdf

A page holds no table — it holds text put at places — so this reports an inference, and extract is where the rules are: a boundary is a stretch every row leaves empty, a repeated one-character cell is a bullet and not a column, and a line far below the last row is not the next one. Ruled lines are not read, so a table held apart by rules rather than by whitespace is not found.

⛔ A page that could not be read is said on the error stream, not left to look like a page with no tables on it — and on the error stream rather than standard output, so that it cannot end up inside the rows somebody is about to parse.

⛔ The first row becomes the Markdown header. A pipe table has nowhere to put a table with no header row, and a grid found by looking at alignment does not know whether its first row is one. The cells go through go-richdoc rather than through a strings.Join here, so a cell holding a | is escaped by the code that already knew it had to be.

⛔ A document may say a word sits 10^308 points along, and a table with one word at minus that and another at plus it is infinitely wide. JSON has no spelling for an infinity, so -as json fails on such a file rather than writing "width": +Inf into something that has already begun parsing it.

markdown reads the pages back as a document — headings, paragraphs, lists and tables — through go-pdfkit/document, and writes them as Markdown. With -json it writes one record per page instead, each the page's Markdown plus the document's own metadata, which is the shape a retrieval pipeline already expects.

pdfops markdown report.pdf > report.md
pdfops markdown -pages 3-7 report.pdf
pdfops markdown -json report.pdf > report-for-indexing.json

⛔ A PDF has no headings. What this writes is an inference from how big the glyphs were and where they sat, and document's README says what it will and will not see — a page in columns, or a table held apart by ruled lines, is not read as either.

⛔ A page range is applied after the document is read, never before. What counts as a heading is decided by the sizes the whole document uses, so asking for page nine alone would otherwise give it different headings from asking for all of them — and a caller doing a document a page at a time would get a different answer for every page. A test asserts that page one reads the same both ways.

⛔ -json decodes the metadata. A title written in UTF-16 behind a byte order mark — which is most of what a modern producer writes — came out of the old code as \xfe\xffL\x00a\x00…. info printed it that way too, for as long as the verb had existed; both now go through reader.TextString.

word and excel write that same reading out as a .docx or an .xlsx, through go-ooxml.

pdfops word report.pdf report.docx
pdfops excel accounts.pdf accounts.xlsx

⛔ What comes out is the document, not the pages. The fonts, the margins, the line breaks and the page breaks are the typesetter's work; a .docx that reproduced them would be a picture of a PDF rather than something anybody can edit.

⛔ excel writes only the tables, one sheet each. A spreadsheet is a grid of cells and a document is not, so the prose around a table has nowhere to go — and a page with no tables gives the one empty sheet a workbook has to have rather than an error.

Both are the same function with a different writer, which is why -pages means the same thing in each and a defect found in one is fixed in both.

objects writes the object graph as JSON: what the file is made of, as opposed to what it says. For looking at a file that will not open anywhere else.

pdfops objects broken.pdf | jq '.objects["12 0"]'

⛔ It is the graph reachable from the trailer, not every object in the file. That is the honest set because it is the one a reader sees: an object no reference points at is in the bytes and in nobody's document.

⛔ A stream's bytes are not in it, its length and its dictionary are. They are the bulk of most files and they are binary, so including them would mean base64 — a 35 MB file becoming a 47 MB JSON nobody reads.

A name keeps its slash ("/Catalog") and a string is wrapped ({"text": "…"} or {"hex": "…"} where it is not text), because the two have to be told apart and a PDF string may be exactly /Catalog. A reference that could not be read is named in unread rather than dropped: "the graph, and these edges lead nowhere" is what somebody debugging a damaged file needs.

textcolour paints the text one colour and leaves the rest of the page alone.

pdfops textcolour -to '#0000ff' report.pdf blue-text.pdf
pdfops textcolour -to black -pages 1-4 scan.pdf fixed.pdf

⛔ Setting a colour before each text object is not enough, and that is what the first draft did. A text object that sets its own colour overrides it, so a page whose paragraphs say 1 0 0 rg came out unchanged while the tool reported success. Every colour operator inside a text object is blanked — g, rg, k, cs, sc, scn and their stroking capitals — so the colour asked for is the one that paints.

⛔ The colour goes outside the text object, wrapped in q/Q. A text object does not save the graphics state, so a colour set inside one leaks into whatever is drawn after it — which is how a recolouring turns a company's logo the same shade as its paragraphs. A test asserts the rectangle beside the text keeps its own red.

Both the filling and the stroking colour are set, because text has render modes: 0 fills, 1 strokes, 2 does both.

⛔ It splices, and does not re-serialise. Writing the operations back out would mean formatting every operand again, including an inline image's still-encoded bytes, where a formatter that is merely lossless-for-objects can change what the image means. So the rewriting keeps every byte it does not deliberately change — which is why reader.Operation carries its byte range, added for this. A stream with no text comes back byte for byte unchanged, and the colour operators are blanked with spaces rather than deleted so that every offset after one is still the offset the scan reported.

⛔ What it cannot reach, it counts. Text drawn through a Form XObject is not recoloured, and the reason is not difficulty: an XObject is shared, so recolouring one would change pages the caller did not name. A document whose text is all inside XObjects would otherwise come back looking converted and be unchanged — the one failure a caller cannot see by opening the result and glancing at it. So the count is said out loud, and Doc.UnreachedText is the same answer in Go.

And #f00 is full red: each digit of a short hex colour is doubled, not shifted, so the component is 0xf × 17. A shift gives 240/255 for what every browser calls full red, and a logo recoloured to 94 % red is a defect nobody can see but a printer.

attachments lists the files a document carries, and writes them out with -to. A name comes out of the document rather than from whoever runs the command, so it is stripped to its last element: ../escaped.txt is written inside the directory asked for, and a name with no usable last element is refused rather than skipped. detach refuses a name the file does not carry, instead of writing an unchanged copy somebody then sends believing it was removed.

outline replaces the bookmarks from a file, or removes them with -drop. The file is one bookmark a line — a title, a tab, a page number — with two spaces of indent for each level of nesting:

Introduction	1
  Why	2
Chapter one	7

A line it cannot read is an error naming the line, because a table of contents quietly missing an entry is worse than one that refuses to be written.

That grammar is ops.ParseOutline(text string) ([]Bookmark, error), in the library rather than in the command, so that anything else asked to write a table of contents — the browser workbench among them — accepts exactly the same file instead of carrying a second parser for it.

contents writes the same file as a printed table of contents — a page of the document, each line a title at the left, its page number at the right and dots leading to it:

Contents

Introduction ....................................... 2
  Why .............................................. 3
Chapter one ........................................ 8

⛔ The numbers it prints are moved by the pages it adds. A contents page put in front of a document pushes every page of it down, so an entry that pointed at page 7 is on page 8 afterwards, and printing the 7 would send a reader one page short of every heading in the book. ops.ContentsPage returns how many pages it added and ops.ShiftBookmarks(marks, from, by) is the same rule for the /Outlines tree — which is what -bookmarks does, so the printed number and the bookmark land on the same page. An entry pointing at a page the document has not got is refused by name rather than printed.

It runs onto as many pages as it needs, because a table of contents quietly missing its last nine entries is worse than one that refuses to be written: the first is a document that looks finished.

A caller that already has the entries, rather than a file of them, goes through ops.Nest([]Level) ([]Bookmark, error): a title, a page and a depth in reading order, which is what an /Outlines tree looks like once it has been walked. ParseOutline is that function with a parser in front of it. A row more than one level under the row before it is a *ops.DepthError naming the row — a tree with a rung missing is one nobody meant to ask for, and every way of guessing the missing level is a different document.

resize sets how big a page is; crop sets what of it shows. Two verbs, because they are two questions. A size is a box in points, or one of a3, a4, a5, letter, legal, tabloid — derived from their definitions rather than typed, so A4 is 210 × 297 mm at 72 points to the inch and not a number somebody copied. -landscape turns a named size on its side.

metadata writes what info prints. A key is given the way the PDF names it, with or without the leading slash, because a person reading info sees Title and should be able to type what they saw.

text -json writes the same reading as the lines, in a shape something else can parse without splitting on tabs. With -layout each run carries its position, its size, and whether it was invisible or unreadable — and those two are written even when false, because a reader that cannot tell "this run is readable" from "this tool did not say" is reading a guess.

text reads the page back as words, and -layout says where every piece of it sits — a page, a place, a size, and what it says. A piece the document gave no way to read comes back marked rather than guessed at, so you can tell a page that says nothing from one that could not be read. images writes out the pictures a page places, a JPEG as a JPEG.

Links and bookmarks are carried over and pointed at the pages they became here, so extracting three pages of a book leaves the links between those three working and drops the ones that led out of them. A link or a bookmark that already led nowhere in the source keeps its place without a destination, because a file with broken bookmarks should keep its shape rather than lose it.

The catalogue is always written from nothing, so a document-level script, an action that runs on opening, or a tree of embedded files never survives any operation at all. sanitize deals with what travels attached to a page: its own actions, an annotation's actions, JavaScript, launching, form submission, the annotation types that exist to play or embed something, and files associated with a page. A link to the web is not executable and stays.

What a screen reader follows

A tagged document says which run of marks on which page is a heading, a paragraph, a table cell, the label of a form field. That is the reading order a screen reader follows, and for a government form it is often what the law requires — RGAA in France, Section 508 in the United States, EN 301 549 in the EU. It cannot be copied across a rebuild: every part of it points into the document, and the number tree that indexes it is keyed by a number each page carries. So it is rebuilt, element by element, around the pages that survived, and those numbers are handed out afresh. A page that kept the number it had in a file it is no longer part of is a page a reader would look up and be told, with every confidence, about somebody else's.

An element whose every page has gone is removed and its parent pruned; one with some pages gone keeps the children that are left. An empty table cell is kept, because the shape of a table is part of what it says. A page written twice carries the structure once, on the first copy, since an element names one page. Merging two files carries no tree at all: two files have two role maps in which the same name may mean two different things, and there is no honest way to choose between them.

Of 1 633 real government forms from eighteen issuers, 1 021 carry a structure tree. After a rotate 1 014 still do — the seven that do not are the seven whose tree was empty in the file we were given — and 991 of the 1 021 are identical down to the last element, mark and annotation reference. The thirty that differ are accounted for one by one in the commit that added this. What cannot be carried is named in catalogue.go with the reason, rather than disappearing quietly.

Text is drawn in the four faces every viewer already has — Helvetica, Helvetica-Bold, Courier and Courier-Bold — so nothing is embedded and a watermark costs about a kilobyte. Stamp text may say {page}, {pages} and {n}, the last being a counter for serial numbering.

⛔ What a rewrite carries, audited rather than assumed

Twice in one afternoon this package was found to be dropping a catalogue entry whose absence is not "no opinion" but a default — so the loss changed the document silently. So the whole list was measured: a document holding twenty-three catalogue entries, a watermark over it, and of each entry the question what would its absence MEAN?

/PageLabels	the viewer numbers pages by position — a book's front matter came back 1, 2, 3 instead of i, ii, iii
/OCProperties	everything optional becomes visible — a hidden layer was revealed
/OutputIntents	no colour management: a prepress file lost the ICC profile it declares, and would print differently with nothing to say why
/URI	relative links resolve against wherever the viewer thinks the document is — the link is not removed, it is silently pointed elsewhere

All four are carried now, the first two rebuilt (they cannot be copied: one is keyed by page index, the other names objects the content reaches another way) and the last two copied.

What is deliberately not carried, and why

• /OpenAction, /AA — what the document does when it is opened. Dropping them is safer, and is what sanitize exists for: carrying them would carry code across a rebuild;
• /Legal, /DSS — a signature's attestation and its validation data. A rewrite has already invalidated the signature the moment it changed a byte, and carrying the validation data of a signature that no longer verifies would say the document is still attested when it is not;
• /Threads, /Collection — these point into the document, at pages and at embedded files, which is why /Names /Dests is not carried either.

⛔ And three the audit reported as lost that are not: /AcroForm, /Outlines and /StructTreeRoot are all carried. The audit's fixture held an empty one of each, and an empty form, outline or structure tree is deliberately not written. A measured absence has to be read before it is believed — which is the other half of why an audit is worth running.

Verified against real files

Every operation is checked on the same corpus of 118 863 real PDFs the reader is measured against — Matplotlib, cairo, pdfTeX, Ghostscript, Adobe, R, Apache FOP, PDF 1.3 through 1.7. For each of the 118 833 that open, four properties have to hold, compared on the bytes of every page's content stream rather than on an exit status:

• writing the document unchanged reproduces every page;
• reversing twice is the identity;
• merging a document with itself doubles it exactly;
• selecting the last page yields exactly that page.

All four hold on all 118 833 files.

Laying pages out is checked the same way: for each of the 1 959 multi-page files in that corpus, two-up produces the right number of sheets and every form drawn on them carries exactly the bytes of the page it stands for, blanks included. All 1 959 hold.

Testing

go test -covermode=set ./...

CI gates on exact 100% statement coverage, go vet, and a cross-compile across linux/{amd64,arm64,riscv64,loong64,ppc64le,s390x}, js/wasm, darwin/arm64 and windows/amd64.

The rest of the family

reader reads and writes the format · ops is the verbs and the pdfops command · render turns a page into pixels · pdffont is what a document says about a font · extract reads a page back · coedit is a PDF several people edit at once · app is the browser workbench.

Documentation for all of it: https://go-pdfkit.github.io/docs/

License

BSD-3-Clause — see LICENSE. Copyright the go-pdfkit/ops authors.

Forms

pdfops fields lists what a form asks for and what it holds; pdfops fill fills it in and saves it; pdfops addfields puts new fields into a document that has none, or adds to the form it already carries.

$ pdfops fields fw9.pdf
note: the file also carries an XFA form, which is not read; the standard one is.
topmostSubform[0].Page1[0].f1_01[0]                       text      ""
topmostSubform[0].Page1[0].Boxes3a-b_ReadOrder[0].c1_1[0] checkbox  "Off"
                                                            buttons [1]

$ pdfops fill -set 'topmostSubform[0].Page1[0].f1_01[0]=Wolfgang Amadeus Mozart' \
              -set 'topmostSubform[0].Page1[0].Boxes3a-b_ReadOrder[0].c1_1[0]=1' \
              fw9.pdf filled.pdf

Creating a form

pdfops addfields is the other half of fill, which can only change a field the document already has. The fields are described in a JSON file, because the thing being described is a layout and a command line is a poor place to put one.

$ cat fields.json
[
  {"name": "applicant", "kind": "text", "required": true,
   "widgets": [{"page": 1, "rect": [72, 700, 300, 720]}]},
  {"name": "agree", "kind": "checkbox", "value": "Yes",
   "widgets": [{"page": 1, "rect": [72, 670, 86, 684], "on": "Yes"}]},
  {"name": "size", "kind": "combo",
   "options": [{"value": "s", "label": "Small"}, {"value": "l", "label": "Large"}],
   "widgets": [{"page": 1, "rect": [72, 630, 220, 650]}]}
]

$ pdfops addfields -fields fields.json blank.pdf form.pdf
added 3 field(s)
$ pdfops fields form.pdf
applicant  text      ""   required
agree      checkbox  "Yes"  buttons [Yes]
size       combo     ""   row "s" row "l"

The kinds are text, checkbox, radio, button, combo, list and signature — the names pdfops fields prints, so a description can be written from a listing. A field showing in several places is one field with several widgets, which is what a radio group is.

It refuses before it writes, because each of these makes a file that opens and is broken: a field with no name (a field is addressed by its name), a choice field with no rows, a checkbox whose on-state is Off (the name the format reserves for not chosen), two widgets of one radio group in the same state, a comb field with no maxLen, an empty rectangle, a page the document does not have, and two fields of one name — the reader keys fields by name, so the second would silently shadow the first.

Unknown keys in the description are refused too: "mutliline" is a form missing the thing it was written to ask for, and a parser that ignores what it does not know says nothing about it.

Two details the format makes easy to get wrong, both held by tests:

• A checkbox's drawing is not generated by the viewer. NeedAppearances covers text and choice fields, but a button shows whatever its own appearance dictionary holds for the state it is in, so one written without them is a box that never looks ticked however often it is clicked. One is drawn for each state, with the path operators rather than with a font, so the document needs no ZapfDingbats to show a tick.
• A widget is marked Print. Without it the field is on screen and gone from the paper, which is where a filled form is most often looked at.

Marking text: pdfops mark

pdfops mark writes the marks a reader makes over a page rather than on it — a highlight, an underline, a strikeout or a squiggly underline (ISO 32000-1, §12.5.6.10, Table 179).

$ pdfops mark -area 1:72,700,300,712 -note "worth coming back to" paper.pdf marked.pdf
marked 1 area(s) as highlight

$ pdfops mark -area 1:72,700,300,712 -and 1:72,686,250,698 -as underline paper.pdf marked.pdf
marked 2 area(s) as underline

An area is page:x0,y0,x1,y1 in points, counting up from the foot of the page — the space a PDF is written in. -and marks another area; each one becomes its own annotation, because a markup annotation's several quadrilaterals are for the several lines of one selection ("each quadrilateral shall encompass a word or group of contiguous words"), and two unrelated marks in one annotation would share a note, a colour and a date, and come off together.

⛔ The page's own content is not touched. That is the whole reason to write an annotation rather than draw on the page: the glyphs underneath are unchanged, the mark comes off again by dropping the annotation, and a reader who wants the document as its author wrote it gets it.

Three details the format makes easy to get wrong, each held by a test that renders the page and looks at the pixels:

• A mark with no appearance stream is invisible. Table 179 says /AP "shall take precedence over QuadPoints"; go-pdfkit/render draws an annotation only from its appearance and skips one that has none. So a highlight written with QuadPoints alone is a perfectly correct annotation that nothing shows.
• A highlight is translucent, not multiplied. Multiply is what a highlighter physically is and what Acrobat writes, but render implements no blend modes at all — a Multiply highlight paints an opaque block over the sentence it marks. Constant alpha is understood everywhere and leaves the words legible.
• The vertex order follows Table 179's figure, not its prose. The text says the four vertices are "counterclockwise"; Figure 64 shows (x1,y1) and (x2,y2) along the top edge and (x3,y3), (x4,y4) along the bottom — a Z. Every widely used reader follows the figure. The Quad type names its corners rather than taking eight numbers, so the mistake cannot be made silently.

And the marks are flagged Print, for the same reason a widget is: without it a highlight vanishes the moment somebody prints what they marked up.

Redacting text: removing it, not covering it

Doc.Redact takes the text inside a region out of the page's content stream. ISO 32000-1 §12.5.6.23, on applying a redaction:

Conforming readers that support redaction annotations shall provide a mechanism for applying content removal, and they shall remove all traces of the specified content. If a portion of an image is contained in a redaction region, that portion of the image data shall be destroyed; clipping or image masks shall not be used to hide that data.

⛔ A black rectangle is not a redaction. It is the standard data leak: the text is still in the content stream, so anyone's copy and paste — or pdfops text — reads it straight back out from under the box. So the test that decides whether this is a redaction at all is not about how the page looks, it is that the redacted string does not appear in the bytes that come out, and a drawn rectangle fails it.

The operator that drew the text is overwritten with spaces of its own length, which does two things at once: the string is gone from the stream, and every offset after it is still where it was, so the text that stays does not move. extract.Run.Op says which bytes to overwrite — deciding which glyphs are inside a region is work extract already does, and a second implementation of it here would cut the document somewhere other than where the caller was told the text was.

The region is then painted over, because an area that was emptied and not marked cannot be told from one that was always blank. That paint is a marking, not the redaction: the text is already gone when it is drawn.

⛔ And it refuses rather than half-redacts, naming the page and the reason. A caller who is told can choose another region or another tool; a caller who is not told believes the document safe to send. It refuses a region holding:

what	why
text inside a Form XObject	the bytes are not the page's, and an XObject is shared — cleaning one page would change every page that draws it. RecolourText stops at the same wall
an image this release cannot rewrite	a plain one is no longer refused: its samples are destroyed and it is written back, which is what §12.5.6.23 asks for. What is refused is a picture whose samples cannot be put back truthfully — see destroying the pixels below
an annotation	its own /Contents can hold the very words being redacted
an annotation this document has added and not yet written	a mark or a widget is held until the write, because each names the page object it sits on, so neither is in the source file — and reading only the source missed them. The workbench walked into it: highlight a line, then decide to redact it, and the text went while the note quoting it stayed. Drop the mark or the field first
an operator only part of whose text is inside the region	blanking it would remove text nobody asked about, and the surviving words cannot simply be re-emitted: the bytes in the stream are font codes, and Unicode does not round-trip back to them through a CID font's map. Keeping the survivors' original bytes and replacing the removed ones with an equivalent kern is the way, and it is next
text that reaches outside the region	a run is the unit that can be removed, so a producer drawing a whole line with one Tj makes that line the run — and redacting one word of it would take the line. That is not a leak, which is why it is tempting to allow; it is the other way a redaction betrays its caller, by removing what they did not ask about and not saying so. A region is allowed to miss a run's own box by up to two points, because a caller mapping a lit rectangle back into page space arrives through screen pixels

The refusal happens when Redact is called, not when the document is written — by the time of Bytes the caller has already been promised a redacted document by everything in between.

Destroying the pixels

§12.5.6.23 is explicit that a clip or a mask will not do, so the samples under the region are overwritten and the picture is written back. A scanned page is one picture, so without this the verb refused everything a scanner produces.

What this does it to is a picture whose samples are plain — one the reader has already unfiltered — eight bits a component, DeviceGray or DeviceRGB, drawn upright and drawn once on the page. Each of the others is refused by its own name, with the reason:

refused	why
still DCTDecode or JPXDecode	its samples would have to be decoded and re-encoded, which is a size and a quality question of its own: re-encoding as Flate makes a scan five to twenty times larger, and re-encoding as JPEG re-compresses every pixel of the page, including the ones nobody asked about
drawn turned or skewed	the box a turned picture covers looks exactly like the box an upright one covers, so mapping the region into its samples by that box would destroy more than was asked for — which is the other way a redaction betrays its caller. extract.Image.Matrix exists for this: b and c are zero for an upright picture, and stay zero for a flipped one
drawn more than once on the page	replacing the picture replaces it for every Do that names it, so blanking it for one placement blanks it for the others, in a different place each time
an inline image	it is written into the content stream rather than held as a resource, so there is nothing to write back in its place
/ImageMask, /SMask, /Mask, /Decode	each changes what a sample means, and a sample set to zero under one of them is not necessarily a destroyed one
/Indexed, /ICCBased, /DeviceCMYK, anything else	in an indexed space a zero is the palette's first colour, which may be white; in the others the component count has to be read out of a stream. What a destroyed sample holds there is not something to guess at
samples that are not all there	the stream is shorter or longer than Width × Height × components, so what is in it is not what it says

⛔ The original picture is not left in the file. The page's /Resources are rebuilt with the blanked picture in place of the original rather than copied — copying them would have written the original into the output beside the blanked one, referenced by nothing and holding every pixel the redaction promised to remove. That is exactly the shape of the leak fixed in #74, one object along, and the test for it walks every object of the written file and decodes it, because a picture's samples are compressed and a raw grep cannot see them.

⛔ And the sample grid counts down while the page counts up. A picture occupies the unit square of the transform in force, where v runs from its bottom edge to its top, while sample row zero is its top row. Reading one for the other blanks the mirror image of what was asked for, which looks like a redaction until somebody compares it with the page. A region ending part way into a sample destroys all of that sample: a third of a secret is a secret.

A redaction also survives the rest of the verbs. That is worth saying because the write path once dropped a rewritten content stream whenever it stamped a page: for a recolouring that loses a colour, but for a redaction it would put the removed text back and draw the fill over it, so the document would look redacted and not be. There is a test that redacts and watermarks the same page and then looks for the string.

A document assembled from several files can still be given a form: two sources' forms cannot be merged — their fields may be named the same — but fields the caller asked for come from neither source and so are in no conflict.

A filled form is written as an incremental update: the original file, byte for byte, with the objects that changed appended after it and a new cross-reference section pointing back at the old one. That is how every program that saves a form saves one, and it is the safest thing a program can do to somebody's document — nothing already there is rewritten, so whatever this does not understand survives untouched, and if the update is wrong the original is still the first part of the file.

The update says where its objects went the same way the file already does. A file whose cross-reference is a stream cannot be pointed back at by a plain table: a reader following /Prev would find an object where it expected the word xref. That is not a nicety — macOS's own renderer draws nothing at all for such a file, which is how the mistake was found.

Two things it will not do. A document that had to be repaired to be read has no cross-reference section worth pointing back at, so it is refused rather than added to. A document that is encrypted has every string and stream in it written through a key, and this does not yet write into one.

Measured on a real form: fw9.pdf filled in every field, written out, read back with every value in place, and rendered by macOS — which drew what we drew, in the same places, comb cells and ticks included.

⚠ Filling a form is the only verb that keeps one. Every other verb here takes the pages apart and builds a new document round them, and a form is tied into a document by object number in a dozen places at once — so merging, splitting or rotating a form's pages loses the form. Use fill on the file itself.

The form inside a blank document

Some PDFs are blank to look at and are not blank. Their pages hold one panel — "Please wait... your PDF viewer may not be able to display this type of document" — and the real form exists only as XML beside them, laid out when Adobe's own reader opens it. Adobe removed the format from PDF 2.0; no browser and no other reader lays one out.

pdfops xfa lays it out and draws it, which turns such a file into one anything can show.

$ pdfops xfa t1135-fill-23e.pdf readable.pdf
6 sheets, 497 elements drawn

Measured over every dynamic form in a corpus of 2 240 real government forms — fourteen of them, from the Canada Revenue Agency, the French cerfa series, the US Department of Labor and the Office of Personnel Management: all fourteen laid out, 111 sheets, 9 808 elements, nothing left unplaced, and every file read back by poppler.

WHERE each element goes is go-pdfkit/xfa's, measured against pdf.js and pdfium. What is drawn INSIDE each box is this package's own and much simpler: the four standard faces, a greedy line break, a rule round each field. So this is legibility rather than fidelity — the difference between a blank sheet and a form somebody can read, not between this and Adobe.

Pictures are drawn — 27 of them across twelve of the fourteen. For most that is a logo; for French cerfa 12064 it is the whole printed form. That one carries 212 fields, 7 draws, 4 images and — counted, not guessed — no <text> and no <caption> whatever, so without its pictures it is a grid of empty boxes and with them it is the customs declaration it is. How big each one is drawn follows pdfium's own arithmetic (xfa/fxfa/cxfa_ffwidget.cpp:55-86), because pdf.js declines to settle fit and actual and leaves them to a browser's sizing of an <img>.

A picture's own size is its pixels at its own resolution, read from a PNG's pHYs chunk or a JPEG's JFIF density, and that is not a detail. Every Canada Revenue Agency form in the corpus writes aspect="actual", and their logos are 1200-dpi PNGs: 2617 pixels across is 157.02 points, which is to a hundredth of a point the width the template writes for the box. Taken at 72 dpi — one point per pixel, the assumption for a picture that declares nothing — the same logo is 2617 points wide and covers a third of the sheet in black. It did, until the render was looked at.

Page numbers come out right — "Page 3 of 11", and "Page 3 sur 4" on a French form — without running the form's scripts. A footer like that is not text: it is rich text holding a floating field naming a hidden field whose value a script sets. All sixteen of those in the corpus ask one of exactly two things:

this.rawValue = xfa.layout.page(this);
this.rawValue = xfa.layout.pageCount();

Those are questions asked of the layout, which has just answered them. Recognising the two and handing back what the pager computed is not evaluating anything. Neither reference does this — pdf.js's layout ignores xfa:embed entirely, pdfium runs a JavaScript engine — so it is named as ours. Anything else a floating field names is left as a gap, because a wrong number that looks right is worse than a blank: four cerfa fields carry a static default beside their script, and printing that would put "Page 1 of 1" on every sheet.

Two limits, both named rather than left to be discovered:

• A picture in a file outside the document is not fetched. That is pdf.js's position and its words: "we don't get remote data and use what we have in the pdf itself, so no picture for non null href". Two of cerfa 12818's are like that, and both name an absolute path on the machine of whoever drew the form — C:\Users\…\Downloads\logo ministere.PNG. They are reported, not passed over.
• Text is set smaller rather than cut off. The box was measured with the font the template names and this draws in Helvetica, which is wider; at the template's own size the words come to more lines than the box holds. The size comes down until they fit, with a floor of four points, because losing the end of a sentence is worse than setting it a point smaller.