diff --git a/CHANGELOG.md b/CHANGELOG.md index 7c2188f..666489e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,12 @@ against an empty feed. decoded as far as it goes. [`ADR-005`](docs/adr/0005-image-filter-decoders.md) scopes the remaining four filters — `CCITTFaxDecode`, `DCTDecode`, `JPXDecode` and `JBIG2Decode` — with the evidence path each one has to meet. +- **CCITT fax images are decoded.** `CCITTFaxDecode` covers Group 3 (one- and two-dimensional) and Group 4 + for bilevel images — `K`, `Columns`, `Rows`, `BlackIs1`, `EncodedByteAlign` and `EndOfLine` are honoured — + so a scanned page now yields its samples (and `Gray8` pixels) instead of `ImageFilterUnsupported`. The + decoder is checked against Group 3 and Group 4 strips Pillow wrote through libtiff, with the bitmap + Pillow decodes as the reference; [`ADR-005`](docs/adr/0005-image-filter-decoders.md) schedules + `DCTDecode`, `JPXDecode` and `JBIG2Decode` next. ## [1.8.0] - 2026-11-14 diff --git a/docs/SharpCorePDF-Manual-1.8.0.pdf b/docs/SharpCorePDF-Manual-1.8.0.pdf index aafe8bb..a432a78 100644 Binary files a/docs/SharpCorePDF-Manual-1.8.0.pdf and b/docs/SharpCorePDF-Manual-1.8.0.pdf differ diff --git a/docs/adr/0005-image-filter-decoders.md b/docs/adr/0005-image-filter-decoders.md index c6703bb..9e670db 100644 --- a/docs/adr/0005-image-filter-decoders.md +++ b/docs/adr/0005-image-filter-decoders.md @@ -1,7 +1,7 @@ # ADR-005: Decoding the image filters the reader passes through -- **Status**: **Accepted** (2026-11-14) — `RunLengthDecode` implemented; `CCITTFaxDecode`, `DCTDecode`, - `JPXDecode` and `JBIG2Decode` accepted in that order, one release slice each +- **Status**: **Accepted** (2026-11-14) — `RunLengthDecode` (2026-11-14) and `CCITTFaxDecode` (2026-11-15) + implemented; `DCTDecode`, `JPXDecode` and `JBIG2Decode` accepted in that order, one release slice each - **Date**: 2026-11-14 - **Related**: [`docs/parity.md`](../parity.md) row 14, [`docs/manual/src/08-inspection.md`](../../manual/src/08-inspection.md), @@ -105,15 +105,15 @@ Two facts the first attempt established, so the next one does not rediscover the horizontal to `a2`, vertical to `a1`) — and to search for the first reference transition **strictly greater than `a0`**. With that, line 12's `b1` is 16, V-3 resolves to 13, and the line completes. -The RunLength slice is shipped; the CCITT decoder is the open work, and the Group 4 fixture now decodes its -first twelve lines correctly against this data. +Both slices are shipped: the Group 4 fixture decodes every line against this data, the Group 3 fixture +decodes as well, and `PdfCcittFilterTests` pins both. ## Status of the slices | Slice | State | | --- | --- | | `RunLengthDecode` | **Implemented** (2026-11-14) | -| `CCITTFaxDecode` | Planned — the next slice | -| `DCTDecode` | Planned | +| `CCITTFaxDecode` | **Implemented** (2026-11-15) | +| `DCTDecode` | Planned — the next slice | | `JPXDecode` | Planned | | `JBIG2Decode` | Planned | diff --git a/docs/manual/src/08-inspection.md b/docs/manual/src/08-inspection.md index 8239fdc..f2af588 100644 --- a/docs/manual/src/08-inspection.md +++ b/docs/manual/src/08-inspection.md @@ -233,8 +233,8 @@ component counts, or an `Indexed` space over any of them. An image that carries case rather than an exception — its payload is its samples — so an inline image, which producers rarely compress, states its pixels the way a Flate one does. Nothing is left behind a filter either, because the filters an image realistically uses are decoded here — Flate and LZW, each with its predictor, the two -ASCII encodings, and run-length — so an image that a viewer can show is an image whose samples this record -states. +ASCII encodings, run-length, and the CCITT fax codings (Group 3 and Group 4) — so an image that a viewer +can show is an image whose samples this record states. Two things are unpacked on the way out rather than handed over as they lie in the file. Samples narrower than a byte are widened to the same scale, so a bilevel image yields `0` and `255` rather than `0` and `1`, a @@ -251,7 +251,7 @@ which one it is: | Case | What the entry says | Why no pixels | | --- | --- | --- | | A `DCTDecode` image | `IsDecodable` true, `Filter` `DCTDecode`, `PixelFormat` null | The bytes are already a JPEG, which a caller can read; unpacking it here would mean shipping a second decoder. | -| A filter the library passes through — `JPXDecode`, `JBIG2Decode`, `CCITTFaxDecode`, or a name it does not know | `IsDecodable` false, `Filter` names it, and the read warns with `ImageFilterUnsupported` | Nothing was decoded, so there are no samples to hand back. [`ADR-005`](../../adr/0005-image-filter-decoders.md) schedules these decoders, in that order. | +| A filter the library passes through — `JPXDecode`, `JBIG2Decode`, or a name it does not know | `IsDecodable` false, `Filter` names it, and the read warns with `ImageFilterUnsupported` | Nothing was decoded, so there are no samples to hand back. [`ADR-005`](../../adr/0005-image-filter-decoders.md) schedules these decoders, in that order. | | A `Separation`, `DeviceN` or `Lab` space, sixteen bits per sample, a palette whose samples run past its table or whose table holds nothing, a width or height of zero, or a geometry the document does not state | `BitsPerComponent` or `ColorSpace` states which | The first needs a function evaluated, the second a rounding rule, the third a decision the file does not support, and the last a rectangle that is not there. Pixels are eight bit and channel-ordered by contract, so converting or guessing would hide the mismatch from the caller. | Neither a colour profile nor a tint transform is applied: the samples are what the document stores. That is diff --git a/docs/mutation-testing.md b/docs/mutation-testing.md index 0b02f6c..c5e32fb 100644 --- a/docs/mutation-testing.md +++ b/docs/mutation-testing.md @@ -850,3 +850,27 @@ and a `count <= 0` guard whose empty write and zero step are the same behaviour rather than pinned, which is what a survivor is for. `PdfImageFilters` gained the whitelist entry and the dispatch case, and the RunLength tests exercise both through `Decode`. +## CCITTFaxDecode — the fax codecs (2026-11-15) + +`CCITTFaxDecode` is the largest decoder in the tree: T.4's two run-length tables, the two-dimensional +modes, Group 3's end-of-line and tag bits, and the bitmap the samples become. Its reading: + +| File | Score | Killed | Survived | No coverage | Mutants | +| --- | --- | --- | --- | --- | --- | +| `Pdf/PdfFilters.Ccitt.cs` | **78.80%** | 394 | 48 | 58 | 580 | + +The survivors divide into three groups. **The parameter defaults** — `columns` and `rows` falling back +from the decode parameters to the image and then to 1728/0, `EncodedByteAlign`, `EndOfLine` — are guards +whose alternatives produce the same bitmap for the fixtures, which state the geometry and leave both flags +at their defaults. **The error branches** (`return false` for an out-of-range transition, a truncated run, +a mode that does not decode) are the no-coverage column: the fixtures are valid streams, and the malformed +stream test covers only the first of them. **The loop guard** (`++guard > columns * 2 + 64`) is never +reached by a valid stream, and its mutants are equivalent for data that terminates. + +Two of the survivors are a **missing fixture** rather than an equivalent: a run longer than 63 pixels (a +make-up code) and a Huffman code thirteen bits wide. The Pillow fixture is 16 x 16, so every run and code +is short; a wide fixture is the test that would close them. + +Three tests were added after the run — a stream without a row count, a bitmap above the budget, and a +stream that stops mid-line — and their effect is not measured here. + diff --git a/docs/parity.md b/docs/parity.md index 69736f7..8ab7325 100644 --- a/docs/parity.md +++ b/docs/parity.md @@ -37,7 +37,7 @@ No “Must” row is unowned. The owner is the story that closes it: `US1 creato | 11 | Digital signatures: sign, validate, timestamp, report integrity and trust separately | PDFsharp: yes. PdfPig: no. | Must: detached CMS over a reserved byte range, X.509, optional caller-supplied RFC 3161 token, integrity and trust as separate results. | 1.1 | US4 trust | `PdfDocumentSignerTests.Writes_a_signature_that_verifies_against_its_own_bytes`, `Signing_a_second_time_keeps_the_first_signature_valid`, `Embeds_the_timestamp_token_the_caller_supplies` (T080, T087), `PdfDocumentSignatureValidatorTests.Reports_a_changed_byte_as_a_digest_mismatch`, `Keeps_integrity_independent_of_trust`, `Reports_a_request_for_an_online_check_without_making_one` (T081) | `python docs/tools/validate-signatures.py --trust-signer ` (pyHanko, T093) and `uv run --with pikepdf python docs/tools/validate-pdf.py ` | Met (2026-10-23) on pyHanko and qpdf: a signed revision reports `intact` with coverage `ENTIRE_FILE`, a twice-signed one reports `intact`/`ENTIRE_REVISION` then `intact`/`ENTIRE_FILE`, both `trusted` against the signer's own certificate as anchor, and qpdf `--check` reports no problems on either file. A timestamp token is embedded, never checked: the authority is the caller's. The `sign-and-archive` example (T094) writes a PDF/A-3b file and signs it afterwards — the order that keeps the archival bytes intact — and pyHanko reads what it wrote as field `ArchivalSignature`, `integrity intact (coverage ENTIRE_FILE)`, signed at the moment the options state (2026-10-24) | | 12 | PDF/A-1b, -2b, -2u, -3b, -3u and -4 output | PDFsharp: PDF/A manager. PdfPig: no. | Must: embed every font, write XMP and an output intent, remove encryption and forbidden actions, refuse what cannot conform. | 1.1 | US4 trust | `PdfATests.Writes_the_structures_the_profile_requires`, `Refuses_a_font_it_cannot_embed`, `Refuses_transparency_for_pdf_a_1b_but_writes_it_for_2b` (T082) | `python docs/tools/validate-pdfa.py --verapdf --profile 2b archive-2b.pdf`, and the same for every other profile: veraPDF 1.30.4 reports `PASS` for all six profiles of an authored page and of `SafeWebCore.pdf`, by hand and on every push (`compliance.yml`; T093, T096) | Met — the output intent is sRGB built in, or the caller's own ICC profile and condition, written as supplied and checked as ICC first; PDF/A-3 embedded files are associated through `/AF` with a stated relationship; levels B and U are written, and PDF/A-4 writes the base level of ISO 19005-4; PDF/A-4f/-4u, a merge-time profile and level A are out of scope, and no conformance is claimed without a validator | | 13 | Positioned letters, words and blocks; configurable segmentation and reading order | PDFsharp: not a reading API. PdfPig: yes. | Must: `PdfLetter` with Unicode, size, font name, box and baseline; word and block strategies; raw and column-aware reading order with the strategy in the result. | 1.1 | US3 reading | `PdfLayoutReaderTests.Reads_one_letter_per_glyph_in_content_order`, `Splits_a_two_column_page_into_one_block_per_column`, `Reads_a_page_of_two_columns_left_to_right_whatever_order_the_content_used` (T059–T060), `PdfLayoutCorpusTests.Meets_SC_004_on_the_labelled_corpus`, `PdfLayoutCorpusTests.Table_page_is_still_read_column_by_column` (T076) | `python docs/tools/score-layout.py --labels tests/SharpCorePDF.Tests/Assets/parity/labelled --known-order-mismatch table` (T075) | Met on the labelled corpus (T075–T076): recall and boxes 100% on all five documents, reading order 100% on four. `table` is recorded as a known order mismatch — a table is symmetric under the XY cut, so a cell becomes a block and the order comes out column by column, which is the order a two-column page needs; table semantics are out of contract. `PdfLayoutCorpusTests.Table_page_is_still_read_column_by_column` fails if the limitation ever goes away | -| 14 | Images, hyperlinks, form values, annotations, embedded files, metadata, bookmarks | PDFsharp: partial. PdfPig: yes. | Must: expose each resource through the read side, decode image pixels when asked, and warn on an unresolved destination or unsupported filter. | 1.1 partial, pixels in 1.2 | US3 reading | `PdfLayoutReaderTests.Reads_the_links_of_a_page_and_their_uris`, `Resolves_a_direct_and_a_named_destination`, `Keeps_an_unresolved_destination_and_says_so`, `PdfContentReaderTests.Returns_every_annotation_with_its_geometry_and_text`, `Returns_the_field_tree_with_qualified_names`, `PdfContentReaderTests.Exposes_the_samples_of_a_decoded_image_as_pixels`, `Does_not_expose_a_jpeg_as_pixels`, `Does_not_expose_the_pixels_of_a_filter_it_cannot_decode`, `Exposes_a_palette_image_as_the_colours_of_its_table`, `Exposes_a_palette_whose_table_is_a_string`, `Exposes_bilevel_samples_as_one_byte_per_sample`, `Exposes_four_bit_samples_at_the_scale_of_their_depth`, `Exposes_two_bit_samples_at_the_scale_of_their_depth`, `Exposes_a_cmyk_image_as_four_device_samples_per_pixel`, `Exposes_the_samples_of_an_iccbased_image`, `Does_not_expose_the_samples_of_a_separation_space`, `Does_not_expose_a_palette_image_whose_index_is_outside_its_table`, `Leaves_an_expansion_above_the_budget_out_of_the_record`, `Exposes_the_samples_of_an_image_that_declares_no_filter`, `Keeps_an_expansion_that_fits_the_budget_exactly`, `Exposes_four_bit_samples_of_a_three_channel_image`, `Does_not_expose_an_image_that_states_no_extent`, `Exposes_the_samples_of_a_profile_by_the_components_it_states`, `Does_not_read_a_palette_whose_table_is_empty`, `Refuses_to_resolve_a_colour_space_without_a_document`, `Resolves_the_calibrated_forms_of_the_device_spaces`, `Does_not_build_a_palette_over_a_palette_or_over_a_function`, `Reads_a_palette_whose_declared_index_is_zero`, `Clamps_a_declared_index_to_the_entries_the_table_holds`, `Does_not_read_an_icc_space_that_states_no_profile`, `Exposes_the_pixels_of_an_inline_image_stated_by_an_abbreviation` (T072) | `uv run --with pypdf python -c "from pypdf import PdfReader; ..."` and `validate-pdf.py` on the corpus, plus the `score-layout.py` run of T079 (5 documents, 0 short of a threshold) and an external parse of all four export formats (T079) | Met in 1.2 (2026-10-24): every resource the row names is exposed — images with their payload, geometry, mask and pixels, hyperlinks with resolved and unresolved destinations, form values through `PdfFormFieldInfo`, annotations, embedded files, metadata and bookmarks; pixels are `Gray8`, `Rgb24` or `Cmyk32` for every filter the library decodes and for an image whose stream declares no filter at all — a palette applied so an `Indexed` image yields colours, samples narrower than a byte widened, four device samples left as CMYK rather than converted — and `null` for a filter it does not decode, which is reported with `ImageFilterUnsupported` rather than dropped. `RunLengthDecode` joined the decoded filters on 2026-11-14 and yields pixels; `CCITTFaxDecode`, `DCTDecode`, `JPXDecode` and `JBIG2Decode` remain pass-through, in the order ADR-005 schedules them | +| 14 | Images, hyperlinks, form values, annotations, embedded files, metadata, bookmarks | PDFsharp: partial. PdfPig: yes. | Must: expose each resource through the read side, decode image pixels when asked, and warn on an unresolved destination or unsupported filter. | 1.1 partial, pixels in 1.2 | US3 reading | `PdfLayoutReaderTests.Reads_the_links_of_a_page_and_their_uris`, `Resolves_a_direct_and_a_named_destination`, `Keeps_an_unresolved_destination_and_says_so`, `PdfContentReaderTests.Returns_every_annotation_with_its_geometry_and_text`, `Returns_the_field_tree_with_qualified_names`, `PdfContentReaderTests.Exposes_the_samples_of_a_decoded_image_as_pixels`, `Does_not_expose_a_jpeg_as_pixels`, `Does_not_expose_the_pixels_of_a_filter_it_cannot_decode`, `Exposes_a_palette_image_as_the_colours_of_its_table`, `Exposes_a_palette_whose_table_is_a_string`, `Exposes_bilevel_samples_as_one_byte_per_sample`, `Exposes_four_bit_samples_at_the_scale_of_their_depth`, `Exposes_two_bit_samples_at_the_scale_of_their_depth`, `Exposes_a_cmyk_image_as_four_device_samples_per_pixel`, `Exposes_the_samples_of_an_iccbased_image`, `Does_not_expose_the_samples_of_a_separation_space`, `Does_not_expose_a_palette_image_whose_index_is_outside_its_table`, `Leaves_an_expansion_above_the_budget_out_of_the_record`, `Exposes_the_samples_of_an_image_that_declares_no_filter`, `Keeps_an_expansion_that_fits_the_budget_exactly`, `Exposes_four_bit_samples_of_a_three_channel_image`, `Does_not_expose_an_image_that_states_no_extent`, `Exposes_the_samples_of_a_profile_by_the_components_it_states`, `Does_not_read_a_palette_whose_table_is_empty`, `Refuses_to_resolve_a_colour_space_without_a_document`, `Resolves_the_calibrated_forms_of_the_device_spaces`, `Does_not_build_a_palette_over_a_palette_or_over_a_function`, `Reads_a_palette_whose_declared_index_is_zero`, `Clamps_a_declared_index_to_the_entries_the_table_holds`, `Does_not_read_an_icc_space_that_states_no_profile`, `Exposes_the_pixels_of_an_inline_image_stated_by_an_abbreviation` (T072) | `uv run --with pypdf python -c "from pypdf import PdfReader; ..."` and `validate-pdf.py` on the corpus, plus the `score-layout.py` run of T079 (5 documents, 0 short of a threshold) and an external parse of all four export formats (T079) | Met in 1.2 (2026-10-24): every resource the row names is exposed — images with their payload, geometry, mask and pixels, hyperlinks with resolved and unresolved destinations, form values through `PdfFormFieldInfo`, annotations, embedded files, metadata and bookmarks; pixels are `Gray8`, `Rgb24` or `Cmyk32` for every filter the library decodes and for an image whose stream declares no filter at all — a palette applied so an `Indexed` image yields colours, samples narrower than a byte widened, four device samples left as CMYK rather than converted — and `null` for a filter it does not decode, which is reported with `ImageFilterUnsupported` rather than dropped. `RunLengthDecode` (2026-11-14) and `CCITTFaxDecode` (2026-11-15) joined the decoded filters and yield pixels; `DCTDecode`, `JPXDecode` and `JBIG2Decode` remain pass-through, in the order ADR-005 schedules them | | 15 | Public read-only object graph and content-stream inspection | PDFsharp: partial. PdfPig: yes. | Must: immutable `PdfObjectView`/`PdfContentOperationView`, dangling references visible, operation offsets reported. | 1.1 tests, the rest 1.2 (completed 2026-11-06, Unreleased) | US3 reading | `PdfObjectInspectorTests.Returns_one_node_per_listed_object_in_number_order`, `Projects_a_reference_as_a_node_and_never_follows_it`, `Reports_a_dangling_reference_once_for_the_target_it_names`, `Reads_the_operations_of_a_page_in_the_order_it_states_them` | `python docs/tools/validate-pdf.py artifacts/graph.pdf` on the object-graph fixture, run 2026-11-06: qpdf names the file valid with warnings, and the warnings are exactly the two deliberate dangling references to object 99 the fixture states; an independent scan of the same bytes finds objects 1-7 and no object 99, which is what the inspector reports (seven nodes, three unresolved references). The mutation attempt is recorded too: 77.8% over `PdfObjectInspector.cs` | Evidenced (2026-11-06) - twelve tests in `PdfObjectInspectorTests` are in the suite, the external run above agrees with the inspector on which objects exist and which reference dangles, and the mutation score for the inspector is recorded rather than assumed | | 16 | Export to ALTO, PAGE XML, hOCR and SVG text | PDFsharp: no. PdfPig: yes. | Must: four exporters in page space, escaped text, a valid empty-page element, and a result that names the strategy used. | 1.1 exporters, the rest 1.2 (completed 2026-11-06, Unreleased) | US3 reading | `PdfLayoutExportTests.Exports_alto_pagexml_hocr_and_svg`, `Writes_a_page_element_for_a_page_with_no_text`, `Escapes_text_that_looks_like_markup`, `Returns_the_blocks_it_wrote_with_the_strategy_that_produced_them` (T063) `Writes_the_metadata_times_the_page_schema_requires` (2026-11-06) | ALTO 4.2 and PAGE XML 2019-07-15 validated with `lxml` against the published schemas (`alto-4-2.xsd`, `pagecontent-2019-07-15.xsd`), run 2026-11-06: both exports valid after the run found the PAGE document stating a creator and no times - the schema requires `Created` and `LastChange`, which the exporter now writes at a fixed instant so the bytes stay reproducible | Evidenced (2026-11-06) - four exporters and their tests are in the suite, the hOCR and SVG documents are checked by their own tests, and the schema run T079 named is recorded: the ALTO and PAGE exports validate against the published schemas, with the PAGE metadata defect the run exposed now fixed and pinned by a test | | 17 | Rendering pages to images or raster previews | PDFsharp: no. PdfPig: no. | Non-goal: no rasteriser, thumbnail generator, viewer or print pipeline. | 1.1 (excluded) | Product boundary (ADR-001) | None by design | Out-of-scope statement in `README.md`, `docs/manual/src/02-what-it-is-not.md` and ADR-001 | Out of scope | diff --git a/src/SharpCorePDF.Infrastructure/Pdf/PdfFilters.Ccitt.cs b/src/SharpCorePDF.Infrastructure/Pdf/PdfFilters.Ccitt.cs new file mode 100644 index 0000000..7013f66 --- /dev/null +++ b/src/SharpCorePDF.Infrastructure/Pdf/PdfFilters.Ccitt.cs @@ -0,0 +1,543 @@ +namespace SharpCorePDF.Infrastructure.Pdf; + +/// +/// The CCITT fax half of : Group 3 (T.4) and Group 4 (T.6) coding for bilevel +/// images, which is the shape a scanned page usually has. +/// +/// +/// +/// The decoder reads runs, not bytes. A line is a sequence of white and black runs whose lengths are a +/// Huffman code; Group 4 codes every line relative to the line above it, and Group 3 codes a line on its +/// own or relative to the one above depending on the tag bit after its end-of-line code. +/// +/// +/// The bitmap is built with one bit per pixel, a set bit for black, rows padded to a whole byte. A +/// BlackIs1 of false — the default — inverts it at the end, which is the value a DeviceGray image +/// states: 0 is black and 1 is white. +/// +/// +internal static partial class PdfFilters +{ + /// The code tables T.4 fixes, as code:run pairs whose code length is their length. + private static readonly string[] CcittWhiteRuns = + [ + "00110101:0", "000111:1", "0111:2", "1000:3", "1011:4", "1100:5", "1110:6", "1111:7", + "10011:8", "10100:9", "00111:10", "01000:11", "001000:12", "000011:13", "110100:14", "110101:15", + "101010:16", "101011:17", "0100111:18", "0001100:19", "0001000:20", "0010111:21", "0000011:22", "0000100:23", + "0101000:24", "0101011:25", "0010011:26", "0100100:27", "0011000:28", "00000010:29", "00000011:30", "00011010:31", + "00011011:32", "00010010:33", "00010011:34", "00010100:35", "00010101:36", "00010110:37", "00010111:38", "00101000:39", + "00101001:40", "00101010:41", "00101011:42", "00101100:43", "00101101:44", "00000100:45", "00000101:46", "00001010:47", + "00001011:48", "01010010:49", "01010011:50", "01010100:51", "01010101:52", "00100100:53", "00100101:54", "01011000:55", + "01011001:56", "01011010:57", "01011011:58", "01001010:59", "01001011:60", "00110010:61", "00110011:62", "00110100:63", + "11011:64", "10010:128", "010111:192", "0110111:256", "00110110:320", "00110111:384", "01100100:448", "01100101:512", + "01101000:576", "01100111:640", "011001100:704", "011001101:768", "011010010:832", "011010011:896", "011010100:960", "011010101:1024", + "011010110:1088", "011010111:1152", "011011000:1216", "011011001:1280", "011011010:1344", "011011011:1408", "010011000:1472", "010011001:1536", + "010011010:1600", "011000:1664", "010011011:1728", "00000001000:1792", "00000001100:1856", "00000001101:1920", + "000000010010:1984", "000000010011:2048", "000000010100:2112", "000000010101:2176", "000000010110:2240", + "000000010111:2304", "000000011100:2368", "000000011101:2432", "000000011110:2496", "000000011111:2560", + ]; + + private static readonly string[] CcittBlackRuns = + [ + "0000110111:0", "010:1", "11:2", "10:3", "011:4", "0011:5", "0010:6", "00011:7", + "000101:8", "000100:9", "0000100:10", "0000101:11", "0000111:12", "00000100:13", "00000111:14", "000011000:15", + "0000010111:16", "0000011000:17", "0000001000:18", "00001100111:19", "00001101000:20", "00001101100:21", "00000110111:22", "00000101000:23", + "00000010111:24", "00000011000:25", "000011001010:26", "000011001011:27", "000011001100:28", "000011001101:29", "000001101000:30", "000001101001:31", + "000001101010:32", "000001101011:33", "000011010010:34", "000011010011:35", "000011010100:36", "000011010101:37", "000011010110:38", "000011010111:39", + "000001101100:40", "000001101101:41", "000011011010:42", "000011011011:43", "000001010100:44", "000001010101:45", "000001010110:46", "000001010111:47", + "000001100100:48", "000001100101:49", "000001010010:50", "000001010011:51", "000000100100:52", "000000110111:53", "000000111000:54", "000000100111:55", + "000000101000:56", "000001011000:57", "000001011001:58", "000000101011:59", "000000101100:60", "000001011010:61", "000001100110:62", "000001100111:63", + "0000001111:64", "000011001000:128", "000011001001:192", "000001011011:256", "000000110011:320", "000000110100:384", + "000000110101:448", "0000001101100:512", "0000001101101:576", "0000001001010:640", "0000001001011:704", "0000001001100:768", + "0000001001101:832", "0000001110010:896", "0000001110011:960", "0000001110100:1024", "0000001110101:1088", "0000001110110:1152", + "0000001110111:1216", "0000001010010:1280", "0000001010011:1344", "0000001010100:1408", "0000001010101:1472", + "0000001011010:1536", "0000001011011:1600", "0000001100100:1664", "0000001100101:1728", "00000001000:1792", + "00000001100:1856", "00000001101:1920", "000000010010:1984", "000000010011:2048", "000000010100:2112", "000000010101:2176", + "000000010110:2240", "000000010111:2304", "000000011100:2368", "000000011101:2432", "000000011110:2496", "000000011111:2560", + ]; + + private static readonly Dictionary CcittWhiteTable = BuildCcittTable(CcittWhiteRuns); + private static readonly Dictionary CcittBlackTable = BuildCcittTable(CcittBlackRuns); + + /// Decodes a CCITT Group 3 or Group 4 stream. + /// The encoded bytes. + /// The decode parameters, or null for the defaults. + /// The image dictionary, which states the width and height the parameters may omit. + /// The largest bitmap that is kept. + /// The packed bitmap, or null above the cap. + /// The bitmap length in bytes, whether or not it was kept. + /// True when the stream decoded. + public static bool TryDecodeCcitt( + ReadOnlyMemory data, + PdfDictionary? parameters, + PdfDictionary? image, + int maxBytes, + out byte[]? bytes, + out long length) + { + bytes = null; + length = 0; + + var k = (int)(parameters?.GetInteger("K") ?? 0); + var columns = (int)(parameters?.GetInteger("Columns") ?? image?.GetInteger("Width") ?? 1728); + var rows = (int)(parameters?.GetInteger("Rows") ?? image?.GetInteger("Height") ?? 0); + var blackIs1 = parameters?.Get("BlackIs1") is PdfBoolean { Value: true }; + var align = parameters?.Get("EncodedByteAlign") is PdfBoolean { Value: true }; + var eol = parameters?.Get("EndOfLine") is not PdfBoolean { Value: false }; + + if (columns is <= 0 or > (1 << 20) || rows < 0) + { + return false; + } + + var rowBytes = (columns + 7) / 8; + var total = rows > 0 ? (long)rowBytes * rows : 0; + var output = total > 0 && total <= maxBytes ? new byte[total] : null; + var stream = rows == 0 ? new MemoryStream() : null; + var keep = rows <= 0 || output is not null; + var row = new byte[rowBytes]; + var transitions = new List(); + var previous = new List(); + var bits = new CcittBits(data.Span); + var decoded = 0; + long produced = 0; + + while (rows == 0 || decoded < rows) + { + if (k >= 0 && eol) + { + if (!TrySkipEol(ref bits)) + { + break; + } + + if (align) + { + bits.AlignByte(); + } + } + + var oneDimensional = k == 0; + + if (k > 0) + { + if (!bits.TryReadBit(out var tag)) + { + break; + } + + oneDimensional = tag == 1; + } + + Array.Clear(row); + transitions.Clear(); + + if (!TryDecodeLine(ref bits, row, oneDimensional ? null : previous, columns, transitions)) + { + if (rows == 0 && decoded > 0) + { + break; + } + + return false; + } + + if (output is not null && decoded < rows) + { + row.CopyTo(output.AsSpan(decoded * rowBytes)); + } + else if (stream is not null) + { + if (stream.Length + rowBytes <= maxBytes) + { + stream.Write(row); + } + else + { + keep = false; + } + } + + produced += rowBytes; + (previous, transitions) = (transitions, previous); + decoded++; + } + + if (decoded == 0) + { + return false; + } + + length = produced; + bytes = output ?? (keep ? stream?.ToArray() : null); + + if (bytes is { } kept) + { + if (!blackIs1) + { + for (var index = 0; index < kept.Length; index++) + { + kept[index] = (byte)~kept[index]; + } + } + + // The bits past the last pixel are padding. They are cleared so a caller comparing the + // bitmap to another decoder's does not see a difference in a bit that is not a pixel. + var remainder = columns & 7; + + if (remainder != 0) + { + var mask = (byte)(0xFF << (8 - remainder)); + + for (var index = 0; index < kept.Length; index += rowBytes) + { + kept[index + rowBytes - 1] &= mask; + } + } + } + + return true; + } + + private static bool TryDecodeLine( + ref CcittBits bits, + byte[] row, + List? reference, + int columns, + List transitions) + { + var position = 0; + var black = false; + var a0 = -1; + var guard = 0; + + while (position < columns) + { + if (++guard > (columns * 2) + 64) + { + return false; + } + + if (reference is null) + { + if (!TryReadRun(ref bits, black, out var run)) + { + return false; + } + + var end = position + run; + + if (end > columns) + { + return false; + } + + if (black) + { + Paint(row, position, end); + } + + // A zero-length run is a transition at the same position: it is what a line that + // starts black states, and leaving it out shifts the parity of every later b1. + transitions.Add(end); + a0 = end; + position = end; + black = !black; + continue; + } + + var b1 = FindB1(reference, a0, black, columns, out var b1Index); + var b2 = NextTransition(reference, b1Index, columns); + + if (!TryReadMode(ref bits, out var mode, out var delta)) + { + return false; + } + + if (mode == 2) + { + if (b2 <= position) + { + return false; + } + + position = b2; + a0 = b2; + continue; + } + + if (mode == 1) + { + if (!TryReadRun(ref bits, black, out var run1) + || !TryReadRun(ref bits, !black, out var run2)) + { + return false; + } + + var a1 = position + run1; + var a2 = a1 + run2; + + if (a2 > columns) + { + return false; + } + + if (black) + { + Paint(row, position, a1); + } + else + { + Paint(row, a1, a2); + } + + transitions.Add(a1); + transitions.Add(a2); + position = a2; + a0 = a2; + continue; + } + + var transition = b1 + delta; + + if (transition < position || transition > columns) + { + return false; + } + + if (black) + { + Paint(row, position, transition); + } + + transitions.Add(transition); + position = transition; + a0 = transition; + black = !black; + } + + return true; + } + + private static bool TryReadRun(ref CcittBits bits, bool black, out int run) + { + run = 0; + + while (true) + { + if (!TryReadHuffman(ref bits, black ? CcittBlackTable : CcittWhiteTable, out var value)) + { + return false; + } + + run += value; + + if (value < 64) + { + return true; + } + + if (run > 2560) + { + return false; + } + } + } + + private static bool TryReadHuffman(ref CcittBits bits, Dictionary table, out int value) + { + var code = 0; + + for (var length = 1; length <= 13; length++) + { + if (!bits.TryReadBit(out var bit)) + { + value = 0; + return false; + } + + code = (code << 1) | bit; + + if (table.TryGetValue((length << 16) | code, out value)) + { + return true; + } + } + + value = 0; + return false; + } + + private static bool TryReadMode(ref CcittBits bits, out int mode, out int delta) + { + mode = 0; + delta = 0; + var code = 0; + + for (var length = 1; length <= 7; length++) + { + if (!bits.TryReadBit(out var bit)) + { + return false; + } + + code = (code << 1) | bit; + + switch (length, code) + { + case (1, 0b1): + return true; + case (3, 0b011): + delta = 1; + return true; + case (3, 0b010): + delta = -1; + return true; + case (6, 0b000011): + delta = 2; + return true; + case (6, 0b000010): + delta = -2; + return true; + case (7, 0b0000011): + delta = 3; + return true; + case (7, 0b0000010): + delta = -3; + return true; + case (3, 0b001): + mode = 1; + return true; + case (4, 0b0001): + mode = 2; + return true; + case (7, 0b0000001): + return false; + } + } + + return false; + } + + /// + /// Finds the first reference-line transition of the opposite colour to the current one, strictly to the + /// right of . + /// + /// + /// The comparison is against — the last changing element — and not against the + /// paint position: after a vertical or horizontal mode the last transition is the new position, so + /// >= position would select the transition the decoder has just consumed. That is the + /// difference between a Group 4 line that completes and one whose vertical mode resolves to the left. + /// + private static int FindB1(List reference, int a0, bool black, int columns, out int index) + { + // A reference line starts white, so an even transition is a change to black and an odd one to + // white. b1 is the first transition of the opposite colour strictly after a0. + for (var candidate = black ? 1 : 0; candidate < reference.Count; candidate += 2) + { + if (reference[candidate] > a0) + { + index = candidate; + return reference[candidate]; + } + } + + index = -1; + return columns; + } + + private static int NextTransition(List reference, int index, int columns) => + index >= 0 && index + 1 < reference.Count ? reference[index + 1] : columns; + + private static bool TrySkipEol(ref CcittBits bits) + { + var zeros = 0; + + while (true) + { + if (!bits.TryReadBit(out var bit)) + { + return false; + } + + if (bit == 1) + { + if (zeros >= 11) + { + return true; + } + + zeros = 0; + continue; + } + + zeros++; + } + } + + private static void Paint(byte[] row, int from, int to) + { + for (var pixel = from; pixel < to; pixel++) + { + row[pixel >> 3] |= (byte)(1 << (7 - (pixel & 7))); + } + } + + private static Dictionary BuildCcittTable(string[] pairs) + { + var table = new Dictionary(pairs.Length); + + foreach (var pair in pairs) + { + var colon = pair.IndexOf(':'); + var code = Convert.ToInt32(pair[..colon], 2); + var run = int.Parse(pair[(colon + 1)..], System.Globalization.CultureInfo.InvariantCulture); + table[(colon << 16) | code] = run; + } + + return table; + } + + private ref struct CcittBits + { + private readonly ReadOnlySpan data; + private int offset; + private int shift; + + public CcittBits(ReadOnlySpan data) + { + this.data = data; + offset = 0; + shift = 7; + } + + public bool TryReadBit(out int value) + { + if (offset >= data.Length) + { + value = 0; + return false; + } + + value = (data[offset] >> shift) & 1; + + if (--shift < 0) + { + shift = 7; + offset++; + } + + return true; + } + + public void AlignByte() + { + if (shift != 7) + { + shift = 7; + offset++; + } + } + } +} + diff --git a/src/SharpCorePDF.Infrastructure/Pdf/PdfImageFilters.cs b/src/SharpCorePDF.Infrastructure/Pdf/PdfImageFilters.cs index 32366b2..afa4917 100644 --- a/src/SharpCorePDF.Infrastructure/Pdf/PdfImageFilters.cs +++ b/src/SharpCorePDF.Infrastructure/Pdf/PdfImageFilters.cs @@ -88,7 +88,7 @@ public static PdfPayload Decode(PdfStream stream, int maxInlineBytes, string sou buffered ? Buffer(current, maxInlineBytes) : null); } - if (!TryDecodeOne(name, current, forFilter, sourceName, maxInlineBytes, out var bytes, out var decodedLength)) + if (!TryDecodeOne(name, current, forFilter, stream.Dictionary, sourceName, maxInlineBytes, out var bytes, out var decodedLength)) { return new PdfPayload(false, true, description, length, buffered ? Buffer(current, maxInlineBytes) : null); } @@ -316,6 +316,7 @@ private static bool TryDecodeOne( string name, ReadOnlyMemory data, PdfDictionary? parameters, + PdfDictionary? image, string sourceName, int cap, out byte[]? bytes, @@ -367,6 +368,9 @@ private static bool TryDecodeOne( return true; } + case "CCITTFaxDecode": + return PdfFilters.TryDecodeCcitt(data, parameters, image, cap, out bytes, out length); + default: // Unreachable: `Decode` passes only the names `IsDecodable` states, and every one of // them has a case above. The arm exists because a switch has to end somewhere, and it @@ -792,9 +796,10 @@ private static bool HasPredictor(PdfDictionary? parameters) => /// Determines whether a payload can be turned into samples by a filter with this name. /// The filter name, with its abbreviation expanded. - /// for the five filters this class decodes. + /// for the six filters this class decodes. private static bool IsDecodable(string name) => - name is "FlateDecode" or "LZWDecode" or "ASCIIHexDecode" or "ASCII85Decode" or "RunLengthDecode"; + name is "FlateDecode" or "LZWDecode" or "ASCIIHexDecode" or "ASCII85Decode" + or "RunLengthDecode" or "CCITTFaxDecode"; /// Determines whether a filter is handed back raw instead of being decoded. /// The filter name, with its abbreviation expanded. diff --git a/tests/SharpCorePDF.Tests/PdfCcittFilterTests.cs b/tests/SharpCorePDF.Tests/PdfCcittFilterTests.cs new file mode 100644 index 0000000..fabf9e0 --- /dev/null +++ b/tests/SharpCorePDF.Tests/PdfCcittFilterTests.cs @@ -0,0 +1,156 @@ +using SharpCorePDF.Infrastructure.Pdf; + +namespace SharpCorePDF.Tests; + +/// +/// Tests of the CCITT Group 3 and Group 4 decoder, against strips Pillow wrote through libtiff: the +/// bitmap Pillow decodes is the bitmap this decoder must state. +/// +public sealed class PdfCcittFilterTests +{ + private const int Width = 16; + private const int Height = 16; + + /// Group 4, as Pillow wrote the 16 x 16 fixture (a diagonal and a block). + private static readonly byte[] Group4 = File.ReadAllBytes(Fixture("ccitt-g4.bin")); + + /// Group 3 one-dimensional, as Pillow wrote the same bitmap. + private static readonly byte[] Group3 = File.ReadAllBytes(Fixture("ccitt-g3.bin")); + + /// The bitmap Pillow decodes: 1 is white, 0 is black, rows padded to a byte. + private static readonly byte[] Expected = File.ReadAllBytes(Fixture("ccitt-expected.bin")); + + private static string Fixture(string name) => Path.Combine(TestAssets.Folder, "ccitt", name); + + /// Group 4 codes every line relative to the one above it. + [Fact] + public void Decodes_a_group_4_strip_to_the_bitmap_the_fixture_states() + { + Assert.Equal(Expected, Decode(Group4, Params(k: -1, blackIs1: true))); + } + + /// Group 3 codes each line on its own after its end-of-line code. + [Fact] + public void Decodes_a_group_3_strip_to_the_bitmap_the_fixture_states() + { + Assert.Equal(Expected, Decode(Group3, Params(k: 0, blackIs1: true))); + } + + /// + /// The raw stream is the complement of the mode-1 pixels, so the default BlackIs1 inverts it. + /// + [Fact] + public void Inverts_the_bitmap_when_black_is_zero() + { + var expected = Expected.Select(value => (byte)~value).ToArray(); + + Assert.Equal(expected, Decode(Group4, Params(k: -1))); + } + + /// The width and height come from the image when the parameters omit them. + [Fact] + public void Reads_the_geometry_from_the_image_when_the_parameters_omit_it() + { + var payload = PdfImageFilters.Decode( + Stream(Group4, Params(k: -1, blackIs1: true, geometry: false)), + 1 << 20, + "ccitt.pdf"); + + Assert.True(payload.IsDecodable); + Assert.Equal(Expected, payload.Data!.Value.ToArray()); + } + + /// A stream that is not CCITT data is a payload the filter refuses. + [Fact] + public void Refuses_a_stream_it_cannot_decode() + { + var payload = PdfImageFilters.Decode(Stream([0, 0, 0, 0], Params(k: -1)), 1 << 20, "ccitt.pdf"); + + Assert.False(payload.IsDecodable); + Assert.True(payload.Failed); + } + + /// + /// A stream that stops mid-line is refused rather than written as far as it got: the image would be + /// missing rows the document says it has. + /// + [Fact] + public void Refuses_a_group_4_stream_that_stops_mid_line() + { + var payload = PdfImageFilters.Decode(Stream(Group4[..10], Params(k: -1)), 1 << 20, "ccitt.pdf"); + + Assert.False(payload.IsDecodable); + Assert.True(payload.Failed); + } + + /// Without a row count the decoder stops when the data does, which is how a fax stream ends. + [Fact] + public void Decodes_a_stream_without_a_row_count() + { + var payload = PdfImageFilters.Decode( + Stream(Group4, Params(k: -1, blackIs1: true, geometry: false), height: false), + 1 << 20, + "ccitt.pdf"); + + Assert.True(payload.IsDecodable); + Assert.Equal(Expected, payload.Data!.Value.ToArray()); + } + + /// A bitmap above the budget is measured and nothing is kept. + [Fact] + public void Measures_a_bitmap_above_the_budget_without_keeping_it() + { + var payload = PdfImageFilters.Decode(Stream(Group4, Params(k: -1)), 16, "ccitt.pdf"); + + Assert.True(payload.IsDecodable); + Assert.Equal(Width * Height / 8, payload.Length); + Assert.Null(payload.Data); + } + + private static byte[] Decode(byte[] encoded, PdfDictionary parameters) + { + var payload = PdfImageFilters.Decode(Stream(encoded, parameters), 1 << 20, "ccitt.pdf"); + + Assert.True(payload.IsDecodable); + Assert.False(payload.Failed); + Assert.Equal(Width * Height / 8, payload.Length); + + return payload.Data!.Value.ToArray(); + } + + private static PdfDictionary Params(int k, bool geometry = true, bool blackIs1 = false) + { + var parameters = new PdfDictionary().SetInteger("K", k); + + if (geometry) + { + parameters.SetInteger("Columns", Width).SetInteger("Rows", Height); + } + + if (blackIs1) + { + parameters.Set("BlackIs1", PdfBoolean.From(true)); + } + + return parameters; + } + + private static PdfStream Stream(byte[] encoded, PdfDictionary? parameters, bool height = true) + { + var dictionary = new PdfDictionary() + .SetName("Filter", "CCITTFaxDecode") + .SetInteger("Width", Width); + + if (height) + { + dictionary.SetInteger("Height", Height); + } + + if (parameters is not null) + { + dictionary.Set("DecodeParms", parameters); + } + + return new PdfStream(dictionary, encoded); + } +}