Skip to content

Phase 3: consent banners, video and PDF thumbnails, smaller PDFs - #15

Open
voslucas wants to merge 2 commits into
mainfrom
phase3-screenshots
Open

voslucas wants to merge 2 commits into
mainfrom
phase3-screenshots

Conversation

@voslucas

Copy link
Copy Markdown
Contributor

Why

Phase 3 of docs/PLAN.md. Screenshots showed cookie banners, YouTube links showed a screenshot of the player, PDF links had no picture (issue #1), and the PDFs were 12–13 MB.

What

  • Consent banners (consent.py): before a screenshot, click a known consent-manager button (OneTrust, Cookiebot, Didomi, Funding Choices, Quantcast, Sourcepoint) or a consent-like button inside a cookie/consent container. Reject / necessary-only is preferred, accept is the fallback. It also searches iframes and open shadow roots (WPConsent), then hides remaining fixed consent overlays.
  • Screenshots are JPEG (quality 80), taken after a 5 s network-idle wait, which also fixes lazily rendered pages.
  • VideoHandler (replaces YoutubeHandler): handles youtu.be, /shorts/, /live/, /embed/, m.youtube and Vimeo.
    • YouTube: Data API with a key (channel, date, duration, description), oEmbed without one (channel and thumbnail).
    • Vimeo: oEmbed. Videos get the HD thumbnail instead of a player screenshot, and a removed video falls back to a screenshot.
  • PDF links (closes Add PDF support #1): the first page is rendered with pypdfium2, with page count and host. arXiv /pdf/ links are included, and a download that isn't a PDF is treated as a web page.
  • Images are stored as baseline JPEG (<index>-img.jpg, transparency on white). Older <index>.png caches still work.
  • utm_* parameters are stripped from article and comment links.
  • The template no longer prints a "…" after articles without text.
  • New dependency: pypdfium2.

Results on #814 (local, Windows)

before after
screenshots with a consent banner 7 / 63 0 / 63
blank / unfinished pages 4 0
PDF size 12.9 MB 6.6 MB
build time (generator) ~3 min ~4 min
TeX errors 0 0

13 pages had a banner handled: 11 by reject/decline, 2 by accept because those sites only offer accept. Still visible: bot walls on 3–4 sites (noted in PLAN).

Testing

  • pytest: 55 passed (24 new: video URL variants, durations, Vimeo ids, the PDF/arXiv test, strip_tracking, image conversion, find_image, PDF page rendering). ruff check and ruff format --check are clean.
  • Live handler check: YouTube without a key (oEmbed), Vimeo (Big Buck Bunny: author, date, duration, description), two PDFs (41 and 245 pages), all with thumbnails.
  • Full local #814 build and contact sheets of all 63 screenshots, reviewed by eye.
  • Not yet: a Linux CI build from this branch.

🤖 Generated with Claude Code

voslucas and others added 2 commits October 20, 2026 12:22
- consent.py: before a screenshot, click a known consent-manager button
  or a consent-like button, preferring reject/necessary-only over
  accept, in the page, its iframes and open shadow roots; hide what is
  left of fixed consent overlays. #814: banners in 7 -> 0 screenshots
- screenshots are JPEG (quality 80) after a 5 s network-idle wait,
  which also fixes lazily rendered pages; #814 PDF 12.9 -> 6.6 MB
- VideoHandler replaces YoutubeHandler: youtu.be, shorts, live, embed,
  m.youtube and Vimeo; YouTube Data API (with duration) or oEmbed
  without a key, Vimeo oEmbed; HD thumbnail instead of a screenshot of
  the player; removed videos fall back to a screenshot
- PDFHandler renders the first page with pypdfium2 (issue #1), shows
  page count and host, also for arxiv /pdf/ links; a download that is
  not a pdf is treated as a web page
- download_image stores any image format as a baseline JPEG
  (<index>-img.jpg, transparency on white); find_image picks the
  article picture, older <index>.png caches still work
- utm_* parameters are stripped from article and comment links
- template: no "..." after articles without text
- 24 new tests (55 total)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add PDF support

1 participant