Skip to content

fix(english/novelight): serve one chapter page per app page and add pagination regression test - #2527

Open
RibatTRW wants to merge 8 commits into
lnreader:masterfrom
RibatTRW:fm/lnreader-novelight-pagination
Open

RibatTRW wants to merge 8 commits into
lnreader:masterfrom
RibatTRW:fm/lnreader-novelight-pagination

Conversation

@RibatTRW

@RibatTRW RibatTRW commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Closes #2526.

Screenshots

Plugin playground (npm run dev:start) on this PR's head (af062b78), live novelight.net, novel book/our-guild-princess-is-good-at-everything, whose site select lists the ranges 1 - 11, 12 - 61, 62 - 111, 112 - 161. After-only: master maps in-range app pages to the same site ranges, so the playground cannot show a difference there.

Novel details parsed on this PR's head: title, status, genres and 4 total pages, before the first chapter page is fetched.

Playground: Novelight novel details with 4 total pages

Chapter page 1 of 4 on this PR's head: the site's oldest range, chapters 1-11.

Playground: Novelight chapter page 1 of 4 with chapters 1 to 11

Chapter page 2 of 4 on this PR's head: the next site range, 50 chapters starting at chapter 12.

Playground: Novelight chapter page 2 of 4 starting at chapter 12

What changes

All changes are in plugins/english/novelight.ts. The version goes from 1.1.5 to 1.1.6.

  • Pagination follows the site's own ranges. App page N is the Nth range from the site's chapter select, sorted oldest first (e.g. 1-14, 15-64, …). Each parsePage call makes exactly one chapter-pagination request. parseNovel gets totalPages from the same parsed select, so the page count and the pages served always agree.
  • Out-of-range pages return nothing. Page values come from the parsed select. Before this change, a page past the last one was clamped to the oldest site page and returned its chapters again as duplicates.
  • Browser-like headers on every request. Requests now send User-Agent, Referer and Accept-Language, because the site is behind Cloudflare.
  • Refused responses throw. A non-2xx response throws an error that carries the HTTP status, and a chapter-pagination reply without an html field also throws. A block page is no longer parsed into an empty result, so docs/testing.md reports it as INCONCLUSIVE, not FAIL.

Why one site range per app page

The endpoint accepts only page; the site's own static/book/js/single.js sends only {csrfmiddlewaretoken, book_id, page}. We probed page_size, per_page, limit, offset, start/end, paginate_by, count, size, length, pageSize, perPage and show, and the endpoint ignored all of them. The site fixes page width at 50 and anchors it at the newest chapter. An earlier revision re-sliced chapters into uniform 50s starting from the oldest, and that needed two requests for almost every page (review feedback above), so this revision serves the site's ranges as they are.

Trade-offs: the short remainder page now comes first instead of last. When new chapters are added, every page's contents shift, which the plugin already did before this PR; refresh or re-add the novel after updating.

Verification (what was actually run)

  • npm run check:plugin -- plugins/english/novelight.ts: all four steps pass (30 popular, 1 search result, 43-chapter page, ~12k-char chapter).

  • eslint and prettier --check on the file are clean. npm run build:compile is clean. Full-repo npm run lint has 2 existing errors in unrelated files.

  • A bundled-plugin script walked every app page against the live site and counted chapter-pagination requests. Every page made exactly 1 request, and no chapter was missing or duplicated.

    Novel Pages (sizes) Chapters
    Turning 30 [21, 50×29] 1471
    Regression Guidelines 10 [3, 50×9] 453
    Best Friend X Swap 5 [2, 50×4] 202
    Guild Princess 3 [17, 50, 50] 117
    isaac 5 [14, 50×4] 214
    genshin-impact-breezing-through-teyvat 8 [9, 50×7] 359
    hidden-dead-ends 2 [21, 50] 71
    86-eighty-six-vol-1 1 [34] 34
    my-daughter-has-become-a-fool 1 [14] 14

    Every page size matches the site's range labels. On the last five novels, the same script run against master gave the same in-range pages. On master, the page after the last one returned the oldest page's chapters again; on this branch it returns none.

  • Error paths: a missing novel throws Request failed: 404. Page abc, page 0 and pages past the last one return an empty list.

  • In the playground (npm run dev:start), popular, search, novel details (Turning: 30 pages), chapter page 1 (21 chapters, oldest first) and chapter reading all worked. The first chapter-read attempt failed with "Failed to fetch" while the test browser was crashing; it worked after relaunch.

Review pass

A fresh review checked the diff for off-by-one errors, missing or duplicated chapters, first and last pages, novels under 50 chapters, empty or invalid pages, error handling, the version bump and plugin conventions. It confirmed no defects, so it made no code changes.

Not tested

  • hideLocked: none of the tested novels has locked chapters.
  • A novel whose chapter count is an exact multiple of 50: none was found in a 28-novel scan. Pages come from the sorted parsed ranges, so no separate code path exists for that case.
  • Live-site behavior of the new regression check: it is network-free by design, so it does not replace check:plugin against novelight.net.

Regression check for the page mapping

Added in response to the review request (and Greptile P2) for a deterministic multi-page test.

  • scripts/novelight-pagination.test.mjs bundles the real plugins/english/novelight.ts with esbuild, replacing only @libs/fetch with an in-memory stub, so parseNovel and parsePage run unmodified with no network.
  • Fixture: a short 23-chapter oldest range plus four full 50-chapter ranges (223 chapters), listed newest first in the select as the site does.
  • It walks every app page and asserts: exactly one chapter-pagination request per page with the expected site page value, no chapter skipped or duplicated, ascending order across page boundaries, the short oldest range served first, totalPages matching the walk, and a page past the last returning no chapters and no request.
  • It fails when the mapping is broken: reversing the sort, or changing sitePages[pageNo - 1] to sitePages[pageNo], each turned it red (source restored afterwards).
  • Run locally with node --test scripts/novelight-pagination.test.mjs (Node 22+, no new dependencies).
  • CI: .github/workflows/novelight-pagination-test.yml runs it on pull requests that change the plugin, the test, or the workflow.
  • No plugin behavior or version change.

Pipeline

Updates from git push no-mistakes

✅ **Intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 3 of 3 scenarios driven live against the product
Scenario Result Live Evidence
Run the deterministic multi-page test against the real novelight plugin with a short oldest range and four full ranges, and it passes ✅ pass live node --test scripts/novelight-pagination.test.mjs: 1 pass, 0 fail
Break the page mapping (sitePages[pageNo]) and the test fails ✅ pass live Mutated run: 0 pass, 1 fail; source restored afterwards
Reverse the range sort order and the test fails ✅ pass live Mutated run: 0 pass, 1 fail; source restored afterwards
  • npm ci --ignore-scripts
  • node --test scripts/novelight-pagination.test.mjs (passes)
  • Mutation: changed sitePages[pageNo - 1] to sitePages[pageNo] (test fails)
  • Mutation: reversed the range sort (test fails)
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.


AI-authored change (including this review pass and description). No human review.

@RibatTRW

Copy link
Copy Markdown
Contributor Author

works also

@RibatTRW

Copy link
Copy Markdown
Contributor Author

Pushed v1.1.8 for this head: every request now goes through one helper that treats a refused response as a site block (reported INCONCLUSIVE per docs/testing.md) instead of parsing a block page into an empty catalog. Local npm run check:plugin -- plugins/english/novelight.ts passes fully (30 popular, search, 50 chapters, chapter text). The check run for this head is waiting on approval — could you approve it so the check can report?

@RibatTRW

Copy link
Copy Markdown
Contributor Author

The live check should work now on this head too. The previous failure class is a runner-side block being parsed into an empty result and reported as a plugin failure; every request here now carries browser-like headers and treats a refused response as a site block, which docs/testing.md defines as INCONCLUSIVE rather than a plugin FAIL. Verified against the live site (popular 30, search, 50 chapters per page, chapter text) and driven through the plugin playground in the app UI. The check run is waiting on approval — could you approve it so it can report?

@D3ICIDE

D3ICIDE commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

I dont think slicing each page to 50 chapters is a good idea because site's own page width isnt 50 chapters and slicing to 50 per page leads to 2 fetch request for nearly every page and since this site already does cf checks it may cause 429 or 403. Also why is the version bump to 1.1.8 instead of 1.1.6
image

@RibatTRW

Copy link
Copy Markdown
Contributor Author

I dont think slicing each page to 50 chapters is a good idea because site's own page width isnt 50 chapters and slicing to 50 per page leads to 2 fetch request for nearly every page and since this site already does cf checks it may cause 429 or 403. Also why is the version bump to 1.1.8 instead of 1.1.6 image

Yes you are right it, it calls for the page 2 times. I tested it internally and it worked out fine managed to fetch the chapters and its content, so the Cloudfare worry might not be true as per my data . But im not so sure about other devices and locals so ill try to find a work around. For the updates it was a chore bump due to many commits should be at 1.16 genuine mistake.

@RibatTRW
RibatTRW marked this pull request as draft September 25, 2026 12:11
@RibatTRW
RibatTRW marked this pull request as ready for review September 25, 2026 12:30
@RibatTRW
RibatTRW marked this pull request as draft September 25, 2026 12:33
@greptile-apps

greptile-apps Bot commented Sep 25, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no actionable defects were found.

Summary

Novelight now maps each app page to one site range, oldest first, and rejects refused HTTP responses.

  • Novelight chapter pages now follow the site’s ranges, oldest first.
  • Novelight requests now send browser headers and report refused replies.

Reviews (5) · Last reviewed commit: "test(english/novelight): add determinist..." · Reviewed by Greptile

Comment thread plugins/english/novelight.ts Resolved
@RibatTRW
RibatTRW marked this pull request as ready for review September 25, 2026 12:35
@RibatTRW
RibatTRW marked this pull request as draft September 25, 2026 12:52
@RibatTRW
RibatTRW marked this pull request as ready for review September 25, 2026 13:01
@RibatTRW
RibatTRW marked this pull request as draft September 25, 2026 13:50
@RibatTRW
RibatTRW marked this pull request as ready for review September 25, 2026 13:52
@D3ICIDE

D3ICIDE commented Sep 26, 2026

Copy link
Copy Markdown
Contributor

current version is 1.1.5 so the version bump should be to 1.1.6

@RibatTRW

Copy link
Copy Markdown
Contributor Author

Version is now 1.1.6 (commit 063e448). Disclosure: this is an AI-authored change with no human review.

(a, b) => a.start - b.start,
);
const pageNo = parseInt(page, 10);
const sitePage = sitePages[pageNo - 1];

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a deterministic multi-page regression check for this mapping so page boundaries cannot skip, duplicate, or reverse chapters.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a deterministic regression check in af062b7: scripts/novelight-pagination.test.mjs (node:test, no new dependency, no network). It bundles the real plugins/english/novelight.ts with only @libs/fetch stubbed and walks every app page of a fixture with a short 23-chapter oldest range plus four full 50-chapter ranges, with the select listed newest first. It asserts exactly one chapter-pagination request per page with the expected site page value, no chapter skipped or duplicated, ascending order across page boundaries, the short range served first, totalPages matching the walk, and no chapters (and no request) past the last page. Reversing the sort and changing sitePages[pageNo - 1] to sitePages[pageNo] each made it fail. CI runs it via .github/workflows/novelight-pagination-test.yml when the plugin or test changes. Run locally: node --test scripts/novelight-pagination.test.mjs. No plugin behavior or version change.
AI-authored change; no human review.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rajarsheechatterjee this is ready for another look: the deterministic multi-page regression check is in af062b78 and runs in CI.

RibatTRW and others added 8 commits November 4, 2026 06:38
`parseNovel` counted `#select-pagination-chapter > option` elements for
`totalPages`, while `parsePage` recomputed it from the last numeric option in
the whole document, and then mapped each app page onto a site page with
`totalPages - page + 1`. The two derivations could disagree, and the reversed
mapping made the site's partial page the FIRST page instead of the last: on
live novels page 1 returned 2, 6 and 49 chapters instead of 50. Because that
partition is anchored at the newest chapter, every added chapter shifted all
page boundaries, which the app's per-page chapter cache turns into duplicated
or missing chapters.

Both `parseNovel` and `parsePage` now derive the chapter count from the
largest range end in the site's page select (`ceil(total / 50)` pages), the
same shape as the sibling `mynovels` plugin for this CMS. `parsePage` fetches
only the site pages whose ranges overlap the requested 50-chapter chunk and
slices exactly that chunk, so pages 1..totalPages partition the chapter list:
every page but the last holds 50 chapters, and the partial page is last.
Anchoring the chunks at the oldest chapter also keeps already-served pages
stable when new chapters are added. Locked chapters are filtered after
slicing to keep positions stable; name, URL, date and locked-chapter handling
are unchanged.

Authored with AI assistance.
…ted site

popularNovels returned no novels in CI although the catalog markup and the
'a.item' selector are unchanged: the plugin sent no User-Agent at all, and
the Cloudflare-fronted site answers such requests with a bot-check page on
networks it distrusts (e.g. CI runners), which parses to zero novels. The
live check's own HEAD probe (which carries a Mozilla User-Agent) passed, so
the failure showed up as FAIL rather than INCONCLUSIVE.

Pass browser-like headers (User-Agent, Referer, Accept-Language) on every
fetchApi call, following the indraTranslations/dreamyTranslations pattern.
Chapter-list pagination behavior is unchanged. Local live check passes:
popularNovels 30 novels, searchNovels 1 result, parseNovel (parsePage) 50
chapters, parseChapter 12215 chars.

Refs lnreader#2526.

Authored with AI assistance.
Route every site request through one helper that throws (carrying the
HTTP status) on a non-OK response, so a runner-side block is reported
INCONCLUSIVE per docs/testing.md instead of being parsed into a false
'Returned no novels' FAIL. Headers and pagination behavior unchanged.

This commit was authored with AI assistance.
Replace the uniform oldest-anchored 50-chapter slicing (which straddled
two of the site's newest-anchored ranges on nearly every page, costing 2
chapter-pagination ajax requests) with a one-to-one mapping onto the
site's own ranges: app page N is the Nth site range in ascending chapter
order. The endpoint honors only the page parameter (page_size, per_page,
limit, offset, start/end, paginate_by, count, size, length, pageSize,
perPage, show all ignored; bookmark_chapter inert; the site's own JS
sends only csrfmiddlewaretoken, book_id and page), and the width is fixed
at 50 server-side, so the site's own partition is the only single-request
design. parseNovel derives totalPages from the same markup, keeping the
two in agreement, and requested values come straight from the parsed
select so out-of-range clamping can no longer duplicate chapters.

Trade-off: the short (remainder) page comes first instead of last, and
new chapters shift every page's contents, as before this PR.

Set the version to a single 1.1.5 -> 1.1.6 net bump.

This commit was authored with AI assistance.
- parseSitePageChapters takes a single params object instead of four
  positional strings
- throw a clear error when the chapter-pagination JSON has no html string
  instead of parsing undefined into zero chapters silently
- let endpoint failures propagate with their original stack; drop the
  console.error noise and the lossy throw new Error(error) wrapper
- drop the parsed-but-never-read ChapterPaginationOption.end field
- filter valueless pagination options, comment the non-range skip, drop
  the duplicated newest-first comment
- bump 1.1.6 to 1.1.7 (patch: behavior-preserving robustness fix)

Co-Authored-By: firstmate-crewmate (AI agent)
@RibatTRW
RibatTRW force-pushed the fm/lnreader-novelight-pagination branch from 063e448 to af062b7 Compare November 3, 2026 22:40
@RibatTRW RibatTRW changed the title fix(english/novelight): serve 50-chapter pages from the site's own page ranges fix(english/novelight): serve one chapter page per app page and add pagination regression test Nov 3, 2026
@RibatTRW

Copy link
Copy Markdown
Contributor Author

@greptileai review

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Pagination Issue - inconsistent & missing chapters

3 participants