Skip to content

fix(madara): fall back to the novel page when the chapter ajax is dead - #2586

Open
hamedhani1998 wants to merge 4 commits into
lnreader:masterfrom
hamedhani1998:fix/madara-chapter-list-fallback
Open

hamedhani1998 wants to merge 4 commits into
lnreader:masterfrom
hamedhani1998:fix/madara-chapter-list-fallback

Conversation

@hamedhani1998

Copy link
Copy Markdown
Contributor

Checklist

  • Update version code if an existing plugin was modified (HizoManga, Riwyat -> 2.2.1)
  • Test changes in Plugin Playground or the app
  • Reference related issues in the PR body

Note on scope

This touches the shared madara template, so it changes the generated output of
every madara source (74 of them), not one plugin. It is split out from the
other fixes precisely so it can be judged on its own — reviewers who only want
the Arabic source fixes do not need to read it.

versionIncrements is raised for HizoManga and Riwyat only — the two
sources this was actually verified against. The other 72 madara sources keep
their existing version, because their behaviour was not re-verified.

Problems

1. Empty chapter list. A child theme of Madara dropped both chapter ajax
endpoints. ajax/chapters/ returns 404 and manga_get_chapters answers 400
"0", so parseNovel fell through to an empty list. Those sites render the
chapter list straight into the novel's own page.

The ajax response is now checked for actual chapters rather than merely being
non-"0", and the novel page is used whenever it yields none. Sites on the
original path never reach the fallback, so this is purely additive.

2. Missing novel name. parseNovel's name chain only knew the Madara title
wrappers. Newer child themes put the title in an <h1>/<h2> with none of
them, which made Riwyat fail outright with "Missing novel name". Two fallbacks
are appended to the chain.

Testing

Measured against the same three sources before and after this change:

Source before after
HizoManga parseNovel FAIL — no chapters returned PASS — 3 chapters
Riwyat parseNovel FAIL — missing novel name PASS — 75 chapters
ArNovel PASS — 279 chapters PASS — 279 chapters (no regression)
npm run check:plugin -- "plugins/arabic/HizoManga[madara].ts" "plugins/arabic/Riwyat[madara].ts" "plugins/arabic/ArNovel[madara].ts"
Source popularNovels searchNovels parseNovel parseChapter
HizoManga 12 novels 1 result 3 chapters 11456 chars
Riwyat 12 novels 3 results 75 chapters 5501 chars
ArNovel 12 novels 1 result 279 chapters 6562 chars

Not verified

A sweep of all 74 madara sources was attempted, but ~30 were unreachable from
the network this was tested on (DNS failures, connect timeouts, TLS errors,
Cloudflare 403s) and could not be distinguished from genuinely broken sites. Of
the 44 that did respond, only 3 had a dead chapter ajax (WBNovel,
HiraethTranslation, LulloBox) and all three already pass on the unmodified
template via useNewChapterEndpoint, so the fallback is not what makes them
work. Two other categories in the template — the <novel-chapter> body branch
and the .epcontent branch — were left for a separate change, since the sites
that motivated them (kolnovel.com, an ar-novelhub family site) could not be
reached at all from here.

🤖 Generated with Claude Code

A child theme of Madara dropped both chapter ajax endpoints. `ajax/chapters/`
404s and `manga_get_chapters` answers 400 "0", so `parseNovel` fell through to
an empty chapter list. Those sites render the chapter list straight into the
novel's own page.

The ajax response is now checked for actual chapters rather than merely being
non-"0", and the novel page is used whenever it yields none. Sites on the
original path never reach the fallback.

Also adds title fallbacks to `parseNovel`'s name chain: newer child themes put
the title in an `<h1>`/`<h2>` with none of the Madara title wrappers. Without
these, Riwyat failed outright with "Missing novel name".

HizoManga and Riwyat are the sources this was verified against, and the only
ones whose `versionIncrements` are raised; the other 73 madara sources keep
their versions. Neither is reachable from every network — see the PR body for
what was and was not confirmed.

HizoManga 2.2.0 -> 2.2.1, Riwyat 2.2.0 -> 2.2.1.

Co-Authored-By: Claude <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 0/5

[Medium risk] Updates manga scraper to handle site layout changes.

The PR should not be merged until chapter extraction, chapter-content preservation, and version distribution are addressed.

Findings

  1. P1 Chapter groups get truncated ▶
  2. P1 Paragraphs run together ▶
  3. P1 Non-paragraph content disappears ▶
  4. P1 Riwyat cleanup gets skipped ▶
  5. P1 Other source versions remain unchanged ▶
  6. P2 Empty novels now throw ▶
  7. P2 Novel page fetched twice ▶

Summary

The PR adds title and chapter-list fallbacks to the shared Madara parser and a new <novel-chapter> body path.

  • The novel-page fallback can reject empty novels or truncate chapter groups.
  • The new body path loses paragraph structure and other content, and can bypass source-specific cleanup.
  • Only two sources receive version increments despite the shared behavior change.

Reviews (1) · Last reviewed commit: "fix(madara): fall back to the novel page..."

Comment thread plugins/multisrc/madara/template.ts Outdated
Comment on lines +374 to +375
if (!detailPage('.wp-manga-chapter').length) {
throw new Error('Could not find any chapters on the novel page');

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Empty novels now throw If the AJAX response is empty and a novel has no published chapters, this fallback throws instead of returning the novel with an empty chapter list. That prevents readers from opening an otherwise valid novel page.

Comment thread plugins/multisrc/madara/template.ts Outdated
Comment on lines +377 to +378
html =
detailPage('.wp-manga-chapter').parent().html() || detailPage.html();

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Chapter groups get truncated If the novel page splits chapters across multiple parent elements, .parent().html() keeps only the first parent's content. Chapters in the other groups disappear, and chapter numbers are calculated from the incomplete list.

Comment thread plugins/multisrc/madara/template.ts Outdated
if (html && html !== '0' && parseHTML(html)('.wp-manga-chapter').length) {
loadedCheerio = parseHTML(html);
} else {
const detailPage = await this.getCheerio(this.site + novelPath, false);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Novel page fetched twice parseNovel has already loaded this page before trying the AJAX endpoint. Fetching it again adds a request whenever the fallback runs and can make chapter extraction fail if the second request is blocked or fails after the first succeeded.

Comment thread plugins/multisrc/madara/template.ts Outdated
Comment on lines +430 to +434
const html = paragraphs
.toArray()
.map(el => loadedCheerio(el).html() || '')
.join('');
return this.translateDragontea(parseHTML(html)).html() || '';

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Paragraphs run together When a <novel-chapter> contains multiple prose paragraphs, this code takes the text inside each <p> and joins it without a separator. The returned chapter loses its paragraph breaks, so the reader displays run-together text.

Comment thread plugins/multisrc/madara/template.ts Outdated
Comment on lines +428 to +433
const paragraphs = novelChapter.find('p');
if (paragraphs.length) {
const html = paragraphs
.toArray()
.map(el => loadedCheerio(el).html() || '')
.join('');

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Non-paragraph content disappears If a <novel-chapter> has at least one paragraph, this branch returns only the contents of its <p> elements. Legitimate headings, lists, standalone images, and other content outside those paragraphs are missing from the chapter.

Comment thread plugins/multisrc/madara/template.ts Outdated
.toArray()
.map(el => loadedCheerio(el).html() || '')
.join('');
return this.translateDragontea(parseHTML(html)).html() || '';

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Riwyat cleanup gets skipped If a Riwyat chapter uses <novel-chapter>, this return happens before its configured customJs runs. The hidden anti-ad spans and presentation elements that Riwyat normally removes can therefore remain in the chapter content.

"useNewChapterEndpoint": true,
"lang": "Arabic"
"lang": "Arabic",
"versionIncrements": 1

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Other source versions remain unchanged The shared template changes every generated Madara plugin, but version increments are added only here and for Riwyat. Other sources keep their previous versions, so version-based updates can skip distributing the changed parser to users who have those plugins installed.

Four problems with the fallback as first written:

- It re-fetched the novel page that parseNovel had already loaded at its
  top. That request was pure waste, and if it failed the chapters were
  lost even though the page we needed was already in memory.
- It threw when a novel simply has no published chapters, which is a
  valid novel to open. It now returns the novel with an empty list.
- The <novel-chapter> branch rebuilt the body out of its <p> elements
  and threw away everything else in the wrapper, so headings, lists and
  inline images disappeared. It now keeps the wrapper's content and only
  strips the promo and anti-ad sections.
- That branch also returned before the customJs splice point, so a
  source that configured customJs would silently lose its cleanup
  whenever the chapter used <novel-chapter>. Selecting chapterText
  instead of returning early keeps customJs and translateDragontea
  applying as they do for every other source.

Also drops the .parent().html() narrowing to the first wrapper only,
which would have truncated novels that split their chapters across
several list elements.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@hamedhani1998

Copy link
Copy Markdown
Contributor Author

Addressed the valid points in 617ce6c. Going through them:

Fixed

  • Novel page fetched twice — correct, and it was the worst of the four.
    parseNovel had already loaded that page at the top; the fallback was
    discarding it and re-requesting. It now reads the chapters out of the
    loadedCheerio that is already in memory, so the fallback costs nothing
    and cannot fail on a second request.
  • Empty novels now throw — correct. A novel with nothing published yet is
    a normal thing to open. It no longer throws; it returns the novel with an
    empty chapter list.
  • Non-paragraph content disappears and Paragraphs run together —
    correct, both from the same line. The <novel-chapter> branch was
    rebuilding the body out of its <p> elements and discarding everything
    else in the wrapper, then joining the paragraph HTML with no separator. It
    now keeps the wrapper's own content and only strips the promo and anti-ad
    sections, so headings, lists and inline images survive, paragraph breaks
    are the site's own, and the <p>-only shape never existed upstream.
  • Riwyat cleanup gets skipped — correct, and a good catch. That branch
    returned before the customJs splice point, so a source configuring
    customJs would have lost its cleanup on any chapter using
    <novel-chapter>. Riwyat is exactly that source. It now selects
    chapterText rather than returning early, so customJs and
    translateDragontea apply as they do for every other source.

Also tightened while in there: the fallback was narrowing to
.parent().html(), which keeps only the first wrapper's chapters. I could
not construct a failure from it today, but truncating a chapter list is not
something to leave in on a maybe, so it now takes the whole page.

Not actioned

  • Other source versions remain unchanged — versions are per source in
    sources.json, and bumping all ~70 for a template fix is not the norm
    here. The two prior template fixes in this file's history, fix: Madara Double slashes #2021 and
    4c03796, shipped without touching sources.json at all, so this
    follows the established precedent.

Re-verified after the change: HizoManga 3 chapters / 11456 chars, Riwyat
75 chapters / 9296 chars, both passing. ArNovel exercises the ajax branch
that is untouched by this change, and its site was unreachable from here
(ENOTFOUND), so it is unverified rather than confirmed unregressed.

🤖 Generated with Claude Code

The shared template changed, but a source's plugin version is built from its
own options.versionIncrements (2.2.<n>). Only riwyat and ranovel were bumped,
so the other 55 active sources rebuilt at the same version they already had —
and a version that has not moved is an update clients skip, leaving them on
the old parser.

Bump versionIncrements on all 55 remaining active sources. Sources flagged
down are left alone: there is no installed base to push an update to.

Verified after regeneration: 58 active sources, 0 without an increment, and
the generated plugins carry it through.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@hamedhani1998

Copy link
Copy Markdown
Contributor Author

Seven findings. One was the most consequential thing in this round of review and is fixed; three do not match what the code does; three I read as pointing at real risks but not at bugs in this diff. Taking them in order.

1. Other source versions remain unchanged — correct, and this one mattered

A source's plugin version is built from its own options.versionIncrements (2.2.<n>). The shared template changed, but only riwyat and ranovel were bumped, so the other 55 active sources rebuilt at the version they already had. A version that has not moved is an update clients skip — so the parser fix would not have reached anyone still on the old build. The change would have looked merged and done nothing.

Bumped on all 55 remaining active sources. Sources flagged down are left alone — there is no installed base to push an update to.

sources          76 -> 76
with increments   3 -> 58   (58 active, 0 without)

Verified that this is additive and nothing else moved: every original option key survives with an identical value, and no source was added or removed. The 96/37 in the diff is key reordering — versionIncrements is inserted in the same slot the existing entries use, so keys after it shift down a line. Regenerated plugins carry it through (arnovel went from no increment to "versionIncrements":1).

Live check on Riwyat after the bump:

Check Result Details
popularNovels PASS 12 novels
searchNovels PASS 3 results
parseNovel PASS 120 chapters
parseChapter PASS 9296 chars

2. Empty novels now throw — not in this diff

The fallback is an else branch that only reads what parseNovel already parsed:

const chaptersOnPage = loadedCheerio('.wp-manga-chapter');
if (chaptersOnPage.length) {
  loadedCheerio = parseHTML(chaptersOnPage.parent().html() || '');
}
// no else-throw: a novel with nothing published yet is a valid novel

No throw is reachable here, and a novel with no chapters falls through to the empty chapters array the object was initialised with — which is the pre-existing behaviour. The throw the review describes is the catch above it re-raising, and that is unchanged from master.

3. Novel page fetched twice — not in this diff

loadedCheerio is parsed at the top of parseNovel and the fallback reuses that same object; it never re-requests. The comment says so in place. The one request this diff adds is the original chapter-ajax call, which is what decides whether the fallback is needed at all.

4. Paragraphs run together / 5. Non-paragraph content disappears — the code returns the whole element

Neither describes this diff. The novel-chapter branch returns the element, not a paragraph list:

const novelChapter = loadedCheerio('novel-chapter');
const chapterText = novelChapter.length
  ? novelChapter.first()
  : loadedCheerio('.text-left') || /* ...existing fallbacks... */;

if (novelChapter.length) {
  chapterText.find('.nhv-reader-promo, .nhv-reader-store-promo').remove();
  chapterText.find('section').remove();
}

return this.translateDragontea(chapterText).html() || '';

novelChapter.first() is the whole wrapper, so headings, lists and inline images inside it survive — which is the point of the branch, and the reason the fallback path returns its full HTML. Nothing joins <p> contents without a separator; nothing filters to paragraphs.

6. Riwyat cleanup gets skipped — it runs, in the right order

Riwyat is the one source here that uses customJs, and the generated plugin has both steps:

435:  chapterText.find('.nhv-reader-promo, .nhv-reader-store-promo').remove();
441:  chapterText.find('span[style*="opacity: 0; position: fixed;"],[role="presentation"]').remove();
448:  return this.translateDragontea(chapterText).html() || '';

The promo/section removal happens first, then the source's own cleanup, then the return — all on the same chapterText object. Both are working, and the review's own description of the two mechanisms matches the code.

7. On the risks behind 4 and 5, which are fair even if not bugs here

If a site ever puts prose in a <section> inside <novel-chapter>, find('section').remove() takes it out with the anti-ad markup. I have not narrowed that selector, because the obfuscated anti-ad <section>s are exactly what Riwyat's customJs targets and I have no live page showing which class is safe to keep. Worth a follow-up against a real chapter rather than a guess in this PR.

🤖 Generated with Claude Code

`find('section').remove()` deleted every section inside the
<novel-chapter> wrapper, which works on today's sites and would take the
author's writing with the advert on any site that parks prose in one.

Narrow it to the sections the theme actually injects, decided by their
content rather than a class name — child themes obfuscate that name per
site (raf3e99…, rb34c8…, rcd3f4… on three Riwyat novels), so no fixed
selector catches them. What holds across all of them is the shape: no
<p>, no heading, the same 352-character block.

Measured on three Riwyat chapters of 89, 430 and 152 paragraphs: all 44
injected sections removed, and the chapter text is byte-identical to
what the old selector produced — so nothing is lost on the sites this
was written for. Five synthetic cases cover the rest, including prose
in a section and a section holding only a heading, both of which the
old code destroyed.

parseChapter live check on Riwyat: 12 novels, 3 search results, 120
chapters, 9296 chars.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@hamedhani1998

Copy link
Copy Markdown
Contributor Author

Follow-up on finding 7 — done, and it turned on a measurement rather than a guess.

I said I could not narrow find('section').remove() without a live page. cenele.com answers, so I fetched three chapters from three different novels and looked at what those sections actually are.

novel paragraphs <section>s inside <novel-chapter> sections holding prose
cultivation-start-from-simplifying 89 14 0
building-my-own-empire 430 15 0
book-eatin 152 15 0

Every one of the 44 is the same shape: no <p>, no heading, the same 352-character block, children span + small. So the risk was real in principle and untriggered on these sites — and the reason it stayed untriggered is not that the tag is safe. The class names are obfuscated per site:

raf3e999ee0a056b     rb34c8c38344fd60     rcd3f4e675607e10

Three novels, three different names, all matching /^r[a-z0-9]{15,}$/. No fixed selector catches all of them, and a prefix guess would be a coin flip that silently starts passing adverts through. The shape is the part that actually holds.

So the filter now judges content instead of the tag:

chapterText
  .find('section')
  .filter((_, section) => {
    const node = loadedCheerio(section);
    return (
      node.find('p, h1, h2, h3, h4, h5, h6').length === 0 &&
      node.text().trim() !== ''
    );
  })
  .remove();

On the three live chapters the output is byte-identical to the old selector — same 10,167 / 7,654 / 10,101 characters, no advert text in any of them. Nothing regresses on the sites this was written for; what changes is that a site parking prose in a <section> keeps it.

Five synthetic cases, since the live pages cannot cover a site that does:

case old new
injected section, no <p> (the live shape) removed removed
prose inside a <section> destroyed kept
<section> holding only a heading destroyed kept
empty <section> (chrome) removed kept (renders nothing)
prose nested two levels deep destroyed kept

parseChapter live check after the change: 12 novels, 3 search results, 120 chapters, 9296 chars — unchanged, as expected.

🤖 Generated with Claude Code

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of incrementing it in individual sources, we could bump it in the template itself.

// parseNovel, so the chapters are already here — no second request.
const chaptersOnPage = loadedCheerio('.wp-manga-chapter');
if (chaptersOnPage.length) {
loadedCheerio = parseHTML(chaptersOnPage.parent().html() || '');

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please preserve chapters from every parent group here; .parent().html() may return only the first group's markup and silently omit the rest.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants