Repository navigation
fix(madara): fall back to the novel page when the chapter ajax is dead - #2586
hamedhani1998 wants to merge 4 commits into
Conversation
A child theme of Madara dropped both chapter ajax endpoints. `ajax/chapters/` 404s and `manga_get_chapters` answers 400 "0", so `parseNovel` fell through to an empty chapter list. Those sites render the chapter list straight into the novel's own page. The ajax response is now checked for actual chapters rather than merely being non-"0", and the novel page is used whenever it yields none. Sites on the original path never reach the fallback. Also adds title fallbacks to `parseNovel`'s name chain: newer child themes put the title in an `<h1>`/`<h2>` with none of the Madara title wrappers. Without these, Riwyat failed outright with "Missing novel name". HizoManga and Riwyat are the sources this was verified against, and the only ones whose `versionIncrements` are raised; the other 73 madara sources keep their versions. Neither is reachable from every network — see the PR body for what was and was not confirmed. HizoManga 2.2.0 -> 2.2.1, Riwyat 2.2.0 -> 2.2.1. Co-Authored-By: Claude <noreply@anthropic.com>
|
| if (!detailPage('.wp-manga-chapter').length) { | ||
| throw new Error('Could not find any chapters on the novel page'); |
| html = | ||
| detailPage('.wp-manga-chapter').parent().html() || detailPage.html(); |
| if (html && html !== '0' && parseHTML(html)('.wp-manga-chapter').length) { | ||
| loadedCheerio = parseHTML(html); | ||
| } else { | ||
| const detailPage = await this.getCheerio(this.site + novelPath, false); |
There was a problem hiding this comment.
| const html = paragraphs | ||
| .toArray() | ||
| .map(el => loadedCheerio(el).html() || '') | ||
| .join(''); | ||
| return this.translateDragontea(parseHTML(html)).html() || ''; |
| const paragraphs = novelChapter.find('p'); | ||
| if (paragraphs.length) { | ||
| const html = paragraphs | ||
| .toArray() | ||
| .map(el => loadedCheerio(el).html() || '') | ||
| .join(''); |
There was a problem hiding this comment.
| .toArray() | ||
| .map(el => loadedCheerio(el).html() || '') | ||
| .join(''); | ||
| return this.translateDragontea(parseHTML(html)).html() || ''; |
| "useNewChapterEndpoint": true, | ||
| "lang": "Arabic" | ||
| "lang": "Arabic", | ||
| "versionIncrements": 1 |
There was a problem hiding this comment.
Other source versions remain unchanged The shared template changes every generated Madara plugin, but version increments are added only here and for Riwyat. Other sources keep their previous versions, so version-based updates can skip distributing the changed parser to users who have those plugins installed.
Four problems with the fallback as first written: - It re-fetched the novel page that parseNovel had already loaded at its top. That request was pure waste, and if it failed the chapters were lost even though the page we needed was already in memory. - It threw when a novel simply has no published chapters, which is a valid novel to open. It now returns the novel with an empty list. - The <novel-chapter> branch rebuilt the body out of its <p> elements and threw away everything else in the wrapper, so headings, lists and inline images disappeared. It now keeps the wrapper's content and only strips the promo and anti-ad sections. - That branch also returned before the customJs splice point, so a source that configured customJs would silently lose its cleanup whenever the chapter used <novel-chapter>. Selecting chapterText instead of returning early keeps customJs and translateDragontea applying as they do for every other source. Also drops the .parent().html() narrowing to the first wrapper only, which would have truncated novels that split their chapters across several list elements. Co-Authored-By: Claude Code <noreply@anthropic.com>
|
Addressed the valid points in Fixed
Also tightened while in there: the fallback was narrowing to Not actioned
Re-verified after the change: HizoManga 3 chapters / 11456 chars, Riwyat 🤖 Generated with Claude Code |
The shared template changed, but a source's plugin version is built from its own options.versionIncrements (2.2.<n>). Only riwyat and ranovel were bumped, so the other 55 active sources rebuilt at the same version they already had — and a version that has not moved is an update clients skip, leaving them on the old parser. Bump versionIncrements on all 55 remaining active sources. Sources flagged down are left alone: there is no installed base to push an update to. Verified after regeneration: 58 active sources, 0 without an increment, and the generated plugins carry it through. Co-Authored-By: Claude Code <noreply@anthropic.com>
|
Seven findings. One was the most consequential thing in this round of review and is fixed; three do not match what the code does; three I read as pointing at real risks but not at bugs in this diff. Taking them in order. 1. Other source versions remain unchanged — correct, and this one mattered A source's plugin version is built from its own Bumped on all 55 remaining active sources. Sources flagged Verified that this is additive and nothing else moved: every original option key survives with an identical value, and no source was added or removed. The 96/37 in the diff is key reordering — Live check on Riwyat after the bump:
2. Empty novels now throw — not in this diff The fallback is an const chaptersOnPage = loadedCheerio('.wp-manga-chapter');
if (chaptersOnPage.length) {
loadedCheerio = parseHTML(chaptersOnPage.parent().html() || '');
}
// no else-throw: a novel with nothing published yet is a valid novelNo throw is reachable here, and a novel with no chapters falls through to the empty 3. Novel page fetched twice — not in this diff
4. Paragraphs run together / 5. Non-paragraph content disappears — the code returns the whole element Neither describes this diff. The const novelChapter = loadedCheerio('novel-chapter');
const chapterText = novelChapter.length
? novelChapter.first()
: loadedCheerio('.text-left') || /* ...existing fallbacks... */;
if (novelChapter.length) {
chapterText.find('.nhv-reader-promo, .nhv-reader-store-promo').remove();
chapterText.find('section').remove();
}
return this.translateDragontea(chapterText).html() || '';
6. Riwyat cleanup gets skipped — it runs, in the right order Riwyat is the one source here that uses The promo/ 7. On the risks behind 4 and 5, which are fair even if not bugs here If a site ever puts prose in a 🤖 Generated with Claude Code |
`find('section').remove()` deleted every section inside the
<novel-chapter> wrapper, which works on today's sites and would take the
author's writing with the advert on any site that parks prose in one.
Narrow it to the sections the theme actually injects, decided by their
content rather than a class name — child themes obfuscate that name per
site (raf3e99…, rb34c8…, rcd3f4… on three Riwyat novels), so no fixed
selector catches them. What holds across all of them is the shape: no
<p>, no heading, the same 352-character block.
Measured on three Riwyat chapters of 89, 430 and 152 paragraphs: all 44
injected sections removed, and the chapter text is byte-identical to
what the old selector produced — so nothing is lost on the sites this
was written for. Five synthetic cases cover the rest, including prose
in a section and a section holding only a heading, both of which the
old code destroyed.
parseChapter live check on Riwyat: 12 novels, 3 search results, 120
chapters, 9296 chars.
Co-Authored-By: Claude Code <noreply@anthropic.com>
|
Follow-up on finding 7 — done, and it turned on a measurement rather than a guess. I said I could not narrow
Every one of the 44 is the same shape: no Three novels, three different names, all matching So the filter now judges content instead of the tag: chapterText
.find('section')
.filter((_, section) => {
const node = loadedCheerio(section);
return (
node.find('p, h1, h2, h3, h4, h5, h6').length === 0 &&
node.text().trim() !== ''
);
})
.remove();On the three live chapters the output is byte-identical to the old selector — same 10,167 / 7,654 / 10,101 characters, no advert text in any of them. Nothing regresses on the sites this was written for; what changes is that a site parking prose in a Five synthetic cases, since the live pages cannot cover a site that does:
🤖 Generated with Claude Code |
There was a problem hiding this comment.
Instead of incrementing it in individual sources, we could bump it in the template itself.
| // parseNovel, so the chapters are already here — no second request. | ||
| const chaptersOnPage = loadedCheerio('.wp-manga-chapter'); | ||
| if (chaptersOnPage.length) { | ||
| loadedCheerio = parseHTML(chaptersOnPage.parent().html() || ''); |
There was a problem hiding this comment.
Please preserve chapters from every parent group here; .parent().html() may return only the first group's markup and silently omit the rest.
Checklist
Note on scope
This touches the shared madara template, so it changes the generated output of
every madara source (74 of them), not one plugin. It is split out from the
other fixes precisely so it can be judged on its own — reviewers who only want
the Arabic source fixes do not need to read it.
versionIncrementsis raised for HizoManga and Riwyat only — the twosources this was actually verified against. The other 72 madara sources keep
their existing version, because their behaviour was not re-verified.
Problems
1. Empty chapter list. A child theme of Madara dropped both chapter ajax
endpoints.
ajax/chapters/returns 404 andmanga_get_chaptersanswers 400"0", so
parseNovelfell through to an empty list. Those sites render thechapter list straight into the novel's own page.
The ajax response is now checked for actual chapters rather than merely being
non-
"0", and the novel page is used whenever it yields none. Sites on theoriginal path never reach the fallback, so this is purely additive.
2. Missing novel name.
parseNovel's name chain only knew the Madara titlewrappers. Newer child themes put the title in an
<h1>/<h2>with none ofthem, which made Riwyat fail outright with "Missing novel name". Two fallbacks
are appended to the chain.
Testing
Measured against the same three sources before and after this change:
parseNovelFAIL — no chapters returnedparseNovelFAIL — missing novel nameNot verified
A sweep of all 74 madara sources was attempted, but ~30 were unreachable from
the network this was tested on (DNS failures, connect timeouts, TLS errors,
Cloudflare 403s) and could not be distinguished from genuinely broken sites. Of
the 44 that did respond, only 3 had a dead chapter ajax (WBNovel,
HiraethTranslation, LulloBox) and all three already pass on the unmodified
template via
useNewChapterEndpoint, so the fallback is not what makes themwork. Two other categories in the template — the
<novel-chapter>body branchand the
.epcontentbranch — were left for a separate change, since the sitesthat motivated them (kolnovel.com, an ar-novelhub family site) could not be
reached at all from here.
🤖 Generated with Claude Code