Repository navigation
Conversation
|
bro the markaz riwayat website is not even up... i need help with kol novel also, i am able to get the words and characters but the website itself is not up for me... |
|
bump |
|
Could we map only recognized status labels and leave unknown values unset? Any nonempty label outside these two Arabic strings is currently marked Completed. |
be436fd to
13ba9a5
Compare
|
Done in 63cf1d1, now on the PR head c87fddb after rebasing onto current
Re-checked after the update: This change was made by an AI coding agent; no human has reviewed it. |
| ]) { | ||
| const candidate = $(selector); | ||
| if (candidate.length && candidate.text().trim()) { | ||
| return candidate.html() || ''; |
There was a problem hiding this comment.
Can we restore the [data-wor-decoy] and [hidden] cleanup before extracting the chapter? Two live chapter bodies contained 28 and 25 hidden elements with text.
There was a problem hiding this comment.
Done in 5e19ac2: parseChapter removes [data-wor-decoy], [hidden] right after loading the page, before a candidate is chosen, so a candidate holding only hidden text no longer counts as the chapter.
Checked live on 9 chapters of 4 novels (first, middle and last chapter of each): the returned HTML had 26 to 63 hidden elements with text before the change and 0 after, with identical visible text. In the plugin playground the first chapter of "Sign in to become a saint" went from 30,821 to 17,307 characters (before/after screenshots are in the PR description). check:plugin passes.
This reply and the change were written by an AI coding agent; no human has reviewed them.
| '.c-blog-post > div > div:nth-child(2)', | ||
| ]) { | ||
| const candidate = loadedCheerio(selector); | ||
| if (candidate.text().trim() || candidate.find('img').length) { |
There was a problem hiding this comment.
Can we check for text or images after cleanup? A promo-only .text-left currently wins, then cleanup leaves it empty even when .reading-content contains the chapter.
There was a problem hiding this comment.
Done in 5e19ac2: the template now runs the source's customJs cleanup on a candidate and takes the first one that still has text or images afterwards. A rejected candidate is cleaned and the next one is read from the pristine page HTML, because some cleanups (Ranovel's) are not idempotent. Sources without customJs follow the same path as before. If every candidate is empty after cleanup it now throws the existing "Chapter text not found" error instead of returning an empty body.
Checked with fixtures: a promo-only .text-left before .reading-content (and before .entry-content) returned an empty string before and returns the chapter now. Every other selector case without customJs returns an identical body. Because the template is shared, I also compared the previous and new template on live chapters (first, middle, last) of the 22 Madara sources reachable from this machine (of 76): all 58 readable chapters of 20 sources were byte-identical. Live selectors seen were .text-left, .reading-content and .entry-content; the others were only covered by fixtures.
This reply and the change were written by an AI coding agent; no human has reviewed them.
| if (nhvStatus.includes('مستمرة')) novel.status = NovelStatus.Ongoing; | ||
| else if (nhvStatus.includes('متوقفة')) | ||
| novel.status = NovelStatus.OnHiatus; | ||
| else if (nhvStatus.includes('مكتملة')) |
There was a problem hiding this comment.
Can we match normalized status labels exactly? includes('مكتملة') classifies غير مكتملة (incomplete) as Completed.
There was a problem hiding this comment.
Done in 5e19ac2: the label is read from the value element only (.nhv-novel-status also contains the "الحالة" caption), normalized (NFKC, tatweel and diacritics removed, whitespace collapsed) and matched exactly against مستمرة, متوقفة and مكتملة. Anything else stays unset.
Live labels on cenele.com (261 novel pages): مستمرة 203, مكتملة 54, متوقفة 4, nothing else, so those map as before. غير مكتملة is not used on the site; with the change it stays unset instead of Completed (before: Completed). I left it unset rather than guess Ongoing or On Hiatus, since the site never produces it. Fixtures confirm غير مكتملة, with or without extra spaces, is unset, while مُكتملة and مكـتملة map to Completed.
This reply and the change were written by an AI coding agent; no human has reviewed them.
parseNovel preferred the data-chapter-id attribute (a WordPress post id) over the chapter link and built `…/chapter-<post-id>/` paths, which the site answers with 404 — only `…/chapter-<number>/<slug>/` resolves, so every chapter list it produced was unusable. parseChapter then asked `wp-json/wor-reader-app/v1/chapters/<id>` first; that route exists but Cloudflare answers 403 to every non-app client (reproduced from a real browser with a same-origin fetch), and the HTML fallback looked for selectors the theme no longer ships. Read the body from the article marked up as https://schema.org/Chapter instead, taking the first candidate that actually contains text. Fixes the chapter body for Galaxy Novels (issue lnreader#2522).
`$('.text-left') || $('.text-right') || …` never reaches the second
selector: a cheerio selection object is always truthy, so a page without
`.text-left` returned an empty chapter instead of trying the rest. Walk
the selectors and keep the first one that actually contains text, and add
the anchors the Madara-derived themes read from (`.text-content`,
`.text-chapter-content`, `novel-chapter`, `.reading-content`).
The riwyat source (cenele.com) also hides scraped-text decoys in
`<section inert data-nosnippet>` blocks that its inline CSS makes
invisible and wraps the body in a per-request random class, so its
customJs now drops those sections along with in-body style/script tags
and the app-promo blocks.
Fixes the chapter body for Riwyat (issue lnreader#2522).
Riwyat (cenele.com) migrated to a custom NHV theme whose novel pages no longer render the classic Madara detail blocks, so parseNovel returned an empty name, cover, summary, author and genres. Try each existing selector first and only fall through to the theme-specific markup (h1.nhv-novel-title, .nhv-novel-cover img, .nhv-novel-synopsis p, .nhv-novel-meta author links, .nhv-novel-genres a, .nhv-novel-status, .nhv-simple-rating__avg) when it yields nothing, leaving classic-theme sources unchanged. Pairs with the parseChapter selector-chain fix for the same source; this change stays out of parseChapter and sources.json. AI-authored change, no human review claimed.
…odies - madara: a chapter candidate that holds only images (illustration or comic chapters) counts as content again; the text-only test skipped it and returned an empty body where master rendered the images. - madara, lightnovelwp: throw a clear error when no chapter body is found instead of returning an empty string. - galaxynovels: throw instead of returning "Content not available." as if it were the chapter text.
The novel page only renders the latest 30 chapters and data-index-url is empty, so parseNovel now reads data-manifest-url from the chapters container, merges the pack file with live_tail the way the site reader does, and falls back to the page-derived list when the manifest is absent or malformed. This change was authored by an AI coding agent. Co-Authored-By: firstmate-crewmate <crewmate@firstmate.local>
…ers oldest first The NHV-theme status fallback mapped every non-ongoing novel to Completed, so novels the site marks متوقفة (stopped/on hold) showed as Completed; they now map to On Hiatus. Galaxy Novels' page-derived fallback list kept the page's newest-first order; it is now reversed so the oldest chapter comes first, like the manifest path. This change was authored by an AI coding agent.
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…rrors lost in rebase
… and status labels - galaxynovels: drop [data-wor-decoy] and [hidden] elements before choosing the chapter body. - madara: choose the first chapter candidate that still has text or images after the source's customJs cleanup; rejected candidates are cleaned on a throwaway copy of the page. - madara: match NHV status labels exactly after Unicode/diacritic/whitespace normalization, reading only the value element so "غير مكتملة" is no longer Completed.
c87fddb to
5e19ac2
Compare
Closes #2522, #2568, #2486, #2483, #2505, #2465, #2501, #2370, #1566, #2442, #2576, #2578, #2667, #2490
Opening a chapter in these Arabic sources showed "no content", even though the cover, description and chapter list still loaded. Each source broke for a different reason. Four are fixed. The fifth, Markazriwayat, could not be checked at all, so it is listed as not verified instead of fixed.
…/chapter-<post-id>/, which 404s), and the JSON endpoint the plugin preferred answers Cloudflare 403 to third-party clients$('.text-left') || …never reached its later selectors (a cheerio selection is always truthy), and the theme's invisible decoy blocks were returned as textbottomnavterminator that the theme dropped, and hard-coded a<div>wrapper where the theme serves<article>kolnovel.com, so the plugin requestedfree.kolnovel.com/https://kolnovel.com/…What master fixed separately. The Kol Novel family and TC & Sega "empty chapter" bodies (
bottomnavterminator,<article class="epcontent">) are fixed onmasterby #2593, #2611 and #2632, and this PR no longer carries that selector change. This PR still changesplugins/multisrc/lightnovelwp/template.tsfor two thingsmasterdoes not have: it keeps only the path when a chapter link is an absolute URL on another host (Free Kol Novel linkskolnovel.com, andparseChapterre-prefixes the site, givingfree.kolnovel.com/https://kolnovel.com/…), andparseChapterthrowsChapter text not found on the page, try to open in webview.instead of returning an empty string. The lightnovelwp template version goes to 1.2.12 and the madara one to 2.5.This PR also fixes Riwyat's novel metadata on the site's new theme, where name, cover, summary, author, genres, status and rating all came back empty. The fix adds fallbacks that run only after the existing selectors find nothing, so sources on the classic theme keep their current behaviour. Issues #2568, #2576 and #2578 report empty Kol Novel-family chapters; they come from the same template defect and this fix covers them.
Strengthening pass
A second pass reviewed the changes to the two shared templates (
madaraandlightnovelwp), because dozens of other sources use them. It made three small changes. Each one prevents a specific failure, shown before and after in the evidence below:<img>inside.text-lefthas no text, so this PR's first version returned an empty body wheremastershowed the images. A candidate that contains images now counts as content too.parseChapterused to return"". It now throwsChapter text not found on the page, try to open in webview.The error only replaces cases that returned an empty string, so any chapter that returned text before returns the same text.<p>Content not available.</p>as though it were the chapter, and a download would have saved that text. The plugin now throws instead.This pass made no other changes. Kol Novel and Free Kol Novel still cannot be reached from the machine used for testing (TCP timeout), so they remain verified through a proxy only.
Issue #2522
#2522 reports "no content" when opening a chapter on Kol Novel, Free Kol Novel and Riwyat (its labels name Galaxy Novels and Markazriwayat, but the report text names the Kol Novel family and Riwyat). Against today's live sites this PR covers it as follows:
check:pluginrun can reach them (fetch failed→ INCONCLUSIVE). The fix was re-verified on live chapter HTML fetched through a proxy: the old regex returns 0 chars and the new selector returns the full body (29,415 chars / 254 paragraphs on the sampled chapter). Free Kol Novel's cross-host chapter links were also re-confirmed.Currency (2026-09-28)
masterhas 16 newer commits since this branch's base. They touch the sharedmadaratemplate (a Translator-author fallback and a Summary fallback) andmadara/sources.json(add Tangerine Archive, drop two dead entries), plus unrelated plugins. None of them invalidate this change, the branch is still mergeable, and the head is unchanged (184b85a), so history was not rewritten. The stacked fork branch for the Galaxy full chapter index sits on an earlier point of this branch and is untouched.Galaxy Novels full chapter list, review fixes and screenshots (2026-10-11)
Full chapter list (#2667, #2490). The novel page only renders the latest 30 chapters, and its
data-index-urlis empty.parseNovelnow reads the chapter container'sdata-manifest-url. The manifest'spack_urlholds every chapter, and itslive_tailholds the newest ones. These are merged by chapter id the way the site's own reader merges them, then sorted oldest first. When the manifest is missing or malformed, the plugin falls back to the 30 chapters on the page. Galaxy Novels goes from 1.1.1 to 1.1.2. This is the change from RibatTRW#13, cherry-picked unchanged.Checked against the live site, comparing with each novel's manifest
total:Before this change chapter 1 was unreachable on every novel with more than 30 chapters.
chapterNumberis the manifestposition, so numbering is contiguous. The chapter name keeps the site's own label, which can differ where the site skips numbers (position 3155 is "الفصل 3164"). The fallback was exercised by making the manifest return 404 and by serving a malformed pack. Both returned the 30 page chapters, which is the behaviour these novels showed before this change.Review fixes. A code review of the whole diff found two defects, both fixed:
cont/surviving/showed Completed. That status now maps to On Hiatus. Completed ("مكتملة") and Ongoing ("مستمرة") are unchanged.Merge with master. The branch had fallen into conflict with
master. A normal merge commit brings it up to date, with no history rewrite or force push. The madara cover conflict is resolved asextractCoveron.summary_imagefirst, then the NHV cover, thendefaultCover.master's own lightnovelwp.epcontentfixes (#2593, #2632) already cover what this PR's selector change did, so the merged template keeps one copy. This PR's remaining lightnovelwp changes are the cross-host chapter-link fix and the "chapter text not found" errors.Checks on head
be436fd:npm run build:compileexits 0.npm run check:plugin:&.Screenshots. These come from this repo's plugin playground, run in headless Chromium, not from the LNReader app.
Galaxy Novels, before: 30 chapters, newest first.
Galaxy Novels, after: all 156 chapters, from chapter 1 to chapter 156.
Galaxy Novels, after: 3155 chapters on i-can-copy-talents, first and last.
Galaxy Novels, after: chapter 1 opens with its text (19,107 chars).
Riwyat, before and after the status fix: an on-hold novel showed Completed and now shows On Hiatus.
Riwyat, after: novel metadata on the NHV theme (cover, status, author, rating, genres, summary, 220 chapters) and a chapter body (13,426 chars).
Kol Novel and Free Kol Novel could not be screenshotted because both sites are down as of 2026-10-11. Both hostnames resolve to the same server (148.251.110.232). It does not answer on ports 80 or 443 from the test machine (TCP timeout), and a second, independent network path gets "connection refused" on 443. A regular browser or the app cannot reach them right now either, so the Kol Novel fixes stay verified only on the HTML captured through a proxy on 2026-09-28.
Known, not changed here. Riwyat chapter release dates show "Invalid Date". That comes from the shared madara date parsing, which this PR does not touch, and it happens on
mastertoo.Not closed
Verification evidence
First pass
npm run check:pluginshows popular 20 ✅, search 1 ✅, parseNovel 30 chapters ✅, parseChapter 13258 chars ✅ (was a 404).lightnovelwpfix was reproduced independently on a reachable source, TC & Sega:parseChapterFAIL (0 chars) → PASS 5905 chars.lightnovelwpsources before and after showed exactly one status change (TC & Sega, FAIL → PASS). 20madarasources were identical, and 12madarasources swept for the metadata fallbacks showed no status or body-length change.kolnovel.comandfree.kolnovel.comtime out over TCP from the test machine. They were verified by running the plugin's own extraction on live HTML fetched through a text proxy, which gave 14002 chars from the same chapter on both hosts. On a 12-paragraph sample, the old regex returned 0 chars and the new selector returned 1,006. At that time the sites themselves were up and only that network route timed out. As of 2026-10-11 both sites are down (see the Galaxy Novels section below).8522c771) atdc58567e: Galaxy NovelsparseChapterwent from FAILCould not reach site (404)to PASS with 49,502 chars, and RiwyatparseNovelwent from FAILMissing novel nameto PASS with full metadata, 100 chapters and a 9,002-char chapter body.KnoxT,ArNovel,LightNovelHeavenandNinjaidentical on head and base.UniversalNovelandMostNovelfail the same way on both, so those failures existed before this PR.Strengthening pass (head
184b85avsdc58567)Fixture before/after, calling the bundled plugins'
parseChapter:dc58567184b85a.text-leftholding only<img>(master returned the images)""<img>paragraphs""Chapter text not found on the page, try to open in webview..epcontent""<article class="epcontent">chapter<p>Content not available.</p>Chapter text not found on the page, open it in WebView.Full regression sweep over all 78
madaraand 36lightnovelwpsources plus Galaxy Novels. Every HTTP response was recorded once and replayed to both heads, so both saw identical pages. For each source the sweep ran popular → first novel → first chapter and compared status, name, author, genres, status, rating, summary length, cover, chapter count, body length and a hash of the body. 115 of 115 were identical. Reachability from the test machine:On every reachable
lightnovelwpchapter, the text extracted by the old regex and by this PR's selector was identical. Only the HTML serialization differs. On every reachable classicmadarachapter,.text-leftholds the text, so the selector walk still stops at.text-left, as it did before.npm run check:pluginon184b85a:fetch failed, TCP timeout on the test machine's route), still verified through a proxy onlynpm run build:compileexits 0.eslintandprettier --checkpass on all changed files.Re-verified 2026-09-28 (head
184b85a, no code changes)Direct probes:
galaxynovels.comHTTP 200,cenele.comHTTP 200,kolnovel.comandfree.kolnovel.comTCP timeout on 443 (fetch failed),markazriwayat.comHTTP 403 (Cloudflare challenge; reader proxy shows "Coming Soon").npm run check:plugin -- plugins/arabic/galaxynovels.ts): popular 20 ✅, search 1 ✅, parseNovel 30 chapters ✅, parseChapter 49502 chars ✅.npm run check:plugin -- "plugins/arabic/Riwyat[madara].ts"): popular 12 ✅, search 1 ✅, parseNovel ❌ FAIL "No chapters returned" — the harness checks the first popular novel (cont/pain-novel/), whose ownajax/chapters/endpoint currently answers "Manga has no chapter yet", so the site itself lists zero chapters there. Targeted runs on novels that have chapters still pass:cont/i-can-see-through-e/→ 120 chapters, full metadata (author, genres, Ongoing status, rating 8.4, cover, 905-char summary), first-chapter body 9,011 chars;cont/starting-over/→ 70 chapters, first-chapter body 634 chars.npm run check:pluginon both generated plugins):siteReachability(fetch failed) — the direct route still times out, so end-to-end verification remains impossible from here. Proxy re-verification on live HTML: the sampled Kol chapter (Chrysalis 162) serves<div class="epcontent entry-content">with nobottomnav; the old regex returns 0 chars and this PR's$('.epcontent').first().find('p')selector returns 29,415 chars (254 paragraphs). The Free novel page still lists chapter links as absolutekolnovel.comURLs, and the path normalization was unit-verified (old:free.kolnovel.com/https://kolnovel.com/…, new:/…); the same Free chapter's content was confirmed present through a reader proxy.npm run check:plugin -- plugins/arabic/Markazriwayat.ts):npm run build:compileexits 0;eslintandprettier --checkpass on all changed files.This PR was written by AI coding agents working for the PR author (the strengthening pass by Claude Opus 5.5, the earlier description refresh by Muse Spark, and the Galaxy full chapter list fold, review fixes, master merge and this update by Claude Opus 5.5). No human has reviewed the code, so please review it with that in mind.
Status mapping and master update (2026-11-04)
Addresses the review comment on the NHV status mapping and the merge conflicts with
master.plugins/multisrc/madara/template.tsthe NHV fallback now maps only recognized labels:مستمرةto Ongoing,متوقفةto On Hiatus andمكتملةto Completed. Any other label leavesstatusunset instead of Completed. (Correction, 2026-11-19:مكتملةdoes appear on live pages, see the maintainer-review section below.).manga-statusfallback. It ran whenever status was unset, and empty text counts as Completed, so an unknown NHV label would still have ended up Completed. It is now skipped when an.nhv-novel-statuslabel is present. Pages with no NHV label behave as before, and the older non-NHV status branches are unchanged.master, which removes the conflicts. The conflict ingalaxynovels.tswas resolved keeping bothmaster's changes and this PR's fixes.npm run build:compilepasses, and prettier passes. eslint reports only the existingno-explicit-anywarning intemplate.ts.check:plugin: Galaxy Novels passes (popular 20, search 1, parseNovel 168 chapters, parseChapter 26,798 chars). Riwyat's default novel has no chapters on the site ("There is no chapters"), so itsparseNovelcheck fails on that data; on another Riwyat novel it passes (302 chapters, 10,758-char body).lightnovelwp/template.tschanges. They are restored inc87fddbe: the cross-host chapter path fix, the two missing-body errors, and the version bumps (lightnovelwp 1.2.12, madara 2.5). Theepcontentselection is not restored becausemasteralready has it (fix(lightnovelwp): parse chapter text when bottomnav is absent #2593, fix(lightnovelwp): support article.epcontent chapters and fix search URL (#2483) #2632). Nothing else was dropped: the Galaxy Novels, Madara template and Madara sources changes are all still in the diff.check:pluginon KnoxT passes; Kol Novel and Free Kol Novel are still unreachable from the test machine (inconclusive).This section was written by an AI coding agent (Claude Sonnet 5.5). No human has reviewed the code.
Maintainer review follow-up (2026-11-19)
Addresses the three review comments of 2026-11-18. Commit
5e19ac27.Galaxy Novels hidden text.
parseChapterremoves[data-wor-decoy], [hidden]again before it picks the chapter body, so a candidate holding only hidden text no longer counts as the chapter. On 9 live chapters of 4 novels the returned HTML held 26 to 63 hidden elements with text before and 0 after, and the visible text was identical. Playground, same chapter (30,821 characters before, 17,307 after):Madara promo-only candidate.
parseChapternow runs the source'scustomJscleanup on a candidate and takes the first one that still has text or images afterwards. A rejected candidate is cleaned and the next one is read from the pristine page HTML, because some cleanups (Ranovel's) are not idempotent. Sources withoutcustomJsfollow exactly the old path. A promo-only.text-leftbefore.reading-contentreturned an empty string before and returns the chapter now. If every candidate is empty after cleanup it now throws the existing "Chapter text not found" error instead of returning an empty body.Madara status labels. The label is read from the value element only (
.nhv-novel-statusalso holds the "الحالة" caption), normalized (NFKC, tatweel and diacritics removed, whitespace collapsed) and matched exactly. Livecenele.com, 261 novel pages:مستمرة203,مكتملة54,متوقفة4, nothing else, so those three map as before.غير مكتملةdoes not occur on the live site; it now stays unset (before: Completed). I left it unset rather than guessing Ongoing or On Hiatus, since the site never uses it.How it was checked (machine with partial network reach; AI-written, no human review):
customJscase (.text-right,.text-content,.text-chapter-content,novel-chapter,.entry-content,.c-blog-post) is identical..text-left(54),.reading-content(3),.entry-content(1). The other selectors were checked only on fixtures. Morenovel and Ranovel (6 chapters) gave the same error before and after.check:pluginpasses for Galaxy Novels, Riwyat and ArNovel; lint, prettier andbuild:compilepass.Pipeline
Updates from git push no-mistakes
✅ **Intent** - passed
✅ No issues found.
✅ **Rebase** - passed
✅ No issues found.
✅ **Review** - passed
✅ No issues found.
✅ **Test** - passed
✅ No issues found.
node h/gal.mjs: galaxynovels parseChapter against synthetic pagesnode h/run.mjs: madara riwyat parseChapter and parseNovel against synthetic pages✅ **Document** - passed
✅ No issues found.
✅ **Lint** - passed
✅ No issues found.
✅ **Push** - passed
✅ No issues found.