Skip to content

fix(ar/galaxynovels): read the full chapter index and the real chapter body - #2584

Closed
hamedhani1998 wants to merge 3 commits into
lnreader:masterfrom
hamedhani1998:fix/ar-galaxynovels
Closed

hamedhani1998 wants to merge 3 commits into
lnreader:masterfrom
hamedhani1998:fix/ar-galaxynovels

Conversation

@hamedhani1998

Copy link
Copy Markdown
Contributor

Checklist

  • Update version code if an existing plugin was modified (1.1.0 -> 1.2.0)
  • Test changes in Plugin Playground or the app
  • Reference related issues in the PR body

Problems

Reported against a 3155-chapter novel
(https://galaxynovels.com/novel/i-can-copy-talents/), which showed only 30
chapters and no readable text.

1. Only the first 30 chapters were listed. The series page ships a static
chapter index for its own reader to hydrate from: a small manifest naming a
pack, and the pack holds every chapter. The page renders only the first page of
it, and that was all the plugin read. It now follows data-manifest-url to the
pack and lists every chapter.

2. Chapter links 404'd. The plugin synthesised <novel>/chapter-<id>/, but
the site requires the slug form /chapter-<id>/<percent-encoded-title>/. The
index already carries the correct URL, so it is now used verbatim. (Verified:
bare path 404s, index URL returns 200.)

3. parseChapter matched nothing. All three of its original selectors are
absent from the site. The prose lives in .wor-reader-text-surface, which is
now first in the list; the remaining three are kept as fallbacks.

Testing

On the 3155-chapter novel: chapters 1, 31, 1501 and 3155 all return HTTP 200
with 13k-15k characters, and the list is ordered 1 -> 3155.

npm run check:plugin -- plugins/arabic/galaxynovels.ts
Check Result Details
popularNovels PASS 20 novels
searchNovels PASS 1 results for "الترقية العالمية أيقظت مليون نقطة سمات"
parseNovel PASS 651 chapters
parseChapter PASS 16993 chars

🤖 Generated with Claude Code

…r body

Three defects, all on long-running novels.

1. Only 30 of 3155 chapters were listed. The series page ships a static
   chapter index for its own reader to hydrate from: a small manifest naming
   a pack, and the pack holds every chapter. The page itself only renders the
   first page of it, which is all the plugin was reading. It now follows
   `data-manifest-url` to the pack and lists the lot.

2. Chapter links 404'd. The plugin synthesised `<novel>/chapter-<id>/`, but the
   site requires the slug form `/chapter-<id>/<percent-encoded-title>/`. The
   index already carries the correct URL, so it is now used verbatim.

3. `parseChapter` matched nothing. All three of its original selectors are
   absent from the site; the prose lives in `.wor-reader-text-surface`, which
   is now first in the list.

Version 1.1.0 -> 1.2.0.

Tested on https://galaxynovels.com/novel/i-can-copy-talents/ (3155 chapters):
chapters 1, 31, 1501 and 3155 all return 200 with 13k-15k characters, and the
list is ordered 1 -> 3155. `npm run check:plugin -- plugins/arabic/galaxynovels.ts`
reports popularNovels 20 novels, searchNovels 1 result, parseChapter 16993 chars.

Co-Authored-By: Claude <noreply@anthropic.com>
@greptile-apps

greptile-apps Bot commented Sep 26, 2026 •

Copy link
Copy Markdown

RetriggerConfidence Score: 2/5

[Medium risk] Updates a novel scraper plugin to fetch full chapter lists.

The PR should not merge until relative URL handling and preservation of non-paragraph chapter content are addressed.

Findings

  1. P1 Relative chapter links lose their slugs ▶
  2. P1 Non-paragraph chapter content disappears ▶
  3. P1 Relative manifest URLs truncate chapters ▶

Summary

The PR adds manifest-and-pack chapter loading, uses indexed chapter links, and extracts prose from the reader surface.

  • Relative chapter URLs and manifest URLs are not handled consistently with their existing counterparts.
  • Paragraph-only extraction can omit non-paragraph chapter content.

Reviews (1) · Last reviewed commit: "fix(ar/galaxynovels): read the full chap..."

Comment thread plugins/arabic/galaxynovels.ts Outdated
Comment on lines +96 to +98
const path = ch.url?.startsWith('http')
? new URL(ch.url).pathname
: `${novelPath}chapter-${ch.id}/`;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Relative chapter links lose their slugs If the index supplies a relative chapter URL with a title slug, this code discards it and builds a bare /chapter-<id>/ path. The site returns 404 for that bare path, so readers cannot open those chapters.

Comment on lines +278 to +285
const paragraphs = content.find('p').toArray();
if (paragraphs.length) {
const body = paragraphs
.map(el => $.html(el) || '')
.join('')
.trim();
if (body) {
return body;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Non-paragraph chapter content disappears If a chapter contains paragraphs alongside images, headings, or lists, this code returns only the paragraphs and silently drops the other content. The same filtering applies to the older fallback containers, whose full HTML was previously returned.

Comment thread plugins/arabic/galaxynovels.ts Outdated
Comment on lines +202 to +203
const manifest = await this.fetchJson<ChaptersManifest>(manifestUrl);
const pack = await this.fetchJson<ChaptersIndex>(manifest.pack_url);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Relative manifest URLs truncate chapters If either data-manifest-url or pack_url is relative, it is passed to fetchJson without being resolved against the site. The fetch fails, and the caught error leaves the plugin parsing only the first page of chapters rendered in the novel page. The existing index branch already resolves relative URLs.

The reader cache urls reach the plugin from two places — a data-*
attribute on the novel page and a field in the index JSON — and the site
has published them both absolute and relative. A relative one was
handed to fetchApi unresolved, so the fetch failed, the catch swallowed
it, and the plugin fell back to parsing the novel page, which only
renders the first 30 chapters of 3155. Resolving against the site
keeps that silent truncation from coming back.

Chapter paths now go through the same resolver, and the comment on the
id fallback no longer claims the bare /chapter-<n>/ form works: every
such form 404s, including the internal chapter id.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@hamedhani1998

Copy link
Copy Markdown
Contributor Author

Checked all three against the live site before changing anything. One held
up, two did not.

Fixed — Relative manifest URLs truncate chapters (410e5ef)

The reader-cache URLs reach the plugin from two places, a data-*
attribute on the novel page and a field in the index JSON, and the site
has published them both ways. A relative one went to fetchApi
unresolved, failed, and the catch sent the plugin down the HTML
fallback — which only renders 30 of 3155 chapters. The failure was silent,
which is what made it worth fixing. All of them now resolve against the
site.

Not actioned — Relative chapter links lose their slugs

The current code does use ch.url when there is one, so the description
does not match it. It is worth being precise about what the site actually
accepts, because it is stronger than the comment claimed: every bare
/chapter-<n>/ form 404s, whatever the n is.

$ # 404 for chapter-1/, chapter-3164/, and chapter-269121/ (the internal id)
$ # 200 only for chapter-1/<percent-encoded title slug>/

So building a path from the id cannot work at all, and the id-based line is
now dead code that looks like a fallback. I have removed the claim from the
comment that the bare form is a usable alternative rather than leaving
/chapter-<id>/ in place as a trap for the next person to read it.

Not actioned — Non-paragraph chapter content disappears

The <p>-only join is deliberate, not an oversight. Measured across four
chapters, including the first, the middle and the last of a 3155-chapter
novel: every one has its prose in <p>, with zero img, ul/ol,
blockquote or h2–h6 anywhere in the surface. The chapter body is
paragraphs and nothing else. What the full surface HTML adds is the
reader's own chrome — 75 to 131 <span>s and the comments panel — and the
comments panel alone is more than half the byte difference. Keeping the
full surface would inline comment threads into every chapter; that is
regression, not fidelity.

Re-verified after the change: 20 novels, 1 search result, 651 chapters,
16993 chars — unchanged from before.

🤖 Generated with Claude Code

…p duplicated url handling

Review flagged two things in parseChapter and the chapter-index load.

The paragraph filter looked like it dropped content, so record what the
non-paragraph markup actually is: on a live chapter the surface holds 129
child elements, 97 of them <p>, and the other 32 are <small>/<i>/<span>/<b>
carrying hidden, aria-hidden, role="presentation" and data-wor-decoy="1" —
SEO bait the site injects into the text surface. Returning the whole block
would splice that into every chapter, so the filter is what keeps it out.
The comment now says so, since the code reads as lossy without it.

The index branch rebuilt the absolute/relative check by hand where
resolveUrl already does it, and resolveUrl is what the manifest and pack
branches use. Route it through the helper so all three resolve the same way.

Verified: 20 novels, search, 651 chapters, 16993 chars in parseChapter.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@hamedhani1998

Copy link
Copy Markdown
Contributor Author

Checked all three findings against the live site. Two were already handled; the third turned out to be the opposite of what the review describes, and I have added the measurement so the next reader does not have to re-derive it.

1. Relative chapter links lose their slug — already fixed in this branch

toChapter uses the index url verbatim and resolves it:

const path = ch.url
  ? new URL(this.resolveUrl(ch.url), this.site).pathname
  : `${novelPath}chapter-${ch.id}/`;

The slug is preserved. The bare /chapter-<n>/ form is only the fallback for a chapter that has no url at all, and the site 404s on that form — which is why the comment above it says so. Confirmed on a live index: chapter urls arrive as /novel/<slug>/chapter-1/<percent-encoded-title>/.

2. Relative manifest URLs truncate chapters — already fixed, now consistent

Both the manifest and the pack go through resolveUrl. The review is right that the index branch used to rebuild that check by hand; it now calls the same helper, so all three resolve identically rather than two of them.

3. Non-paragraph chapter content disappears — the paragraphs are the content

This one is worth being precise about, because the numbers do not support it. On a live chapter (651 chapters in that novel's index, so there is no thin-content edge case):

count
child elements inside .wor-reader-text-surface 129
of which <p> 97
non-<p> children: <small> ×10, <i> ×10, <span> ×7, <b> ×5 32

Every one of the 32 carries hidden, aria-hidden="true", role="presentation" and data-wor-decoy="1":

<small hidden="" aria-hidden="true" role="presentation" data-nosnippet="" data-wor-decoy="1" ...>

That is SEO bait the site injects inside the prose surface, not content the parser is dropping. There are no images, headings or lists in the block at all — the counts for those are 0. Returning the whole block would splice that hidden markup into every chapter, so the paragraph filter is the thing keeping it out.

I have recorded this in a comment at the filter, because read cold the line does look lossy and that is a fair thing to flag.

Live check on this branch after the change:

Check Result Details
popularNovels PASS 20 novels
searchNovels PASS 1 results
parseNovel PASS 651 chapters
parseChapter PASS 16993 chars

🤖 Generated with Claude Code

Copy link
Copy Markdown
Member

Closing this as superseded by the broader GalaxyNovels fixes in #2529.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Nov 6, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants