Skip to content

fix(scripts): report Cloudflare-challenged live-check steps as inconclusive - #2674

Open
RibatTRW wants to merge 3 commits into
lnreader:masterfrom
RibatTRW:fm/lnreader-plugins-livecheck-cf-inconclusive
Open

RibatTRW wants to merge 3 commits into
lnreader:masterfrom
RibatTRW:fm/lnreader-plugins-livecheck-cf-inconclusive

Conversation

@RibatTRW

@RibatTRW RibatTRW commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

On GitHub's runners Cloudflare can block every HTML page of a source, while the checker's root reachability probe (HEAD of the site root) still succeeds. fetchApi returns the Response for any status and never throws, so a Cloudflare challenge page parses to zero novels and the live check reports FAIL ("Returned no novels") instead of "could not test". This makes the shared Plugin Live Check report those steps as INCONCLUSIVE. Failing run that motivated it: https://github.com/lnreader/lnreader-plugins/actions/runs/38055684264.

What Changed

  • scripts/live-check-plugin.js now wraps the global fetch during each step to spot Cloudflare challenge responses (cf-mitigated: challenge, or a 403/503 carrying cf-* headers) on the plugin's own page loads. A step that would otherwise FAIL after such a response is reclassified as INCONCLUSIVE with a "Blocked by Cloudflare" detail. Passing steps and failures with no observed challenge are unchanged.
  • The script now exports describeCloudflareChallenge, installChallengeDetector and checkPlugin, and only runs main() when executed directly, so it can be imported by tests.
  • Added scripts/live-check-plugin.test.mjs, which covers the new detection and reclassification logic. Updated the INCONCLUSIVE description in docs/testing.md to mention challenge pages served for a plugin's page loads.

🤖 Generated with Claude Code

Risk Assessment

✅ Low: The change is confined to the live-check script and only downgrades FAIL to INCONCLUSIVE when a Cloudflare challenge response was seen during that step; its tests run the real checker against a local server.

Testing

The tests in scripts/live-check-plugin.test.mjs pass. They run the real checkPlugin against local HTTP servers that serve a challenge as 403 or 200, a 403 with Cloudflare headers, an empty result, a plain 403 and a passing page. I also ran the checker as a CLI against a local server that returns a Cloudflare challenge on its list page while the root probe succeeds. The report showed INCONCLUSIVE ("Blocked by Cloudflare…"), "no hard failures" and exit code 0. I did not test against the real royalroad.com. The worktree stayed clean.

  • Live validation: ✅ go - 6 of 6 scenarios driven live against the product
Scenario Result Live Evidence
Challenge page (403 with cf-mitigated) on a list load gives INCONCLUSIVE ✅ pass live node --test output and CLI report
Challenge served as HTTP 200 gives INCONCLUSIVE ✅ pass live node --test
403 with Cloudflare headers gives INCONCLUSIVE ✅ pass live node --test
Adversarial: an empty result with a plain cf-ray header, or a plain 403, still FAILs ✅ pass live node --test
Adversarial: a challenged response on a step that passed stays PASS ✅ pass live node --test
CLI entry still runs and exits 0 on an inconclusive result ✅ pass live CLI transcript
Evidence: CLI report for a challenged plugin
| `popularNovels` | ⚠️ INCONCLUSIVE | Blocked by Cloudflare: HTTP 403 (Cloudflare challenge) (was: Returned no novels) |
**Result: ⚠️ Inconclusive — no hard failures.**

Review follow-up: late responses and test wiring

  • installChallengeDetector now reads currentStep before awaiting originalFetch, so a response is attributed to the step that sent the request (review comment from @rajarsheechatterjee and the matching Greptile P1).
  • New regression test in scripts/live-check-plugin.test.mjs: a first step's request is still in flight when the step ends, the next step starts, then the first request returns a Cloudflare challenge. The later step's real failure must stay FAIL. It uses a local server gate, no timers. Against the old order it fails (6 of 7 pass); with the fix all 7 pass.
  • .github/workflows/live-check-tests.yml runs node --test scripts/live-check-plugin.test.mjs on pull requests that touch the checker, its tests or the workflow (Greptile P2: the tests were not run anywhere). The existing live-check workflow is unchanged.
  • Commit authored with AI assistance (Claude Code).

Pipeline

Updates from git push no-mistakes

✅ **Intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

✅ **Review** - passed

✅ No issues found.

⚠️ **Test** - 1 info
  • ℹ️ package.json - No package script or workflow runs scripts/live-check-plugin.test.mjs (the Greptile P2 on the PR). Tests only run via node --test scripts/live-check-plugin.test.mjs. Adding a test:live-check script and a CI step needs a maintainer decision.
  • Live validation: ✅ go - 3 of 3 scenarios driven live against the product
Scenario Result Live Evidence
Late Cloudflare-challenged response from an earlier step does not mark the later step INCONCLUSIVE (local server, real live-check script) ✅ pass live node --test scripts/live-check-plugin.test.mjs: 7/7 pass
Adversarial: with the old order (read currentStep after the await), the late-response test fails, so the test detects the bug ✅ pass live The temporary revert gave 1 failure, in the late-response test; the file was restored
Cloudflare challenge on a failing step becomes INCONCLUSIVE, while plain failures and passing steps are unchanged (local server, real script) ✅ pass live The other 6 tests in node --test scripts/live-check-plugin.test.mjs pass
  • npm ci --ignore-scripts
  • node --test scripts/live-check-plugin.test.mjs
  • Temporarily reverted the fix (capture currentStep after the await), reran the tests, then restored the file
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

@greptile-apps

greptile-apps Bot commented Oct 15, 2026 •

Copy link
Copy Markdown

RetriggerView in GreptileConfidence Score: 4/5

[Medium impact] Changes how the live-check test script reports Cloudflare blocks.

Fix response-to-step tracking before merging so a timed-out search cannot hide a real plugin failure.

Findings

  1. P1 Late responses hide failures ▶
  2. P2 New tests stay manual ▶

Summary

The checker now observes Cloudflare responses during plugin requests and changes affected failed steps to INCONCLUSIVE.

  • Plugin checks report blocked page loads as inconclusive.

Reviews (1) · Last reviewed commit: "no-mistakes(document): Document Cloudfla..." · Reviewed by Greptile

Comment thread scripts/live-check-plugin.js Outdated
Comment on lines +52 to +53
const res = await originalFetch(...args);
const step = currentStep;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Late responses hide failures

installChallengeDetector reads currentStep after the request finishes. If searchNovels times out, its request keeps running while the checker starts parseNovel. A late Cloudflare response from that search then marks parseNovel as blocked, hiding a real failure there as INCONCLUSIVE.

Capture the step before awaiting the request, and add a test for a late response crossing steps.

Suggested change
const res = await originalFetch(...args);
const step = currentStep;
const step = currentStep;
const res = await originalFetch(...args);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 74e8d1a: currentStep is captured before the await, and scripts/live-check-plugin.test.mjs now has a test of a late challenge response crossing steps (fails against the old order, passes with the fix). (AI-assisted: Claude Code.)

@@ -0,0 +1,97 @@
// Run with: node --test scripts/live-check-plugin.test.mjs

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 New tests stay manual

No package script or workflow runs scripts/live-check-plugin.test.mjs. The live-check workflow also skips execution when only checker files change, so these six tests cannot catch regressions automatically.

Add a CI step that runs these local-server tests when the checker or its tests change.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed in 74e8d1a: new workflow .github/workflows/live-check-tests.yml runs node --test scripts/live-check-plugin.test.mjs on pull requests that change the checker, its tests or the workflow itself. The existing live-check workflow is unchanged. I did not add an npm script. (AI-assisted: Claude Code.)

const originalFetch = globalThis.fetch;
globalThis.fetch = async (...args) => {
const res = await originalFetch(...args);
const step = currentStep;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please capture currentStep before awaiting originalFetch; otherwise a late response can be attributed to the wrong live-check step.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 74e8d1a: installChallengeDetector now reads currentStep into step before await originalFetch(...), so a late response is attributed to the step that sent it. Added a regression test where a first step's request outlives that step and returns a Cloudflare challenge while the next step runs; the later step's real failure stays FAIL (red against the old order, green with the fix). (AI-assisted: Claude Code.)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rajarsheechatterjee this is ready for another look: currentStep is now captured before the await in 74e8d1a9, with a regression test for a late response crossing steps.

…e request

Capture currentStep before awaiting the original fetch so a late
Cloudflare challenge cannot mark a later step INCONCLUSIVE. Add a
regression test for a response crossing steps and run the checker
tests in CI when the checker or its tests change.
@RibatTRW
RibatTRW force-pushed the fm/lnreader-plugins-livecheck-cf-inconclusive branch from 1483e28 to 74e8d1a Compare November 3, 2026 22:42
@RibatTRW RibatTRW changed the title fix(scripts): report Cloudflare-blocked live-check steps as inconclusive fix(scripts): report Cloudflare-challenged live-check steps as inconclusive Nov 3, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants