Skip to content

fix(remediate): save canonical pages in batches and resume a killed run - #86

Merged
Rl0007 merged 3 commits into
feat/import-actions-menufrom
fix/remediate-batch-save
Oct 6, 2026
Merged

Rl0007 merged 3 commits into
feat/import-actions-menufrom
fix/remediate-batch-save

Conversation

@Rl0007

@Rl0007 Rl0007 commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor
  1. A long PDF (about 350+ pages) is killed by the 1-hour job timeout during remediation, and every remediated page is lost because the canonical markdown was saved only at the very end.
  2. Remediation now saves the canonical markdown every 10 pages, so a killed run keeps the pages it finished. Running the import job again skips those pages and continues from the first unsaved one, without parsing the PDF again. An import left in Parsing or Remediating with no job behind it now shows "Resume import" in the Actions menu, which starts that run.

You can visit an import's page in /wikify after a killed run and open Actions to see it.

Fixes #22

Merge #85 first; this PR is stacked on it.

Before

After

Verification
  • New test test_killed_run_keeps_saved_batches_and_resume_skips_them (25 pages, run killed at page 18): red on main ([] != [1..10]), green on this branch. A resumed run calls the page model 15 times, not 25. Removing the batch commit also fails it (0 != 1).
  • New TestResumeImport tests (resume re-queues the parse job for Parsing and Remediating; refused while the parse or remediation job is queued or running; refused for a Review import; refused for a user without write): red without the resume commit (6 errors), green with it.
  • Parse and remediation jobs are now queued with a fixed job id per import, so the app can tell a stuck import from a running one.
  • Full suite on the devbox: 490 tests, 0 failures, 11 skipped (the corpus-only RAG eval tests).
  • Devbox, real wikify.jobs.parse.run on a 6-page [test] import with batch size 2: the process was SIGKILLed at page 5. Pages 1-4 were saved and the status stayed Remediating. A second run made 2 page-model calls (pages 5-6), reused the same Source Document and reached Review with 9 sections.
  • Devbox end to end in the browser: the import was started from the API and its worker's job process was SIGKILLed at page 5 (RQ moved the job to failed; status stayed Remediating). "Resume import" showed in Actions; clicking it ran pages 5-6 only on the same Source Document and the page moved to Review live. On a Completed import the item is hidden and resume_import returns 417.
  • The page model (VLM), cleanup and judge LLM calls were stubbed; the devbox has no model key.
  • Reviewers: Frappe review asked for full variable names, deduplicate=True and a Parsing-status test (done); Vue review approved.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@Rl0007
Rl0007 force-pushed the fix/remediate-batch-save branch from 425af97 to f980b2d Compare October 6, 2026 05:02
@Rl0007
Rl0007 changed the base branch from main to feat/import-actions-menu October 6, 2026 05:02

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Your trial has ended. Reactivate Greptile to resume code reviews.

@Rl0007
Rl0007 merged commit 9f47bb7 into feat/import-actions-menu Oct 6, 2026
3 of 6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant