Repository navigation
fix(report-generator): wait for an evaluable run before generating - #1576
Open
Naveen Chatlapalli (1aifanatic) wants to merge 1 commit into
Open
Naveen Chatlapalli (1aifanatic) wants to merge 1 commit into
Naveen Chatlapalli (1aifanatic) wants to merge 1 commit into
Conversation
A report triggered while its run was still pending or processing completed immediately as a stale "0 turns / 0%" report that later looked like the real result. The worker now keeps such reports pending and re-checks the run every 60s, and fails reports for runs that finished without any turn with an explicit non-evaluable reason instead of scoring them. Fixes microsoft#1287. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Naveen Chatlapalli (1aifanatic)
requested review from
Cedric Vidal (cedricvidal) and
Wassim Chegham (manekinekko)
as code owners
October 26, 2026 14:14
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1287.
A report triggered while its run was still
pending/processingwas generated immediately. It completed as a "run not executed, 0 turns, Overall Score 0%" report that stayed around after the run later succeeded, which inflated completed-report counts and skewed aggregates.The gate lives in the report-generator worker, so it covers every way a report is created (
POST /reports, bulk create, trigger, bulk trigger):run.statusispending,queued,processingorpaused, the report stayspending. The worker logsWaiting for run <id> to finish (status: …)on the report's log stream and hides the queue message forRUN_NOT_FINISHED_DEFER_SECONDS(60s) via the base processor'ssafeDeferMessage. It never runs an LLM session or writes a completed report for an unfinished run.failedwitherror: "Non-evaluable run: Run finished (<outcome>) without any turns …", including the run's own error when present, instead of producing a0%implementation score. A newreport_generator.run_non_evaluableevent tracks these.Not in this PR: the issue's last acceptance item, a canonical-report marker when a run has several reports.
bulk-summaryalready counts only the latest report per template. With premature reports no longer completing, the stale-duplicate case this item addresses should largely go away, so I'd keep any further UI/API work for a follow-up.Documented in
docs/architecture/app-design.md(report queue section) anddocs/architecture/observability.md(new event).Validation:
pnpm exec vitest run apps/workers/report-generator(43 passed). New tests cover readiness classification for pending/queued/processing/paused/no-run/done states, defer-and-stay-pending, and non-evaluable failure without an LLM session. Alsopnpm --filter report-generator exec tsc --noEmitandgit diff --check.🤖 Generated with Claude Code