Skip to content

feat(tasks): watchdog re-invokes stalled scheduled tasks - #258

Merged
harlan-zw merged 5 commits into
mainfrom
fix/issue-256
Sep 22, 2026
Merged

harlan-zw merged 5 commits into
mainfrom
fix/issue-256

Conversation

@harlan-github-agent

@harlan-github-agent harlan-github-agent Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

🔗 Linked issue

Closes #256

❓ Type of change

  • 📖 Documentation
  • 🐞 Bug fix
  • 👌 Enhancement
  • ✨ New feature
  • 🧹 Chore
  • ⚠️ Breaking change

📚 Description

The daily check-in found that backfill-skill-assets, drain-skill-dirty, and sync-reviewed-skill-repos stopped producing scheduled_runs rows at 2026-09-18T20:20Z while ai-ready:cron on the same tick kept firing. Zero rows, no error anywhere: their queues just backed up, and nothing would have noticed until the nightly report went red.

Every dispatch failure mode ends the same way, so this adds a watchdog on the same five-minute tick. It reads each policy task's recent run history and re-invokes the stalled ones through the same task registry the cron dispatch uses, so a silent gap heals within one silence window. It re-invokes silence, not failure: a task whose last run failed keeps its normal cadence, and a task with a live run is left alone.

Two judgement calls a reviewer may want to overrule:

  • A run past its own runtime ceiling (overdue_started) counts as stalled, so an abandoned in-flight row heals too.
  • The watchdog cannot re-invoke itself; if it stalls, that stays the daily check-in's finding.

Architecture: a new scheduled-cadence-watchdog task shares the five-minute cron tick, reads latest and latest-terminal run history from scheduled_runs, decides through the pure assessTaskCadence core, and re-invokes stalled policy tasks through the same Nitro task registry the cron dispatch uses

Data flow of one watchdog tick: cron fires the watchdog, it reads ranked run rows, assessTaskCadence returns a reinvoke or leave decision per task, the watchdog calls runTask with the Cloudflare env inside its runtime budget, the re-invoked task writes a fresh run row that restarts the silence window, and the watchdog emits reinvoked and failed task names as wide events

🤖 AI disclosure: Harlan Agent Kit modified this description. My AI open-source policy.

…s left three */5 tasks silently stalled with no run rows and no error. A watchdog on the same tick now re-invokes tasks whose run history went quiet, so the gap heals with
@harlan-github-agent harlan-github-agent Bot added harlan-agent-running An Agent holds a Task on this issue or pull request right now. harlan-agent-review-required Pull request triage requires an adversarial Review for this head commit. labels Sep 18, 2026
@harlan-github-agent

harlan-github-agent Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 MERGED

Harlan Agent Kit posted this automated review. It is not Harlan's personal review or approval. AI open source policy. Last updated: 2026-09-22 07:10 UTC.

GitHub merged this pull request.

No material findings were recorded.

4232a263-596f-432e-8e6e-3af40c352f7a

@harlan-github-agent harlan-github-agent Bot added harlan-agent-running An Agent holds a Task on this issue or pull request right now. harlan-agent-blocked The automated Review found a material defect in this head commit. and removed harlan-agent-running An Agent holds a Task on this issue or pull request right now. harlan-agent-review-required Pull request triage requires an adversarial Review for this head commit. harlan-agent-blocked The automated Review found a material defect in this head commit. labels Sep 18, 2026
…g target's cached runTask promise never settles, so the unbounded await stalled the watchdog itself on every tick. Race each re-invocation against the remaining watchdog b
@harlan-github-agent harlan-github-agent Bot added harlan-agent-running An Agent holds a Task on this issue or pull request right now. harlan-agent-blocked The automated Review found a material defect in this head commit. and removed harlan-agent-running An Agent holds a Task on this issue or pull request right now. harlan-agent-blocked The automated Review found a material defect in this head commit. labels Sep 18, 2026
@harlan-github-agent harlan-github-agent Bot added harlan-agent-running An Agent holds a Task on this issue or pull request right now. harlan-agent-blocked The automated Review found a material defect in this head commit. and removed harlan-agent-blocked The automated Review found a material defect in this head commit. harlan-agent-running An Agent holds a Task on this issue or pull request right now. labels Sep 19, 2026
@harlan-github-agent harlan-github-agent Bot added harlan-agent-running An Agent holds a Task on this issue or pull request right now. and removed harlan-agent-running An Agent holds a Task on this issue or pull request right now. labels Sep 22, 2026
nuxt-wide-events rejects spreads in field objects, which failed the Nitro build. Emit failed.tasks unconditionally; it is an empty string when nothing failed.
@harlan-github-agent harlan-github-agent Bot added harlan-agent-running An Agent holds a Task on this issue or pull request right now. and removed harlan-agent-running An Agent holds a Task on this issue or pull request right now. labels Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Three */5 scheduled tasks stopped dispatching with no run rows

1 participant