[harper-joseph]

Why

On 2026-09-28 the change probe found 60% of product pages changed overnight on one deployment. That was ~696k across 4 nodes; the day before it was ~74k. Changes went through an in-memory trigger queue:

• the queue drained at trigger.ratePerSecond (3/s per node there), was capped at trigger.maxPending (50k), and everything past a cap was deferred to the next pass;
• each node spent over 12 hours draining and deferred ~33k changes;
• a restart dropped the ~18k still queued.

That left ~152k pages the probe knew were wrong still being served until the next day's pass. Anchored mode also never resumed a pass a restart cut short.

The limits protected things that no longer exist, or no longer need it:

• the claim floor, and the claim scan's slowdown when rows pile onto one minute (both gone in 0.93.0);
• the render fleet's capacity, which bounds how fast pages re-render, not how fast a known-wrong page should stop being served.

Supersedes #178, which moved toward queuing everything but kept the queue.

What changed

Plugin 0.94.0

• A detected change is acted on when it is found (util/changeActions.js).

  • actOnChange hard-expires the cached device pages and files the render.
  • Actions run beside the walk, at most trigger.concurrency (8) at once. When every slot is busy the walk waits for one (backpressure): nothing is refused, deferred or dropped, and there is no per-pass budget.
  • The new baseline is written only after its action succeeds, so a failure or a restart leaves the change detectable.
  • Disabling the probe stops the pass within a batch; at most the in-flight actions finish.

• Changed pages render first.

  • The schedule row carries a durable mark, the new RenderSchedule.changedAt column.
  • The keeper ranks marked rows queue.ready.changedHeadStart cadences ahead (default 1). The head start is additive because a just-filed row has lateness 0, which no multiplier can move.
  • Starvation stays bounded: a routine row wins once it is more than changedHeadStart cadences later than the changed rows being held (documented in util/renderPriority.js). So during a big wave, a 1h-cadence homepage re-renders about an hour late at worst.
  • A failed render's retry carries the mark; the render's own reschedule clears it.
  • queue-state gains now.dueChanged and now.oldestChangedAt.

• fileDueNow (util/renderSchedule.js) is now used by every "render this now" writer: the probe action, revalidate, render-now and admin rejoin.

  • An already-due row keeps its due time and its mark, so a change found again, or re-filed for another reason, never moves back in the queue.
  • This works on the row's owner, where the read is local. Elsewhere the row is filed plainly, as before.
  • A revalidate now only ever lowers a page's expiry. Before, it could put a probe-hard-expired page back into stale-while-revalidate serving.

• Restart resilience.

  • The sweep's heartbeat publishes a walk cursor: the last fully probed batch, held back to the lowest URL with an action still in flight.
  • On boot in anchored mode, a claim that is still running but whose heartbeat stopped (stale after 5 min) is resumed from that cursor.
  • The resume keeps the interrupted pass's dry-run and reseed settings and the start of the pass it continues. A config apply before the check fires re-arms it rather than dropping it.

• The canary's reseed stays a dry run. The trip has already invalidated the whole scope. Per-URL expiry on top would make a false trip irreversible, and would re-expire pages that had already re-rendered correctly.

• Visibility.

  • The pass records triggered (changes acted on), errors, maxActionsInFlight, and actionWaitMs, the time the walk spent blocked on a full pipeline.
  • The running pass reports actionsInFlight, cursor, reseed and originStartedAt.

• Removed:

  • config changeProbe.maxTriggersPerSweep, changeProbe.trigger.ratePerSecond and changeProbe.trigger.maxPending;
  • series probe_deferred and probe_trigger_queue_depth;
  • pass fields queued, deferred and triggerQueueDepth.

  trigger.concurrency now means actions in flight, and defaults to 8.

Console 0.19.0

• Probe view:
  • The queue-depth and deferral alarms, rows and chart series are gone, along with the budget and drain-rate settings rows.
  • New: "Acted on" and "Action errors" (a watch on any error), and actions in flight with the time spent waiting for a slot. A watch fires when that wait passes 10% of the pass's time and at least a minute.
  • A resumed pass shows the pass it continues and its resume point.
• Queue:
  • a "Changed, waiting" tile, judged by the worse of the oldest changed page's actual wait and the time to re-render the count (2h watch, 8h bad);
  • a per-node changed column;
  • changed classes marked in the lateness table.
• Health: a "Changed pages waiting" check.
• Guard test: the probe view must not read a series or field the plugin no longer declares.
• On older plugins the new displays stay hidden.

Deploying

• Deploy the plugin to every node before console 0.19.0.
• Config: delete the three removed keys from config.yaml and from any stored overrides. A stored override logs a warning until cleared.
• Schema: RenderSchedule.changedAt is additive. I verified it with an in-place 0.93.1 → 0.94.0 upgrade on Harper 5.2.14.
• On a wave day, origin load goes up by design. A changed page is expired when it is found, and is served from the origin until its render lands. The fleet re-renders changed pages first. Before this change, most such pages kept serving known-wrong content from the cache.
• Today's leftovers: changes deferred or dropped before this deploy stay detectable (their baselines are stale). The next pass acts on all of them, from the moment it finds them.

Verification

• Unit tests: plugin node --test 1403/1403, console 444/444. eslint and prettier --check are clean.
• Mutation checks: I broke each property below and confirmed a test fails:
  • baseline written after the action;
  • backpressure;
  • head start, and its bound;
  • re-detection keeps its place and its first changedAt;
  • no second patch of an expired page;
  • retry carries the mark;
  • the keeper load reads the mark;
  • the resume cursor held back to in-flight actions;
  • the resume walks from the cursor, keeps the dry-run setting, and is armed at boot;
  • an early config apply cannot drop a pending resume;
  • revalidate never raises an expiry;
  • fileDueNow never demotes;
  • wait time is counted;
  • both reseed paths are dry runs.
• Real Harper 5.2.14 in Docker, with a local fake price origin:
  • Upgrade: 0.93.1 → 0.94.0 in place added changedAt, and the keeper came up exact.
  • Pass with 150 of 300 prices changed:
    • 150 acted on, 0 errors, 6.3 s, the same duration as a pass with no changes;
    • each changed row was marked and its page's expiry backdated past the swr window;
    • dueChanged: 150;
    • the claim returned only changed product pages, ahead of 20 routine rows that were 6 hours overdue.
  • Restart mid-pass (400 rows, 20% changed, restarted after 196 were probed):
    • it resumed 304 s after the restart, from the published cursor, and probed the remaining 205 rows;
    • that's 405 probes in all for 400 rows, against about 560 with the pre-review "skip by baseline age" and 600 from the top;
    • it acted on the other 40 changes, so all 80 changed rows ended up marked;
    • it recorded the interrupted pass's start as resumedFrom, and actionWaitMs was 0.
  • 0 Harper [error] lines.
• Review: one fresh-context adversarial review of the plugin diff.
  • Bugs: the resume re-probed every unchanged row (fixed with the cursor), and the reseed tests passed only by the schema's default (fixed).
  • Risks: an acting reseed made a false trip irreversible (fixed by keeping it a dry run); backpressure was invisible (fixed with actionWaitMs); the resume dropped the interrupted pass's mode (fixed).
  • Nits: a config apply could cancel a pending resume; writers dropped the mark; a revalidate could raise a hard expiry; BigInt, Infinity and zero-value edge cases; stale docs. All fixed.

Not covered

• No multi-node cluster run. In particular, a render-now or revalidate issued on a node that does not own the row cannot see the mark, and its write replaces the owner's row. That is the same cross-node behaviour as before.
• The canary is unchanged apart from staying a dry run. At its 0.7 threshold, a mass change still bulk-invalidates the scope and relies on the accelerator to heal.
• A crash can cost the actions in flight (at most trigger.concurrency). They are re-probed on resume because the cursor is held back to them.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request upgrades the prerender plugin to version 0.94.0 and the console to version 0.19.0, replacing the in-memory trigger queue with an immediate action pipeline that hard-expires changed pages and files their renders ahead of routine rotation. It also introduces restart resilience, allowing interrupted anchored sweeps to resume from their last walk cursor on boot, and updates the console UI to display these new metrics. The feedback recommends ensuring that getScheduleRow is properly imported in RenderQueue.js and suggests using the !(elapsed < interval) pattern in changeProbe.js to safely handle potential NaN values during elapsed time comparisons.

[gemini-code-assist]

Please ensure that getScheduleRow is imported from ../util/renderSchedule.js at the top of this file, as it is now being called here to retrieve the changedAt field.

[harper-joseph]

It is imported: getScheduleRow is in the ../util/renderSchedule.js import block at the top of RenderQueue.js (line 28), and the retry test (renderQueueVariants.test.js, "a CHANGED row keeps its mark through a failed render") exercises this call. No change.

[gemini-code-assist]

When writing elapsed time comparisons where the timestamp might be null, undefined, or unparseable (resulting in NaN), use the pattern !(elapsed < interval) instead of elapsed >= interval (or elapsed > interval). This ensures that NaN values fail the inequality comparison and correctly fall through to the fallback/unconditional branch (failing closed by not resuming the pass).

Suggested change

	if (Date.now() - originStartedAt > RESUME_WITHIN_MS) return decided({ resumed: false, reason: 'older than a day' });
	const elapsed = Date.now() - originStartedAt;
	if (!(elapsed < RESUME_WITHIN_MS)) return decided({ resumed: false, reason: 'older than a day' });

References

1. When writing elapsed time comparisons where the timestamp might be null, undefined, or unparseable (resulting in NaN), use the pattern !(elapsed < interval) instead of elapsed >= interval.

[harper-joseph]

Fixed in c7aeca7: the age test is now !(Date.now() - originStartedAt < RESUME_WITHIN_MS), and an unusable recorded originStartedAt falls back to the claim's own (validated) startedAt. NaN would also have thrown in the log line's toISOString() and silently cancelled the resume. Test: "an unusable recorded origin falls back to the claim's own start".