[harper-joseph]

Mechanical follow-up to #164, no new code.

#164 was opened against fix/retire-gone-discovered (#163's branch) as a stacked PR. #163 merged to main first, and #164 was merged before GitHub had retargeted it — so its two commits landed on fix/retire-gone-discovered instead of main. This carries them across.

Diff is exactly:

3ed0d4a Merge pull request #164 from HarperFast/feat/sitemap-departure-check
0cd2b07 feat(plugin): re-check URLs that leave a sitemap, per route; v0.68.0

Content, rationale and tests are all reviewed on #164 — gemini's review there raised no comments. Ships inert: no route sets departureAction and sitemap.departure.dryRun defaults true.

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces a sitemap departure check feature (v0.68.0) to handle URLs that leave a sitemap, allowing routes to opt into actions like hard-expiring cached pages or scheduling immediate re-renders. The changes include new configuration options, post-walk processing logic, metrics updates, and comprehensive unit tests. Feedback on the implementation suggests wrapping the individual URL processing logic within the batch execution in a try-catch block to prevent a single database failure from aborting the entire departure check.

[gemini-code-assist]

The apply function processes each departed URL individually. However, if any single URL fails (e.g., due to a transient database read/write error or a missing record), the entire applyInBatches promise will reject, aborting the departure check for all remaining URLs. Wrapping the inner logic of the apply function in a try...catch block ensures that a single failure is isolated, logged, and counted, allowing the rest of the departed URLs to be processed successfully. Additionally, this implementation correctly parallelizes the independent sub-operations (cache keys/device variants within a single URL) using Promise.all while serializing the top-level URL batches.

		apply: async (url) => {
			try {
				const target = await Target.get({
					id: url,
					// `sitemapUrl` is the shear guard, `state` keeps suppressed targets out, and the two
					// cadence fields are what `resolveEffectiveInterval` needs to file a rung-correct row.
					select: ['url', 'sitemapUrl', 'state', 'renderInterval', 'demandInterval'],
				});

				const { action, reason } = decideDeparture({ url, target });
				if (action === DepartureAction.NONE) {
					run.countDeparture(reason);
					return;
				}

				// Check and increment in ONE synchronous block. `applyInBatches` runs a batch's items in
				// parallel, so a check that awaited anything before incrementing would let a whole batch
				// through a cap of one.
				if (acted >= maxActions) {
					run.countDeparture('capped');
					return;
				}
				acted++;

				if (dryRun) {
					run.countDeparture(`would-${action}`);
					return;
				}

				// HARD-expired, past the stale-while-revalidate window rather than to `now`. A plain
				// `Date.now()` expiry leaves the page 'swr' (`util/pageFreshness.js`), i.e. still SERVING
				// for another `page.swrTtl` — and the swr window exists to smooth over a late re-render of
				// content presumed still right, which is exactly what a departed product page is not.
				// This matches `changeProbe.triggerRevalidate`, which backdates for the same reason;
				// `Target.revalidate` keeps the plain expiry deliberately, because an operator asking for
				// a re-render is not asserting the content is wrong.
				const hardExpiredAt = Date.now() - config.page.swrTtl;
				await Promise.all(
					cacheKeysOf(url).map(async (cacheKey) => {
						const page = await PrerenderedPage.get({ id: cacheKey, select: ['cacheKey', 'expiresAt'] });
						if (page) await PrerenderedPage.patch(cacheKey, { expiresAt: hardExpiredAt });
					})
				);

				if (action === DepartureAction.RENDER) {
					// THE CURRENT MINUTE, PER URL — never captured once for the whole pass. Rows are
					// residency-routed, and a minute more than `queue.claimFloor.guard` old lands below the
					// owner's floor and is never claimed again. This is the Target.revalidate lesson.
					await writeSchedule(url, {
						nextRenderTime: currentMinuteMs(),
						// Derived, never a literal `false`, even though `decideDeparture` has already proved
						// this target has no attribution and so this can only evaluate false. The literal is
						// what `test/queueFunnel.test.js` forbids outright, and for a good reason: the
						// derivation stays correct if this guard ever moves, where a literal would silently
						// un-flag a sitemap-listed URL the day someone loosened the check above it.
						fromSitemap: !!target.sitemapUrl,
						effectiveInterval: resolveEffectiveInterval(url, target),
					});
				}
				run.countDeparture(action);
			} catch (e) {
				logger.error(`[prerender] Failed to process departure for ${url}: ${describeError(e)}`);
				run.countDeparture('failed');
			}
		},

References

1. When performing batch database writes or revalidations, serialize the top-level batches (e.g., per-URL) to prevent long transactions from spanning monitor ticks and being aborted, but parallelize independent sub-operations (e.g., device variants within a single URL) using Promise.all.