Repository navigation
fix(plugin): let sitemap departure run uncapped, and say that capped departures are dropped for good; v0.87.0 - #207
Merged
Merged
Conversation
…departures are dropped for good; v0.87.0 sitemap.departure.maxActions and maxCandidates had no "no ceiling" value: 0 means act on / collect nothing (and 0 is also how a walk with no opted-in route keeps collection off), so an operator who wants every departure acted on had to fake it with a huge number. -1 now means uncapped on both. It is a plain number every layer already validates and carries (config.yaml, the merge type check, schema min, override validation, the console number field, JSON), unlike `null`, which already means "no value" in all of them, or `Infinity`, which cannot cross JSON. Both caps are read through `departureLimit`, which turns -1 into Infinity before it meets a comparison, so uncapped never reports `capped`. Defaults (5000 / 50000) and 0 are unchanged. The maxActions doc claimed overflow is "left for the next walk rather than silently dropped". It is not: the prune unlinks the target (sitemapUrl -> null) before it becomes a candidate, and later walks only find departures among targets still linked to a sitemap, so a URL past either cap is never offered again. Both option docs, the module docs, the walk's log line and METRICS.md now say so, with what the caps guard against (a truncated-but-valid child sitemap) and how to turn them off. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Code Review
This pull request introduces support for uncapped limits on departed URLs by allowing maxActions and maxCandidates to be set to -1 (representing no ceiling). It updates the configuration schema, introduces helper functions departureLimit and departureCandidateCap to map -1 to Infinity internally, and updates the sitemap processing logic to handle these uncapped limits. Additionally, a comprehensive test suite has been added to verify the behavior of these uncapped settings. There are no review comments to address, and I have no further feedback to provide.
# Conflicts: # package-lock.json # packages/plugin/package.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reserves plugin v0.87.0. PR #205 holds 0.85.0 and a sibling PR holds 0.86.0.
Problem
sitemap.departure.maxActions(default 5000) andsitemap.departure.maxCandidates(default 50000) had two problems.0means act on nothing / collect nothing, and0is also howSitemap.refreshswitches collection off when no route opts in, so it can't be reused. A deployment that wants every departure acted on had to fake it with a huge number (1,000,000).maxActionsdoc was wrong. It said overflow "is counted ascappedand left for the next walk rather than silently dropped". In fact the prune unlinks the target (sitemapUrl → null,Sitemap.jsprune) beforerun.addRemovedrecords it as a candidate. Later walks only find departures by scanning for targets still linked to the sitemap being pruned, so a URL past either cap never comes up again. Overflow is gone for good.METRICS.mdrepeated the same claim fordeparture_capped.Change
-1means uncapped on both options. Defaults are unchanged (5000 / 50000), and0still means none. The schema floor goes frommin: 0tomin: -1.Both caps now go through a single
departureLimit()inutil/sitemapDeparture.js, which turns a negative value intoInfinitybefore any comparison. Compared raw,acted >= -1would refuse every candidate.departureCandidateCap()moves the per-walk candidate-cap expression out ofSitemap.refreshso it can be tested. It still returns0when the check is off or no route opts in, whatevermaxCandidatessays.Uncapped never reports
capped:departures.cappedstaysfalse, and nocappedoutcome is counted.Docs corrected:
sitemapDeparture.jsmodule doc (new section: why a capped departure is gone for good)processDeparturesdoc and itsCAPPEDlog linesitemapRun.jscommentMETRICS.md, all three copies of thedeparture_cappedtext, kept the same width so prettier doesn't realign the tablesEach one says plainly that overflow is dropped for good and why. It also says what the cap guards against: a child sitemap that is truncated but still valid, which makes every URL it no longer lists look departed. And it says how to turn the cap off. The
maxCandidatesdoc keeps its memory rationale and adds the uncapped bound: the URLs that walk actually unlinks, at mostscan.collectCapper child sitemap. It also says how to count the loss:removed − departures.considered.processDeparturesis now exported, marked Exported for tests, which follows an existing pattern in this repo.Why
-1, notnullorInfinityI found no "no limit" value in
configSchema.jsthat fits here. The schema's usual choice is "0 disables", and that's ruled out.nullalready means "no value" in every layer.mergeIntoskips it and keeps the default.validateOverriderefuses it ("no value — to remove an override, clear it instead"). The console's number field reportsnullfor an emptied box, andstage()drops it as "not a value yet". Giving it the meaning "uncapped" would silently change a blank or commented-outmaxActions:in config.yaml from the default 5000 to no guard at all, which is the dangerous direction. The console also couldn't set it without its own change and a console release.Infinitycan't be sent as JSON, so no override row or console edit could carry it, andvalidateOverridedeliberately rejects non-finite numbers.-1passes through every existing layer without changes beyond the floor: YAML,mergeInto's type check,enforceSchemaConstraints,validateOverride, the console's number input (which readsminfrom the schema) and JSON. Nobody types it by accident. The console needs no change because its field shows the updated description.Tests
New file
packages/plugin/test/sitemapDepartureCap.test.js, 14 tests. It runs the realprocessDeparturesagainst fake Harper tables.-1is kept for both options. Values below-1are rejected and the default is kept.validateOverrideaccepts-1, refuses-2, and still accepts0.resolveConfigapplies a-1override, including over a finite config.yaml value.maxCandidates: -1collects all 60,000 unlinked URLs, which is more than the default, and reportscapped: false.maxActions: -1acts on all 6,000 candidates, again more than the default, in a dry run (would-render). WithdryRun: falseit acts on all 5,200 (expire). Neither reportscapped.maxActionsstill caps at 5000 and counts 1000 ascapped. The defaultmaxCandidatesstill caps at 50000.0keeps its meaning:maxCandidates: 0collects nothing and isn't reported as overflow.maxActions: 0acts on nothing, and every candidate is still re-read and countedcapped.-1doesn't switch collection on when no route opts in orenabled: false.Revert check. Each fix was undone on its own and the new tests re-run:
min: -1→min: 0processDeparturescompares rawmaxActionsdepartureCandidateCappasses rawmaxCandidatesWith the fixes in place:
packages/plugin:node --test, 1224/1224 passpackages/console:node --test, 277/277 passnpm run lintandnpm run format:checkpassNot in this PR
collectConfigWarningsfinding for "uncapped and live". A severity-infofinding would show this dangerous-but-deliberate mode in the console preview, the same way staging passthrough is shown. I left it out to keep the PR focused.🤖 Generated with Claude Code