Skip to content

fix(plugin): let sitemap departure run uncapped, and say that capped departures are dropped for good; v0.87.0 - #207

Merged
harper-joseph merged 2 commits into
mainfrom
fix/departure-uncapped
Sep 24, 2026
Merged

harper-joseph merged 2 commits into
mainfrom
fix/departure-uncapped

Conversation

@harper-joseph

Copy link
Copy Markdown
Contributor

Reserves plugin v0.87.0. PR #205 holds 0.85.0 and a sibling PR holds 0.86.0.

Problem

sitemap.departure.maxActions (default 5000) and sitemap.departure.maxCandidates (default 50000) had two problems.

  1. There was no way to turn them off. 0 means act on nothing / collect nothing, and 0 is also how Sitemap.refresh switches collection off when no route opts in, so it can't be reused. A deployment that wants every departure acted on had to fake it with a huge number (1,000,000).
  2. The maxActions doc was wrong. It said overflow "is counted as capped and left for the next walk rather than silently dropped". In fact the prune unlinks the target (sitemapUrl → null, Sitemap.js prune) before run.addRemoved records it as a candidate. Later walks only find departures by scanning for targets still linked to the sitemap being pruned, so a URL past either cap never comes up again. Overflow is gone for good. METRICS.md repeated the same claim for departure_capped.

Change

  • -1 means uncapped on both options. Defaults are unchanged (5000 / 50000), and 0 still means none. The schema floor goes from min: 0 to min: -1.

  • Both caps now go through a single departureLimit() in util/sitemapDeparture.js, which turns a negative value into Infinity before any comparison. Compared raw, acted >= -1 would refuse every candidate. departureCandidateCap() moves the per-walk candidate-cap expression out of Sitemap.refresh so it can be tested. It still returns 0 when the check is off or no route opts in, whatever maxCandidates says.

  • Uncapped never reports capped: departures.capped stays false, and no capped outcome is counted.

  • Docs corrected:

    • both option descriptions
    • the sitemapDeparture.js module doc (new section: why a capped departure is gone for good)
    • the processDepartures doc and its CAPPED log line
    • the sitemapRun.js comment
    • METRICS.md, all three copies of the departure_capped text, kept the same width so prettier doesn't realign the tables

    Each one says plainly that overflow is dropped for good and why. It also says what the cap guards against: a child sitemap that is truncated but still valid, which makes every URL it no longer lists look departed. And it says how to turn the cap off. The maxCandidates doc keeps its memory rationale and adds the uncapped bound: the URLs that walk actually unlinks, at most scan.collectCap per child sitemap. It also says how to count the loss: removed − departures.considered.

  • processDepartures is now exported, marked Exported for tests, which follows an existing pattern in this repo.

Why -1, not null or Infinity

I found no "no limit" value in configSchema.js that fits here. The schema's usual choice is "0 disables", and that's ruled out.

  • null already means "no value" in every layer. mergeInto skips it and keeps the default. validateOverride refuses it ("no value — to remove an override, clear it instead"). The console's number field reports null for an emptied box, and stage() drops it as "not a value yet". Giving it the meaning "uncapped" would silently change a blank or commented-out maxActions: in config.yaml from the default 5000 to no guard at all, which is the dangerous direction. The console also couldn't set it without its own change and a console release.
  • Infinity can't be sent as JSON, so no override row or console edit could carry it, and validateOverride deliberately rejects non-finite numbers.
  • -1 passes through every existing layer without changes beyond the floor: YAML, mergeInto's type check, enforceSchemaConstraints, validateOverride, the console's number input (which reads min from the schema) and JSON. Nobody types it by accident. The console needs no change because its field shows the updated description.

Tests

New file packages/plugin/test/sitemapDepartureCap.test.js, 14 tests. It runs the real processDepartures against fake Harper tables.

  • Defaults are unchanged (5000 / 50000).
  • config.yaml -1 is kept for both options. Values below -1 are rejected and the default is kept.
  • validateOverride accepts -1, refuses -2, and still accepts 0. resolveConfig applies a -1 override, including over a finite config.yaml value.
  • maxCandidates: -1 collects all 60,000 unlinked URLs, which is more than the default, and reports capped: false.
  • maxActions: -1 acts on all 6,000 candidates, again more than the default, in a dry run (would-render). With dryRun: false it acts on all 5,200 (expire). Neither reports capped.
  • The default maxActions still caps at 5000 and counts 1000 as capped. The default maxCandidates still caps at 50000.
  • 0 keeps its meaning: maxCandidates: 0 collects nothing and isn't reported as overflow. maxActions: 0 acts on nothing, and every candidate is still re-read and counted capped.
  • -1 doesn't switch collection on when no route opts in or enabled: false.

Revert check. Each fix was undone on its own and the new tests re-run:

Reverted Failing tests
schema floor min: -1 → min: 0 7
processDepartures compares raw maxActions 2 (both "acts on every candidate")
departureCandidateCap passes raw maxCandidates 3

With the fixes in place:

  • packages/plugin: node --test, 1224/1224 pass
  • packages/console: node --test, 277/277 pass
  • npm run lint and npm run format:check pass

Not in this PR

  • No collectConfigWarnings finding for "uncapped and live". A severity-info finding would show this dangerous-but-deliberate mode in the console preview, the same way staging passthrough is shown. I left it out to keep the PR focused.
  • No console change is needed.

🤖 Generated with Claude Code

…departures are dropped for good; v0.87.0

sitemap.departure.maxActions and maxCandidates had no "no ceiling" value:
0 means act on / collect nothing (and 0 is also how a walk with no
opted-in route keeps collection off), so an operator who wants every
departure acted on had to fake it with a huge number.

-1 now means uncapped on both. It is a plain number every layer already
validates and carries (config.yaml, the merge type check, schema min,
override validation, the console number field, JSON), unlike `null`,
which already means "no value" in all of them, or `Infinity`, which
cannot cross JSON. Both caps are read through `departureLimit`, which
turns -1 into Infinity before it meets a comparison, so uncapped never
reports `capped`. Defaults (5000 / 50000) and 0 are unchanged.

The maxActions doc claimed overflow is "left for the next walk rather
than silently dropped". It is not: the prune unlinks the target
(sitemapUrl -> null) before it becomes a candidate, and later walks only
find departures among targets still linked to a sitemap, so a URL past
either cap is never offered again. Both option docs, the module docs, the
walk's log line and METRICS.md now say so, with what the caps guard
against (a truncated-but-valid child sitemap) and how to turn them off.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces support for uncapped limits on departed URLs by allowing maxActions and maxCandidates to be set to -1 (representing no ceiling). It updates the configuration schema, introduces helper functions departureLimit and departureCandidateCap to map -1 to Infinity internally, and updates the sitemap processing logic to handle these uncapped limits. Additionally, a comprehensive test suite has been added to verify the behavior of these uncapped settings. There are no review comments to address, and I have no further feedback to provide.

# Conflicts:
#	package-lock.json
#	packages/plugin/package.json
@harper-joseph
harper-joseph merged commit e6f3c7d into main Sep 24, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant