Design: render priority as lanes encoded in the high bits of nextRenderTime (no schema change) #80

Problem

The render queue has no concept of priority. claim is strictly nextRenderTime-ascending, so under
any capacity deficit the queue serves whatever is oldest-due — regardless of whether a page was
submitted by the site owner or discovered by a third-party crawler, and regardless of how tight its
freshness budget is.

Two independent measurements say this is the wrong order under scarcity, and capacity is expected to
stay below demand indefinitely.

Provenance. On a production deployment during a multi-hour backlog, ~46% of the overdue queue
was bot-discovered rather than sitemap-submitted (239,090 of 521,929 rows). Roughly half the render
capacity was being spent on pages the site owner never submitted, while submitted pages aged past
their SWR window and fell through to origin.

TTL-blindness — the sharper of the two. Ordering by absolute due time treats a 1 h-TTL homepage
3 h overdue exactly like a 48 h-TTL product page 3 h overdue. The first is 300% stale, the second 6%.
So the shortest-TTL route is structurally the most damaged, and it is damaged even at full
capacity. Simulated over the real corpus (20-lanesim.mjs), measuring age ÷ TTL rather than
absolute lag:

Every policy renders the identical number of pages — this is pure redistribution, never extra
work. The model validates against production: it predicts ~3.3 h standing lag at ~100% utilisation,
where production measured 3.05 h at 98%.

The observable symptom today is a consistently stale homepage: a 1 h route sitting in a queue with a
~3 h standing lag, with a flat swrTtl shared across 1 h and 48 h routes, so worst-case served age is
7× its own interval while a product page is at 1.1× of its.

Scheduling intent is scattered across five mechanisms

There is no way to express "this matters more" today. Instead, five separate mechanisms overload the
same conflated field:

Each is individually reasonable. Together they mean scheduling intent has five homes, none of them
named "priority", and every new requirement overloads nextRenderTime again.

Proposal

1. dueAt and priority are separate concepts

They are conflated today, which is the root of the problem above.

Truth about content vs. truth about importance. Neither is ever encoded into the other at the API
level.

2. Queue order is (lane, then dueAt ascending) — NOT relative lateness

An earlier revision of this issue proposed ordering within a class by relative lateness,
(now − dueAt) / interval descending, on the grounds that it makes "shorter TTL wins" fall out for
free. That reasoning is right about why short-TTL routes need protecting, but relative lateness
cannot be the comparator, because it is not a static order:

relativeLateness(t) = (t − dueAt) / interval is linear in t with slope 1/interval. Two rows
with different intervals therefore have different slopes and cross exactly once. No stored key
can express an order that changes with the clock.

Re-ranking inside the claim window does not rescue it either: the claim pass reads a bounded window
(~140 rows in production) that is already an EDF prefix, so a 1 h page sitting 600 minutes down the
queue never appears in it to be re-ranked.

The resolution: relative lateness becomes the rationale for lane assignment, not the comparator.
Lanes are derived from TTL and provenance, so within a lane the intervals are similar by construction
and EDF on dueAt is both index-backable and a good approximation of relative lateness. EDF is
provably optimal for maximum lateness, so any deviation inside a lane only ever costs.

3. priority is an ordered enum, and it is DERIVED, not stored

urgent      operator intent — manual clear / invalidate / force-render
submitted   present in a sitemap
discovered  crawler-found, never submitted
cold        repeatedly failing, or never successfully rendered

Named and ordered rather than a free-form integer: self-documenting in logs and the admin UI,
reviewable in a diff, and it cannot drift into arbitrary magic numbers.

Derived is load-bearing. Resolve the lane at write time from stable inputs (presence of
sitemapUrl, route match, failure count) instead of storing it — the same discipline
resolveRenderInterval already uses (route > stored > default). A config change is then retroactive
on each key's next render with no sweep of the corpus.

4. Fairness is scheduler policy, NOT part of the ordering key — and it must be FLOORS

queue:
  lanes:
    urgentMaxShare: 0.2          # drain-share cap on lane 0
    minShare: { discovered: 0.10, cold: 0.02 }

Two findings from the simulation, both load-bearing:

• Strict priority starves the tail at EVERY capacity level, including 100%. Its lag numbers look
  excellent precisely because it drops work — starvation is invisible in a lag metric. Do not use it
  for anything but a tiny rate-limited lane 0.
• Floors beat fixed shares. Fixed shares summing to 1.0 leave nothing for a global EDF sweep, and
  since EDF is optimal for maximum lateness the deviation only ever costs. Reserving a minimum for
  the lanes that need protecting and letting the default lane compete for the remainder is far better
  in the tail: discovery reaches 71 h vs 133 h at 50% capacity, 29 h vs 40 h at 75%.

For lane 0, cap its drain share, not its admission (urgentMaxShare): strict-first but never more
than that fraction of a claim batch. That is a hard structural bound — lanes 1–3 always get ≥80% —
that needs no token bucket and no admission bookkeeping. EDF within lane 0 so a flood is served
oldest-first.

Keeping the fairness bound out of the ordering key means it is tunable live with no rewriting of
stored rows. Any scheme that bakes fairness into the key requires re-encoding every stored value to
change the bound. This remains the strongest single argument for keeping ordering and fairness
separate.

5. One operation replaces the ad-hoc tricks

POST /render_queue/prioritize   { url | scope, class: urgent }

This turns the cache-clear case into explicit policy rather than an accident of which mechanism the
operator reached for:

• clear-and-rush — drop the cached page and re-render at urgent
• clear-and-wait — drop the cached page, let normal cadence refill it, serve origin meanwhile

Both are legitimate; today the choice is implicit. It also retires nextRenderTime = 1, which is a
documented hack that only works if you happen to target the owner node.

6. Observability is part of the API, not an afterthought

• backlog depth per lane
• lane as a dimension on the render metrics

Without this there is no way to answer "is prioritisation working". Note the existing constraint that
recordAnalytics allows exactly three dimensions per metric — so this likely needs its own metric
name rather than an extra dimension on an existing one.

Encoding — DECIDED: lane in the high bits of nextRenderTime

The encoding was previously deferred. It is now settled, because one option turns out to need no
schema change and no migration at all.

Store nextRenderTime = lane × STRIDE + dueAtMs, with lane 0 = identity. The column stays
Long @indexed; the existing index is reused, not supplemented.

Why this one:

• Zero migration. Every existing row is already a valid lane-0 row. No backfill, no dual-write
  window, and rollback is "stop encoding".
• Lower value = claimed first, so the lane that matters most is the unencoded one. Urgent needs no
  encoding, and the hottest path is untouched.
• A lane change is an in-place numeric update, not a delete + put.
• Numerically safe with room to spare. Harper's Long is 52-bit-safe (9.007e15) and current
  timestamps are ~1.79e12. A stride of 2^42 (≈139 years, comfortably above any real timestamp)
  yields 2,048 lanes; four are needed.

Measured support for one interleaved index over separate structures (21-duerank.mjs, 200k rows,
5.1.26 / rocksdb-js 2.4.1):

• Three lanes interleaved in ONE index, each with its own watermark: 0.29–0.32 ms per lane.
  Interleaving costs nothing.
• The same lane with its watermark reset to zero: 3.46 ms — the watermark is the entire win, not
  the separation.
• Buckets are unnecessary: bulk-enqueueing 100,000 keys at a single timestamp reads at 0.28 ms and
  stays 0.29–0.43 ms while draining. Exact-ms ranking is fine.
• A second index is not free: the existing secondary index is 39–48% of reschedule wall clock, so
  a separate per-lane index or table roughly doubles the hot write.

Alternatives, and why they lose:

The one real cost, and how it is guarded

For a non-zero lane the stored value is no longer a timestamp, while still looking exactly like
one. This is the same class of trap the maybeUnpinFloor comment already warns about — the row is the
only record, it outlives the pass that wrote it, and a later reader has no way to know the value was
synthetic. Guard it two ways:

1. dueAtOf(row) / laneOf(row) accessors in util/renderSchedule.js, as the only way in or out.
2. A sanity horizon so a raw read of an encoded value throws loudly rather than returning a
   plausible year-2109 date.

Implementation surface (audited against main @ v0.47.1)

Writes need no change at any call site. Every schedule write already funnels through
writeSchedule / writeSchedules — Target.put, Target.revalidate, suppression recheck, sitemap
ingest, reconcile, bot-request discovery, admin, the job-result path, and the unpin path. The lane is
resolved from the cacheKey's URL inside the funnel, so all ~14 writers keep passing plain
timestamps and never learn lanes exist.

Reads are the actual work — five sites:

Claim path: N seeks per pass instead of 1, one per lane at ~0.3 ms each — noise. The claim floor
becomes per-lane: one additional Uint32 per lane in the existing shared buffer.

Invariants any implementation must hold

1. expiresAt always reflects true dueAt. Never a lane-adjusted value. Violating this serves
   pages as fresh when they are not — silent and severe.
2. The claim floor is per-lane. A single watermark shared across lanes lets one low-priority row
   pin the entire queue — the generalisation of the floor-pin failure mode fixed in v0.37.0.
3. Nothing outside the queue module reads the raw sort key. Callers see dueAt and lane.
4. No corpus-wide sweep is required to change lane policy.

Cheaper levers available first

Neither is a substitute — under permanent scarcity the ordering is the fix — but both buy runway
without touching the schema:

• Per-route swrTtl. A single global value cannot fit 1 h, 6 h and 48 h routes at once; the
  short-TTL routes are the ones paying for it. Small plugin change, already flagged as the real fix in
  the deployment's own config comments.
• The existing manual force-render path, for a handful of URLs.

Open questions

1. Should urgent bypass the fairness reservation entirely, or compete within a reserved share?
   Answered: neither — cap its drain share (urgentMaxShare), which bounds it structurally with
   no admission bookkeeping.
2. Should cold be a lane at all, or is that the demand ladder's job? (The ladder treats the route's
   base interval as its own top rung, so it can only make pages faster, never slower — a never-visited
   page sits at base forever. Extending it downward overlaps with cold.)
3. Does bulk invalidation default to clear-and-rush or clear-and-wait?
4. Is relative lateness the right within-class comparator for urgent? Answered by §2: relative
   lateness is not a viable comparator anywhere; every lane is EDF on dueAt internally.
5. How many lanes actually ship? The simulation covers four; submitted vs discovered carries the
   provenance win and TTL carries the staleness win, and they may not need to be independent axes.