[mickvandijke]

Linear issue

Closes V2-1833

Risk tier

• T0 — docs / tooling / CI / pure UX-output. Repo CI only.
• T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
• T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
• T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Compatibility

• Wire: none. No message, field or protocol id changes. Prune audits still use the existing AuditChallenge path, and they are only sent under disk pressure.
• Storage: no change to record or paid-list formats. New files record_out_of_range.bin and pointer_out_of_range.bin hold the saved clocks, and older binaries ignore them. The paid-list LMDB map grows from 256 MiB to 1 GiB; its file grows only as entries do. Nodes keep out-of-range records until disk pressure, so disks fill to about 85–90% before anything is erased.
• API: breaking. Changes:
  • PrunePassContext gains disk_pressure.
  • ReplicationConfig gains paid_list_evict_start_entries.
  • PointerReplication::prune_pass takes a &DiskPressure.
  • audit::audit_tick_with_repair_proofs takes the PaidList, to sample only in-range records.
  • NeighborSyncState::prune_cursor is removed.
  • replication::config::MAX_FAST_PRUNE_DELETIONS_PER_PASS is removed, and MAX_PRUNE_AUDIT_REQUESTS_PER_PASS rises from 256 to 1,024.
  • PruneResult gains disk_pressure and records_retained, and its paid counters become paid_entries_evicted.
  • The prune log line gains disk_pressure= and retained=, drops the fast-path fields, and its paid part becomes paid total= evicted=.
  • PaidList loses its paid timers, its prune cursor and all_keys, and gains keys_near and evict_far. Its record timestamps are SystemTimes, and remove/remove_batch leave them alone.
  • New: replication::config::PAID_HINT_FULL_SCAN_ENTRIES, storage::DiskSpace, ChunkStore::disk_space, ChunkStore::disk_reserve, and the test-only ChunkStore::set_test_disk_space.

Semver impact

• breaking
• feature
• fix

Test evidence

• Unit: cargo test --lib --features test-utils: 1248 passed, 0 failed.
  • replication::retention: the thresholds and their hysteresis, the reserve-based bounds, a failed measurement erasing nothing, farthest-first order with failed proofs last, pass spacing, and two fairness tests. a_few_new_candidates_each_pass_do_not_stop_failed_ones_being_retried failed before the retry order was fixed, with a failed record getting 1 turn in 40 passes.
  • replication::out_of_range_clocks: a saved clock keeps its start and counts time offline, a wall clock set back, saving only on change, a corrupt or missing file.
  • replication::neighbor_sync: a far key read for hints while the list can be read whole, and only near keys above that.
  • replication::paid_list: near ranges, eviction farthest first up to its limit and never what it keeps or what is near, record clocks surviving a reopen, the file not taking the whole map up front.
• End to end: cargo test --test e2e --features test-utils -- --test-threads=1 paid_ prun pointer out_of_range_records_stay audit: 41 passed, 0 failed.
  • A record the whole close group holds is kept, and its group never asked, while the disk has room; under pressure it goes on the group's proofs.
  • Disk pressure evicts no paid entry, and an entry near the node stays over the threshold.
  • A neighbour gets a paid hint for a key far from the node. This fails with the full scan switched off.
• Paid-list capacity: measured with a temporary probe that isn't committed, using single-key commits as inserts do: 4.41 million entries at 256 MiB and 17.67 million at 1 GiB, about 61 bytes each. LMDB's Windows code grows the file incrementally unless built with MDB_FIXEDSIZE, which this build doesn't set, and a test pins it.
• Hint cost: measured with a temporary probe that isn't committed, on a developer laptop: a routing table lookup takes 4.7 µs with 4 peers, 7.1 µs with 9 and 14.5 µs with 24, which extrapolates to about 100 µs with 200. Reading a key takes under 0.2 µs.
• Lint and format:
  • cargo clippy --all-features -- -D clippy::panic -D clippy::unwrap_used -D clippy::expect_used: clean.
  • cargo clippy --all-targets --all-features -- -D warnings: no findings in changed code. The six remaining warnings predate this PR.
  • cargo fmt --all -- --check and RUSTDOCFLAGS="-D warnings" cargo doc --all-features --no-deps: clean.
• Dev testnet, first run (60 nodes, 9 regions, a test build with a 20-minute hysteresis, 6 nodes on small filesystems at 9% free, on an earlier commit of this PR):
  • Nodes with room erased nothing and sent no prune audits, keeping 112–192 candidates each.
  • Nodes under pressure erased 911 records on their groups' proofs, with no failed proofs and audit rounds of 2–14 s.
  • Erasing stopped at 15% free with candidates left, and a node at 13.4% stayed out of pressure until 10%.
  • A restarted node reported the same 117 candidates on its first pass.
  • No paid entry was removed.
• Dev testnet, confirmation run (on c435a2bf, before the follow-ups; same fleet, with a test build using a 20-minute hysteresis, a paid threshold of 1,000 and no near margin; 12 GB uploaded, no failures):
  • The 54 nodes with room sent no prune audits and erased nothing over 332 passes, keeping up to 265 candidates.
  • The 6 nodes under pressure erased 1,173 records on their groups' proofs, with no failed proofs, nothing deferred and audit rounds of 1-7 s.
  • Erasing stopped at 15.0% free with 96-139 candidates left on each. With an uploader running, each fell below 10% again and erased 3-12 minutes later. Two then reached their reserve with nothing left to erase.
  • Under pressure, passes were at least 10 minutes apart, apart from one extra pass per node from a sync cycle completing as pressure began.
  • A restarted node reported the same 96 candidates on its first pass, 32 s after restarting.
  • Eviction removed 2,541 paid entries on 18 nodes, with and without disk pressure. Passes ended at 1,000 entries, or above it once eviction had nothing left it may remove. No store filled.
  • Four nodes that joined after the first 5 GB held 459-721 paid entries, against about 550 expected.
  • Not shown: whether hint builds stay flat as lists grow. Only three builds passed the 250 ms log line, all during the first upload, and lists this small can't show it.
  • Breaking a node's store path logged the measurement warning, and the node carried on once restored. Defect found: the warning repeats every 500 ms, because a polling path shares the measurement. Fixed in ed82e2f4.

New dependency

none. fs2 is already a dependency; this adds a call to its total_space.

ADR

• ADR-0028: pressure-gated retention of out-of-range records (new). I numbered it 0028 because main jumps from 0023 to 0027, and 0027 is already taken twice; renumber if 0024–0026 are free.
• ADR-0005 decision 11: amended by ADR-0028.
• ADR-0016: pointer pruning text updated.
• docs/REPLICATION_DESIGN.md: §11 and "Capacity-managed mode" updated. The spec already asked to evict the farthest out-of-range keys when full; nothing implemented it until now.

Mitigation / rollback

Revert this PR to go back to scheduled pruning. Watch disk usage and the prune line's disk_pressure, retained, budget_deferred, audit_below_threshold and paid total= counters. A node that reaches its reserve while still holding retained records means erasure isn't keeping up.

Summary

A node erases a record it is no longer responsible for only when its disk needs the room: farthest first, and each only after its close group proves it holds the record.

When pressure applies:

	Free share		Free bytes
Starts	below 10%	or	below twice the store's reserve
Ends	at least 15%	and	at least three times the reserve

Under pressure:

• Records go farthest from the node first. Those whose proofs failed in the current spell go last, oldest failure first.
• Each deletion needs 6 of 7 proofs from the current close group, plus the existing re-check.
• Free space is measured again before every deletion, and a failed measurement erases nothing and logs a warning.
• Passes also run between neighbour-sync cycles, once the minimum sync interval has passed since the last pass finished.

Without pressure: no prune audits are sent at all.

Why: an out-of-range copy costs a node nothing until its disk needs the room, and until then it is one more copy repair can draw on.

Paid-list entries: a retained record keeps the entry that authorises its repair, so entries are no longer pruned by rank.

• Above 15 million entries, about 85% of the store, each pass evicts down to the threshold: farthest from the node first, never entries near it or those of records it still stores. Eviction is local, since with evenly spread IDs the node has no vote on far keys, and a confirmation round can't keep up with a full list.
• Sync hints read every entry up to 100,000 entries and only those near the node above that, so their cost stops growing with the list.

Clocks: out-of-range clocks are wall-clock times, saved when they change, before a pass audits or erases anything. A restart doesn't leave a node with nothing to erase. Time offline counts, and in-range records clear their clock only once bootstrap has drained.

Also:

• Every prune deletion needs close-group proofs: the fast path and the pointer far path are gone. This supersedes fix(replication)!: require close-group proofs for every prune deletion #266.
• The prune audit budget rises to 1,024 requests per pass, about 146 far candidates.
• Pointer proofs ask the whole group with a 5 s limit per request and await every answer.
• Atomic file writes now work on Windows, which also fixes the commitment retention snapshot there.
• Removing a paid entry no longer resets its record's clock.

Code:

• src/replication/retention.rs (new): the shared pressure state, its hysteresis, the erase order and pass spacing.
• src/replication/out_of_range_clocks.rs (new): keeping and saving the clocks.
• src/replication/paid_list.rs: near ranges and eviction.
• src/replication/pruning.rs, src/replication/pointer.rs: gate on pressure, order farthest first, demote failed proofs, re-check before each delete.
• src/storage/chunk_store.rs: disk_space() measures the partition the chunk and pointer stores share.

Known limits:

• Erasure has a rate. A large node in a fast-growing network may still fall behind and reach its reserve.
• Erasure is not tied to writes. It starts within one sync round of pressure, and a node with nothing past the hysteresis keeps filling to its reserve, as a full node does today. Evicting at write time is follow-up work.
• Above 100,000 paid entries, a hint for a key far from the node can be missed where few peers are known near the key. With evenly spread IDs, a simulation of 100 to 2,000 nodes found none.
• A record whose proof failed waits while records not yet tried fill a pass's budget, and stays on disk until then.
• On filesystems that show freed space late, a pass may erase a little past the exit line, bounded by its budget.
• Reading from retained copies is a separate client change.

🤖 Generated with Claude Code

[MaidSafe-Hermes]

REQUEST_CHANGES — two source-confirmed lifecycle defects remain.

Coverage: 2/2 seats reviewed across all four scoped source bundles (sol-1 gpt-6.1-sol xhigh; glm-1 z-ai/glm-5.3), after one narrowed retry for unavailable scope responses. This is review coverage, not team consensus. The seats differed on starvation; I checked the objection against the ordering and budget code and the stated contract. The GLM review itself notes that arrivals at or above the limit can defer retries for the whole pressure spell.

The CI test, format, platform and storage checks pass. Security Audit fails on inherited crypto-bigint 0.5.5 (RUSTSEC-2026-0387); this PR does not change Cargo.lock, so it is not introduced by this change. The ADR also says the required dev testnet validation has not been run. The PR remains a draft.

No local test suite was run during this review. Author-reported test totals are not treated as independently executed results.

[MaidSafe-Hermes]

[P2] Keep failed candidates from starving under a saturated arrival stream.

erasure_order puts every never-demoted candidate before every failed candidate, then the caller applies a bounded candidate/request budget. If each pressure pass brings at least that many new eligible candidates, the budget is consumed by the new (undemoted) prefix every time; a previously failed candidate is never retried, even after its close group becomes able to prove it. This contradicts the module contract at lines 169–172 that a failed record gets another turn within the spell “however many new candidates arrive.”

Reserve bounded retry capacity or rotate the complete eligible set so failed candidates make progress under a saturated stream. Add a regression where arrivals meet or exceed the per-pass selection budget.

[mickvandijke]

Confirmed: the doc comment overclaimed, and the test only covered a few arrivals per pass. Fixed in cdfea33, which corrects the comment and ADR-0028 and renames the test: failed records wait while records not yet tried fill a pass's budget.

I'm keeping the order rather than reserving retry capacity. A waiting record stays on disk, which is the safe direction, and the budget still goes to records that may free space. The one loss is a pass where every new record fails while an older one would now pass. a_limit_takes_the_first_records_of_the_whole_order pins this order.

[MaidSafe-Hermes]

[P2] Make cleared record clocks durable before they can be reused.

This removes a clock only from memory. The saved snapshot is rewritten later, after the prune pass; if the process crashes before that write (or the write fails), the old file still contains the mature timestamp. After restart, if the record is out of range again before a completed pass durably clears the old snapshot, that stale age makes it immediately eligible instead of starting a fresh hysteresis period. This breaks the continuous-out-of-range prerequisite and can permit early erasure after a topology change.

Persist a reset/tombstone with crash-safe ordering (or use a generation scheme) before a later out-of-range observation may reuse the old clock. Add a regression for clear → interrupted/failed save → restart → out-of-range again.

[mickvandijke]

Confirmed. In cdfea33 the record pass saves its clocks before any audit or deletion, as the pointer pass already does, so a restart during those can't bring back a clock the pass stopped.

A crash before that save, or a failed write, still can. I'm not adding a tombstone or generation scheme for that window: ADR-0028 already accepts the same result after a long outage, and I've noted this case there. Erasing the record still needs disk pressure and 6 of 7 proofs from the current group. I haven't added the regression test, because it would need test hooks inside the pass to crash at the right point.

[MaidSafe-Hermes]

REQUEST_CHANGES — a paid-hint prefilter omits keys for which the target peer is responsible.

On this head, both paid-hint builders restrict enumeration to keys near this node before checking each key's actual close group. A sparse far-key region breaks that implication. With zero-extended IDs self=0, peers A=1 and B=2, and group size 2, the local radius is 2 and near_bits_for_radius returns 4. The scan around self therefore covers 0..15 and drops K=129. Yet for K=129, XOR distances are A=128, self=129, B=131; find_closest_nodes_local_with_self(K, 2) includes A and self. A therefore misses a paid hint it should receive. The same self-only prefilter is used in build_sync_hints_for_peers and build_paid_hints_for_peer.

Use a conservative filter that cannot exclude a key for which any target peer may be in the paid close group (or keep the full scan), and add regression tests for both builders with a sparse/far key.

Coverage: both seats independently returned CHANGES_REQUIRED on the current neighbor-sync scope, and both returned APPROVE on the focused lifecycle-finding re-review (2/2 seats each: sol-1 gpt-6.1-sol xhigh; glm-1 z-ai/glm-5.3). The previous four-scope review covered the earlier 20-file diff at head 406b37b; it is not claimed as full-panel coverage at this head. The prior findings were re-reviewed separately. No merge authority is implied.

[MaidSafe-Hermes]

[P2] Keep the paid-hint candidate filter conservative.

Filtering by this node's radius drops paid keys for which the target peer is still in the key's paid close group. With zero-extended IDs self=0, peers 1 and 2, group size 2, the self-radius is 2 and the 4-bit prefix scan includes only 0..15; key 129 is excluded. But for key 129, XOR distances are 128 to peer 1, 129 to self, and 131 to peer 2, so peer 1 and self are the closest two. The per-key responsibility check never sees this key. build_paid_hints_for_peer uses the same prefilter. Use a conservative candidate bound that cannot omit a key needed by a target peer, or keep the full scan, and add regression tests for both builders.

[mickvandijke]

Confirmed: where few peers are known near a key far from this node, this node or a neighbour can still be in its paid close group, and my comment saying otherwise was wrong. Fixed in 468bc3b: both builders now share one step that reads every entry up to 100,000 entries, as main does.

Above 100,000 entries it still reads only near keys, so the gap you describe remains there. I'm not keeping the full scan at any size because of its cost:

• Each entry read costs a routing table lookup, measured at 14.5 µs with 24 peers, which extrapolates to about 100 µs with 200.
• Hints are built every round and for every inbound sync, and entries are no longer pruned by rank.

I also looked at a filter based on bucket fill, which would skip nothing. On our 60-node testnet only 10 nodes had a full far bucket, so most would read everything anyway. With evenly spread IDs, a simulation of 100 to 2,000 nodes found no misses with the near filter.

Tests:

• a_neighbour_gets_a_paid_hint_for_a_key_far_from_this_node drives build_paid_hints_for_peer on a 25-node network. It failed on both runs with the full scan switched off.
• a_far_key_is_read_for_hints_until_the_list_is_too_large_to_read_whole uses your example.
• build_sync_hints_for_peers has no test of its own. It is covered only through the shared step, because it isn't public.