Repository navigation
feat(storage)!: reject recognizable file headers on client PUTs - #286
Open
mickvandijke wants to merge 3 commits into
Open
mickvandijke wants to merge 3 commits into
mickvandijke wants to merge 3 commits into
Conversation
2 of 7 tasks
mickvandijke
marked this pull request as ready for review
November 10, 2026 17:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Client chunk PUTs currently accept obvious raw files, such as a PNG, after payment. This adds a bounded header-admission heuristic before duplicate acknowledgement, payment verification, or storage. It uses file-format with deep readers disabled, an explicit vetted-format policy, and corroboration for weak signatures; headerless or disguised plaintext can still pass.
Companion client PR: WithAutonomi/ant-client#254.
Linear issue
Closes V2-1911
Risk tier
Compatibility
Semver impact
Test evidence
Rebased onto current origin/main (43a7ffd).
cargo test --locked --lib --features test-utils storage:: -- --test-threads=4 --quiet: 108 passed.
cargo test --locked --lib --features test-utils web_rtc::errors:: -- --quiet: 5 passed.
Coverage includes raw native/typed rejection, unpaid/cached-paid requests, duplicate rejection, existing reads, browser error preservation, weak crate matches, and a real self-encrypted upload round trip.
cargo fmt --all -- --check; git diff --check; python3 scripts/adr-governance.py: passed.
cargo check --locked --lib --no-default-features: passed.
Strict all-target/all-feature Clippy was attempted. Current upstream code has pre-existing warnings in src/devnet/run_dir.rs (unnecessary_wraps), replication/pointer.rs and tests/e2e/pointer_replication.rs (duration_suboptimal_units), and storage/chunk_store.rs tests (significant_drop_tightening). The same command with --cap-lints warn completes with only those unchanged-file warnings; this change introduces none.
Local isolated 100,000-iteration header-check samples measured about 0.48–0.63 microseconds/check. This is a local microbenchmark, not a network throughput claim.
Paired-branch dev-testnet validation is complete (below). Mixed-version rollout checks remain required before release. The repeatable native and Chromium tests are in companion client PR chore(release): promote rc-2026.10.2 #254.
Paired-branch local testnets passed on 2026-11-10 against enforcing node commit 8324790 and the modified client (test commit a4c5599).
Native:
cargo --config 'patch.crates-io.ant-node.path="<node-checkout>"' test -p ant-core --test e2e_chunk_admission -- --ignored --nocapture: passed in 26.28 seconds. Fourteen real QUIC nodes and an Anvil chain; 15 fixtures (PNG/JPEG/GIF/WAV/MP4/PDF/HTML/gzip/tar/SQLite/XZ/XML/ZIP/EXE/DLL) produced typed client rejections and the exact node StorageFailed message when raw requests bypassed preflight. Every rejected address remained absent from all stores; token and gas balances were unchanged. File upload self-encrypted the same PNG-prefixed bytes, paid for 3 chunks, and downloaded all 4,096 bytes correctly after clearing the client chunk cache. The existing MiniTestnet uses its test-only paid-quote membership override for its in-process topology; signatures, payments, receipts, QUIC, admission and disk storage are real.Browser:
ANT_NODE_DIR=<node-checkout> npm test -- --grep 'rejects recognizable|authenticates, resumes'from ant-core/browser-tests: both selected tests passed in 13.4 seconds, using seven modified nodes, Anvil and Chromium's real WebRTC. PNG/SQLite/XML/gzip errors reached JavaScript before any wallet callback; the PNG-prefixed file then self-encrypted, paid once, reached at least 4 replicas, and downloaded byte-for-byte. The existing paid interruption/resume/download test also passed. No transport or RPC mocks; local node listeners used OS-assigned ports.Client-upload-only enforcement confirmed: the classifier is called only by the client PUT handler; fresh replication and repair fetches write directly through ChunkStore, preserving existing data.
Live QUIC replication regression:
cargo test --locked --test e2e --features test-utils file_headers_ -- --nocapture --test-threads=1: both tests passed (12.60 seconds). PNG/gzip/XML/SQLite/XZ records rejected by client PUT are explicitly accepted by the FreshReplicationOffer receiver and stored byte-for-byte; separately, the real repair fetch worker downloads and stores the same records. Tests seed existing data and use the standard test-only payment cache, without weakening production payment checks. No production-code change was necessary for this boundary.cargo clippy --locked --test e2e --features test-utils -- -D clippy::panic -D clippy::unwrap_used -D clippy::expect_used: passed, with only the two previously documented warnings in unchanged files. Formatting, diff check, and ADR governance passed.New dependency
file-format 0.29.0, with default/deep-reader features disabled and no new transitive dependencies. Explicitly selected for this change; maintainer dependency acknowledgement belongs in review.
ADR
ADR-0030: Reject recognizable file headers on client chunk uploads — Proposed.
Mitigation / rollback
Ship the companion client preflight/error handling before fleet enforcement; revert the node admission gate if legitimate encrypted uploads are rejected. Nodes cannot refund payments older clients already submitted on-chain.