Skip to content

feat(storage)!: reject recognizable file headers on client PUTs - #286

Open
mickvandijke wants to merge 3 commits into
mainfrom
feat/reject-plaintext-chunk-uploads
Open

mickvandijke wants to merge 3 commits into
mainfrom
feat/reject-plaintext-chunk-uploads

Conversation

@mickvandijke

@mickvandijke mickvandijke commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Client chunk PUTs currently accept obvious raw files, such as a PNG, after payment. This adds a bounded header-admission heuristic before duplicate acknowledgement, payment verification, or storage. It uses file-format with deep readers disabled, an explicit vetted-format policy, and corroboration for weak signatures; headerless or disguised plaintext can still pass.

Companion client PR: WithAutonomi/ant-client#254.

Linear issue

Closes V2-1911

Risk tier

  • T0 — docs / tooling / CI / pure UX-output. Repo CI only.
  • T1 — client-only, no network-facing behavior change. CI + prod compat smoke.
  • T2 — node/client logic with behavioral surface, no protocol/format/economics change. Dev testnet + ADR.
  • T3 — protocol / storage format / payments / routing. T2 evidence + adversarial testing.

Compatibility

  • Wire: Existing ChunkPutResponse::Error(StorageFailed) variant and wire discriminants remain unchanged. The exact new admission message is preserved through browser redaction so the client can distinguish it from disk-full.
  • Storage: No layout or migration change. Existing reads and node replication remain available; duplicate client PUTs with recognizable headers are refused.
  • API: Client PUT admission changes: recognizable raw file headers are no longer accepted. No public Rust API signature changes. This heuristic does not verify self-encryption.

Semver impact

  • breaking
  • feature
  • fix

Test evidence

  • Rebased onto current origin/main (43a7ffd).

  • cargo test --locked --lib --features test-utils storage:: -- --test-threads=4 --quiet: 108 passed.

  • cargo test --locked --lib --features test-utils web_rtc::errors:: -- --quiet: 5 passed.

  • Coverage includes raw native/typed rejection, unpaid/cached-paid requests, duplicate rejection, existing reads, browser error preservation, weak crate matches, and a real self-encrypted upload round trip.

  • cargo fmt --all -- --check; git diff --check; python3 scripts/adr-governance.py: passed.

  • cargo check --locked --lib --no-default-features: passed.

  • Strict all-target/all-feature Clippy was attempted. Current upstream code has pre-existing warnings in src/devnet/run_dir.rs (unnecessary_wraps), replication/pointer.rs and tests/e2e/pointer_replication.rs (duration_suboptimal_units), and storage/chunk_store.rs tests (significant_drop_tightening). The same command with --cap-lints warn completes with only those unchanged-file warnings; this change introduces none.

  • Local isolated 100,000-iteration header-check samples measured about 0.48–0.63 microseconds/check. This is a local microbenchmark, not a network throughput claim.

  • Paired-branch dev-testnet validation is complete (below). Mixed-version rollout checks remain required before release. The repeatable native and Chromium tests are in companion client PR chore(release): promote rc-2026.10.2 #254.

  • Paired-branch local testnets passed on 2026-11-10 against enforcing node commit 8324790 and the modified client (test commit a4c5599).

  • Native: cargo --config 'patch.crates-io.ant-node.path="<node-checkout>"' test -p ant-core --test e2e_chunk_admission -- --ignored --nocapture: passed in 26.28 seconds. Fourteen real QUIC nodes and an Anvil chain; 15 fixtures (PNG/JPEG/GIF/WAV/MP4/PDF/HTML/gzip/tar/SQLite/XZ/XML/ZIP/EXE/DLL) produced typed client rejections and the exact node StorageFailed message when raw requests bypassed preflight. Every rejected address remained absent from all stores; token and gas balances were unchanged. File upload self-encrypted the same PNG-prefixed bytes, paid for 3 chunks, and downloaded all 4,096 bytes correctly after clearing the client chunk cache. The existing MiniTestnet uses its test-only paid-quote membership override for its in-process topology; signatures, payments, receipts, QUIC, admission and disk storage are real.

  • Browser: ANT_NODE_DIR=<node-checkout> npm test -- --grep 'rejects recognizable|authenticates, resumes' from ant-core/browser-tests: both selected tests passed in 13.4 seconds, using seven modified nodes, Anvil and Chromium's real WebRTC. PNG/SQLite/XML/gzip errors reached JavaScript before any wallet callback; the PNG-prefixed file then self-encrypted, paid once, reached at least 4 replicas, and downloaded byte-for-byte. The existing paid interruption/resume/download test also passed. No transport or RPC mocks; local node listeners used OS-assigned ports.

  • Client-upload-only enforcement confirmed: the classifier is called only by the client PUT handler; fresh replication and repair fetches write directly through ChunkStore, preserving existing data.

  • Live QUIC replication regression: cargo test --locked --test e2e --features test-utils file_headers_ -- --nocapture --test-threads=1: both tests passed (12.60 seconds). PNG/gzip/XML/SQLite/XZ records rejected by client PUT are explicitly accepted by the FreshReplicationOffer receiver and stored byte-for-byte; separately, the real repair fetch worker downloads and stores the same records. Tests seed existing data and use the standard test-only payment cache, without weakening production payment checks. No production-code change was necessary for this boundary.

  • cargo clippy --locked --test e2e --features test-utils -- -D clippy::panic -D clippy::unwrap_used -D clippy::expect_used: passed, with only the two previously documented warnings in unchanged files. Formatting, diff check, and ADR governance passed.

New dependency

file-format 0.29.0, with default/deep-reader features disabled and no new transitive dependencies. Explicitly selected for this change; maintainer dependency acknowledgement belongs in review.

ADR

ADR-0030: Reject recognizable file headers on client chunk uploads — Proposed.

Mitigation / rollback

Ship the companion client preflight/error handling before fleet enforcement; revert the node admission gate if legitimate encrypted uploads are rejected. Nodes cannot refund payments older clients already submitted on-chain.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant