Skip to content

bench: partial materialization cases and Git working-copy baseline - #323

Merged
geekgonecrazy merged 10 commits into
checkpoint/span-native-integrationfrom
bench/partial-materialize-and-git-baseline
Nov 10, 2026
Merged

geekgonecrazy merged 10 commits into
checkpoint/span-native-integrationfrom
bench/partial-materialize-and-git-baseline

Conversation

@geekgonecrazy

@geekgonecrazy geekgonecrazy commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #319 (base: checkpoint/span-native-integration).

What

Two benchmark harnesses under tools/materialize-bench with the frozen-runner identity discipline (fingerprinted offline release builds, pinned binary SHA-256, raw per-sample JSON retained), plus two performance fixes the harnesses uncovered:

  1. Partial materialization cases in the paired graph/span materialize_bench (materialize_paths_noop, materialize_paths_missing, materialize_prefix_missing), with per-sample assertions that the partial call rewrites exactly its subset.
  2. Agent-sandbox sync benchmark (libatomic/examples/agent_sandbox_bench.rs + run_agent_sandbox.py, --span for the span backend): a cold, empty sandbox syncing over the production daemon RPC stack — tonic server hosting the real SandboxImpl handlers on a Unix socket, fresh client channel per sample, streaming Materialize RPC, partial via path_prefix, files received through write_sandbox_entry — with git clone over file:// as the wire-protocol comparison on matched synthetic histories. Partial sandboxes must hold exactly the requested subtree.
  3. perf: verified-change parse cache (e17a086) — change files are content-addressed and write-once, so a parsed, hash-verified change is immutable for the handle's lifetime; cache it.
  4. perf: shared span render context per view walk (66c2f6d) — materialize_view_entries built a fresh SpanRenderContext per rendered file (4,213 contexts per materialize at 2,048 files, ~20.7 s of rayon-hidden CPU); now built once per walk, lazily, shared across the rayon chunks.

Every sample is byte/mode-verified; atomic samples additionally verify a single completed working-copy operation head.

Recorded results (macOS/aarch64, loopback)

Local materializer (many-files: 24 files × 128 lines, 8 measured iterations):

Case Atomic graph Atomic span
full missing 564.8 ms 573.3 ms
partial noop (8 files) 187.1 ms 186.0 ms
partial missing (8 files) 316.8 ms 313.3 ms
prefix missing (10 files) 372.6 ms 364.7 ms
render probe (session) 24.2 ms 17.2 ms

Cold-sandbox sync at monorepo scale (2,048 files ≈ 11 MB, 49 commits, agent requests a 64-file subtree; 5 iterations):

Case graph span, before fixes span, after fixes
atomic RPC full 781 ms 34,645 ms 794 ms
atomic RPC partial 307 ms 1,127 ms 341 ms
atomic COW sandbox provision — — 276 ms
git clone full 732 ms 732 ms 732 ms

Findings

  • Span full-render collapse, found and fixed by this benchmark: without the fixes, a 2,048-file span view rendered in 34.6 s — every per-file span context re-parsed the 11 MB baseline change (~85 ms per render), and 4,213 contexts were built per materialize. The parse cache (e17a086) plus the shared context (66c2f6d) bring span rendering to parity with the graph backend.
  • Partial materialization is not proportional to the subset. 8-of-25 files takes ~55% of full time, and the partial noop floor (~187 ms) equals the full noop — the journaled plan/visibility/lease/journal path dominates before any write. That floor is the remaining optimization target; path_prefix prunes the render but the baseline-change parse and the journaled operation path still scale with total repo size.
  • Span capability is within 2% on graph-vs-span local materializer cases; its advantage stays confined to the render probe.
  • Cold sandbox over the wire: at 2,048 files the atomic partial holds ~2.3× over git clone on both backends; at 256-file scale full sync is parity (~3%). Loopback hides Git's extra round trips (clone negotiation, per-object work), so Git numbers are its upper bound.
  • COW provision is the fastest cold start but scales with file count, not bytes: provision_sandbox reflinks one file at a time (deliberate single path), so 2,049 files cost ~276 ms of syscall overhead at ~0.13 ms/file. macOS clonefile(2) clones a whole tree in one syscall and would make this near-instant, but it is whole-tree — cloning .atomic would snapshot the shared database, so the fast path needs design around excluding the repository directory.

Testing

  • python3 -B -m unittest discover -s tools/materialize-bench -p 'test_*.py' — 9 tests pass
  • cargo test -p libatomic --example agent_sandbox_bench (span smoke), cargo test --offline --locked -p atomic-repository --example materialize_bench — smoke passes every case on both backends
  • cargo test -p atomic-repository --lib — 1,559 passed with both perf changes; integration materialize tests pass
  • cargo clippy -p atomic-repository -p libatomic --example agent_sandbox_bench — clean

vinceblock99 and others added 5 commits November 7, 2026 18:05
Every caller of retrieve_span_if_present built a fresh SpanRenderContext,
which verifies and decodes the whole visible closure, for every file it
rendered. On a span import of hyperfine (1,097 commits, 75 files) one
`atomic view switch` built the context 312 times, `atomic record -a` 466
times and `atomic restore -f` 157 times, at roughly 55 ms each, while the
materialization itself took milliseconds.

- Add SharedSpanRender: a lazily built context shared by every file read
  through one transaction and visibility closure. The per-file root,
  capability and visibility checks are unchanged; only the closure decode
  is shared.
- Use it in project_tree, project_change_closure_with_txn and
  plan_file_moves, which render every projected file in one transaction.
- Add Repository::get_file_contents_on_view and recorded_presence_on_view
  so callers that read many paths get one transaction, one projection and
  one context, with the absent-liveness check scoped to exactly those
  paths. Use them in the shim checkpoint observation, working-copy
  verification and the CLI and daemon restore paths, which previously
  rendered (and discarded) every file one at a time.

Batched reads are tested byte for byte against the single-path reads for
graph-origin, span-rooted, empty, directory and absent paths on two views.
On the hyperfine span import, `atomic view switch` drops from 25-34 s to
4-7 s (12 context builds) and `atomic restore -f` of all 75 files from
15 s to 4.8 s (8 builds); graph-only repositories are unaffected.
Checkpoint observation no longer reads graph content for paths that carry
conflict-marker bytes. Name-conflict errors carry the same path and
claimant counts as the single-path reader. The daemon and CLI restore
paths classify every path up front but still walk and report them in
status order, so untracking, failures and output lines happen exactly
where they did before. The batch-read test now asserts that the fixture
really records span-rooted files.
Add materialize_paths_noop, materialize_paths_missing and
materialize_prefix_missing to the paired graph/span materialization
benchmark, with per-sample subset write-count assertions, and a new
git_baseline.py harness that times git checkout -- full working-tree
restores on the same synthetic history shape.

First paired run (macOS/aarch64, many-files, 8 iterations): partial
materialize_paths restore of 8 of 25 files takes ~315 ms against
~565 ms full, while the partial noop floor (~187 ms) equals the full
noop — the journaled plan/lease/journal path dominates, not content
work. Span capability stays within 2% on every materialize case.
Git restores the full tree in ~22 ms.
Add libatomic/examples/agent_sandbox_bench.rs plus a frozen runner: a
cold, empty sandbox syncs over the production tonic stack (SandboxImpl
handlers on a Unix socket, streaming Materialize with path_prefix,
write_sandbox_entry receiver), with git clone over file:// as the
wire-protocol comparison on the same synthetic history shape. Partial
sandboxes must hold exactly the requested subtree and nothing beyond it.

Recorded loopback runs: full-sync parity with git clone at both scales
(66 vs 72 ms small, 190 vs 185 ms large). The atomic prefix partial
holds a near-constant request phase because the handler walks the whole
view closure and filters at stream time — filtering during projection is
the measured next optimization.
@geekgonecrazy
geekgonecrazy force-pushed the bench/partial-materialize-and-git-baseline branch from 8efcd41 to 827337a Compare November 10, 2026 01:02
load_verified_change re-read and re-parsed the change file on every
call. Change files are content-addressed and write-once, so a parsed,
hash-verified change is immutable for the handle's lifetime; cache it.
revalidate_verified_change callers keep the old re-read-and-compare
behavior, and explicit revalidation requests bypass the cache.

Measured on the agent-sandbox benchmark (2,048-file span fixture, 49
changes, one 11 MB baseline): a full view render went from 34.6 s to
1.08 s — every per-file span context was re-parsing the baseline, at
roughly 85 ms per render. Span partial materialize dropped from 1.13 s
to 634 ms. Graph-origin paths do not consult this cache and are
unchanged.
@geekgonecrazy
geekgonecrazy force-pushed the bench/partial-materialize-and-git-baseline branch from 3a1aded to e17a086 Compare November 10, 2026 05:20
materialize_view_entries rendered each file through a fresh
SpanRenderContext, and every context decodes and verifies the whole
visible closure: a 2,048-file span view built 4,213 contexts per
materialize (traced), ~20.7 s of combined CPU hidden by rayon
parallelism.

SharedSpanContext builds once per walk, lazily on the first span-rooted
file, and is safe to share across the rayon render chunks (OnceLock;
single-threaded OnceCell cannot cross them). Graph-origin files still
fall back to the graph renderer, and callers without a shared context
keep the old per-call path.

Measured with the parse cache on the agent-sandbox benchmark (2,048-file
span fixture): partial materialize 639 -> 341 ms, full 1,091 -> 794 ms.
Span render now matches the graph backend on this fixture.
… into bench/partial-materialize-and-git-baseline

# Conflicts:
#	atomic-repository/src/repository/content.rs
The materialize_bench comparison set is atomic-internal: paired
graph/span cases and partial-vs-full ratios. Wire-protocol Git
comparison stays in the agent-sandbox benchmark, where both sides
start from nothing over their real transports.
atomic_sandbox_cow times Repository::provision_sandbox on the same
synthetic fixture: the local copy-on-write clone an agent gets on a
machine that already holds the repository. Measured (2,048 files,
11 MB, reflink on APFS): 276 ms median — per-file reflink syscall
overhead dominates at ~0.13 ms/file, so provision time scales with
file count, not bytes. Beats the RPC partial (336 ms) and git clone
(779 ms); the whole-tree clonefile(2) fast path is the follow-up that
would make it near-instant.
@geekgonecrazy
geekgonecrazy marked this pull request as ready for review November 10, 2026 06:34
@geekgonecrazy
geekgonecrazy merged commit 6524c44 into checkpoint/span-native-integration Nov 10, 2026
@geekgonecrazy
geekgonecrazy deleted the bench/partial-materialize-and-git-baseline branch November 10, 2026 17:36
geekgonecrazy added a commit that referenced this pull request Nov 11, 2026
…#319)

* feat: checkpoint native span integration

Add capability-gated schema-3 span changes, native recording and rendering, and transactional insertion of indexed and incoming span changes with portable semantic owners. Preserve schema-2 compatibility and add focused lifecycle and insertion tests.

Checkpoint only: spans are not enabled by default. Concurrent span conflicts and span-backed name conflicts currently fail closed; rename with simultaneous content edits and legacy span semantic-owner migration remain incomplete. The diagnostic 25-suite core harness reported 8 passed and 17 failed suites, including 5 that exited before their assertion summaries.

* fixing harness 1-9 for spans

* feat: checkpoint span bridge and consolidated shim harness for manual testing

Preserve the exact working tree developed and tested in atomic-span-clean, including experimental span import, projected semantics, durability fencing, merge fixes, and consolidated shim coverage. AI-assisted development checkpoint; not a production-readiness claim. Span import remains opt-in and strict shim guard failures remain documented. No remote publication.

* native atomic harness runs complete

* shim fix for spans and doctor bug

* final docs

* faster materialization with spans

* bench: partial materialization cases and Git working-copy baseline (#323)

* perf: share one span render context per transaction

Every caller of retrieve_span_if_present built a fresh SpanRenderContext,
which verifies and decodes the whole visible closure, for every file it
rendered. On a span import of hyperfine (1,097 commits, 75 files) one
`atomic view switch` built the context 312 times, `atomic record -a` 466
times and `atomic restore -f` 157 times, at roughly 55 ms each, while the
materialization itself took milliseconds.

- Add SharedSpanRender: a lazily built context shared by every file read
  through one transaction and visibility closure. The per-file root,
  capability and visibility checks are unchanged; only the closure decode
  is shared.
- Use it in project_tree, project_change_closure_with_txn and
  plan_file_moves, which render every projected file in one transaction.
- Add Repository::get_file_contents_on_view and recorded_presence_on_view
  so callers that read many paths get one transaction, one projection and
  one context, with the absent-liveness check scoped to exactly those
  paths. Use them in the shim checkpoint observation, working-copy
  verification and the CLI and daemon restore paths, which previously
  rendered (and discarded) every file one at a time.

Batched reads are tested byte for byte against the single-path reads for
graph-origin, span-rooted, empty, directory and absent paths on two views.
On the hyperfine span import, `atomic view switch` drops from 25-34 s to
4-7 s (12 context builds) and `atomic restore -f` of all 75 files from
15 s to 4.8 s (8 builds); graph-only repositories are unaffected.

* fix: keep batched restore and checkpoint reads behaviour-identical

Checkpoint observation no longer reads graph content for paths that carry
conflict-marker bytes. Name-conflict errors carry the same path and
claimant counts as the single-path reader. The daemon and CLI restore
paths classify every path up front but still walk and report them in
status order, so untracking, failures and output lines happen exactly
where they did before. The batch-read test now asserts that the fixture
really records span-rooted files.

* fix: stream checkpoint and verify file contents

* feat: partial materialization cases and Git working-copy baseline

Add materialize_paths_noop, materialize_paths_missing and
materialize_prefix_missing to the paired graph/span materialization
benchmark, with per-sample subset write-count assertions, and a new
git_baseline.py harness that times git checkout -- full working-tree
restores on the same synthetic history shape.

First paired run (macOS/aarch64, many-files, 8 iterations): partial
materialize_paths restore of 8 of 25 files takes ~315 ms against
~565 ms full, while the partial noop floor (~187 ms) equals the full
noop — the journaled plan/lease/journal path dominates, not content
work. Span capability stays within 2% on every materialize case.
Git restores the full tree in ~22 ms.

* feat: agent-sandbox sync benchmark over the daemon Materialize RPC

Add libatomic/examples/agent_sandbox_bench.rs plus a frozen runner: a
cold, empty sandbox syncs over the production tonic stack (SandboxImpl
handlers on a Unix socket, streaming Materialize with path_prefix,
write_sandbox_entry receiver), with git clone over file:// as the
wire-protocol comparison on the same synthetic history shape. Partial
sandboxes must hold exactly the requested subtree and nothing beyond it.

Recorded loopback runs: full-sync parity with git clone at both scales
(66 vs 72 ms small, 190 vs 185 ms large). The atomic prefix partial
holds a near-constant request phase because the handler walks the whole
view closure and filters at stream time — filtering during projection is
the measured next optimization.

* perf: cache verified change parses in the change store

load_verified_change re-read and re-parsed the change file on every
call. Change files are content-addressed and write-once, so a parsed,
hash-verified change is immutable for the handle's lifetime; cache it.
revalidate_verified_change callers keep the old re-read-and-compare
behavior, and explicit revalidation requests bypass the cache.

Measured on the agent-sandbox benchmark (2,048-file span fixture, 49
changes, one 11 MB baseline): a full view render went from 34.6 s to
1.08 s — every per-file span context was re-parsing the baseline, at
roughly 85 ms per render. Span partial materialize dropped from 1.13 s
to 634 ms. Graph-origin paths do not consult this cache and are
unchanged.

* perf: share one span render context per view walk

materialize_view_entries rendered each file through a fresh
SpanRenderContext, and every context decodes and verifies the whole
visible closure: a 2,048-file span view built 4,213 contexts per
materialize (traced), ~20.7 s of combined CPU hidden by rayon
parallelism.

SharedSpanContext builds once per walk, lazily on the first span-rooted
file, and is safe to share across the rayon render chunks (OnceLock;
single-threaded OnceCell cannot cross them). Graph-origin files still
fall back to the graph renderer, and callers without a shared context
keep the old per-call path.

Measured with the parse cache on the agent-sandbox benchmark (2,048-file
span fixture): partial materialize 639 -> 341 ms, full 1,091 -> 794 ms.
Span render now matches the graph backend on this fixture.

* chore: drop the local git working-copy baseline harness

The materialize_bench comparison set is atomic-internal: paired
graph/span cases and partial-vs-full ratios. Wire-protocol Git
comparison stays in the agent-sandbox benchmark, where both sides
start from nothing over their real transports.

* feat: COW sandbox provision case for the agent-sandbox benchmark

atomic_sandbox_cow times Repository::provision_sandbox on the same
synthetic fixture: the local copy-on-write clone an agent gets on a
machine that already holds the repository. Measured (2,048 files,
11 MB, reflink on APFS): 276 ms median — per-file reflink syscall
overhead dominates at ~0.13 ms/file, so provision time scales with
file count, not bytes. Beats the RPC partial (336 ms) and git clone
(779 ms); the whole-tree clonefile(2) fast path is the follow-up that
would make it near-instant.

---------

Co-authored-by: Vincent <vince@atomic.dev>

* style: cargo fmt the merged vault handler

* feat: switch-from-empty case for the materialization benchmark (#324)

switch_from_empty rests on a view where every tracked file is absent
(untimed setup switch) and times the switch to the full view: maximal
pure materialize work through the switch entry point — every tracked
file rendered and written, no deletions, no revert machinery.

Measured many-files: 887 ms graph / 872 ms span, comparable to
switch_many — the switch machinery itself, not the write volume,
dominates at this fixture size.

* style: cargo fmt across the merged tree

* fix: hex-encode the sha256 digest without LowerHex

The digest array no longer implements LowerHex after the dependency
refresh on dev; encode explicitly with HEXLOWER instead.

* fix: get CI green on the span branch (#328)

* fix: build agent_sandbox_bench only on Unix

The benchmark drives the daemon over a Unix domain socket, so the
example did not compile on Windows and the Windows test jobs never
ran. Gate its items to Unix and leave a stub main elsewhere.

* fix: update span import for the git2 0.21 shorthand API

Reference::shorthand returns a Result since git2 0.21, so the
experimental-span-import feature no longer compiled. CI does not
build that feature.

* fix: let the Git import preflight repair a database left by a killed process

The preflight opened the database without recovery and refused any
database that needed repair, so `git shim reconcile` failed after the
watch daemon was killed. Allow redb's own storage-level repair once
the common lock is held. It changes no repository state. Dry runs,
format upgrades and repository recovery are still refused.

Adds a regression test using a copy of a repository that is still open.

* fix: stale tests and two small behaviour gaps on the span branch

- Span incremental import takes its target from the pinned Git branch
  instead of Repository::current_view(). The refusal when the current
  view differs is unchanged; it moves next to the importer's own check.
- for_each_switch_target_file returns early when there is nothing to
  read, so a switch that writes no files no longer needs the target
  view's dependency closure.
- whole_file_diff_test: pure deletions and a missing final newline are
  no longer stored as whole-file rewrites. Keep the diff assertions and
  drop the rewrite precondition for those three fixtures.
- remove_recorded_path_test: count oneline log entries by line instead
  of by the default format's "#N" header.
- routing_parity_test: the service path now prints the same follow-up
  hints as the local path.

* test: unix-gate tests exercising the unix-only projected span pipeline

Projected semantic recording is refused off-unix (lossless_unix_paths =
cfg!(unix)), so the 16 stream tests and 2 native_span_record tests that
exercise it failed on Windows CI. Gate them #[cfg(unix)], matching the
existing pattern in both files.

span_compiler_replays_imported_prefix_and_matches_fresh_extension gets a
note: it raw-reads the open redb database to assert read-only validation,
which Windows refuses with a sharing violation (OS error 33) regardless
of the span pipeline.

---------

Co-authored-by: Aaron Ogle <geekgonecrazy@users.noreply.github.com>
Co-authored-by: Vincent <vince@atomic.dev>
Co-authored-by: Aaron Ogle <aaron@geekgonecrazy.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants