Repository navigation
merkle: let the thread joining the update run queued shard tasks - #987
Open
citizen-stig wants to merge 1 commit into
Open
citizen-stig wants to merge 1 commit into
citizen-stig wants to merge 1 commit into
Conversation
This was referenced Nov 18, 2026
The commit_concurrency workers and the thread waiting in UpdateHandle::join take shard tasks from a shared queue, so the caller works instead of idling once its own part of Session::finish is done. No threads are added. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
citizen-stig
force-pushed
the
nikolai/merkle-caller-runs-tasks
branch
from
November 18, 2026 09:39
9476e95 to
210b841
Compare
citizen-stig
marked this pull request as ready for review
November 18, 2026 12:25
This was referenced Nov 18, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Updater::updateputs the merkle update tasks (one per page-cache shard) in a shared queue. Thecommit_concurrencypool workers and the thread that callsUpdateHandle::joinall take tasks from it until it is empty.Why
After #978 there are 64 shard tasks. The thread calling
Session::finishused to block injoinwhile the pool worked through them. Now it runs tasks too, sofinishgets one more worker without adding a thread.Benchmarks
finish_benchmeanfinishover 8 interleaved runs. Master6ee36d3is run atcommit_concurrencyN and N+1, this PR at N:into_overlayis unchanged in every configuration (53–55 ms).Is this just
commit_concurrency + 1?Since #978, master sends 64 shard tasks to a
commit_concurrency-thread pool, and the pool's queue already balances them across its threads. This PR adds one more consumer: the thread callingjoin, which would otherwise sleep inrecv. Raisingcommit_concurrencyby one also gives N+1 consumers, but it adds an OS thread.So the change is worth keeping where a node already sets
commit_concurrencyto the core count or higher. With spare cores, raisingcommit_concurrencyby one gets the same result. Also,finishkeeps scaling past 6 workers on this machine: master goes from 95 ms at 6 to 62 ms at 13.finish_bench: an isolatedSession::finishbenchmark with the user-namespace access profile of a Sovereign SDK node (benchtop/src/bin/finish_bench.rs@ 3469d5c, not part of this PR; a bench-only env var setscommit_concurrency). 50k–145kReadThenWriteaccesses per slot (68% updates with the prior value, 32% inserts), 36-byte values,BinaryHasher<Sha256>, rollback on, tree growing from 5.35M leaves, 10 warm-up + 20 measured slots from a restored seed DB. AMD Ryzen 7 7700 (8 cores, 16 threads). "12 pinned" istaskset -c 2-7,10-15; "all 16" istaskset -c 0-15. All 9 configurations ran in one batch, with order rotated each round. Every run ends at the same root.🤖 Generated with Claude Code