Repository navigation
docs: sync operator guides with main after the API, lock and layout changes - #99
Merged
Merged
Conversation
…hanges README, OPERATIONS, DEPLOYMENT, TROUBLESHOOTING and API-CONFIG still said there was no metrics endpoint, that cache hit rates were wrong, that a broadcast failure only logged a warning, that lock waits always ended in EIO after 30 seconds, that the inode count was fixed at 4096, and listed env vars that do not exist. They now cover the HTTP API, gfsctl and /metrics, the unicast port and the broadcast startup check, lock wait limits for live and failed holders, the v2 and pre-#85 format errors, when -skip-recovery is safe, the log lines that show whether recovery ran, what SIGINT and SIGTERM do at shutdown, and how to run the tests -short skips. Commands and log lines were checked on a test host at 30500b7. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings README.md, docs/OPERATIONS.md, DEPLOYMENT.md, TROUBLESHOOTING.md and API-CONFIG.md in line with main at 30500b7.
/metricsand the dashboard replace the old "no metrics endpoint" text. A new troubleshooting table covers API startup refusals, token-file errors, 401/403/429/503, and an unbuilt dashboard.-unicast-port, and that mount now stops when broadcast cannot start (fix(broadcast): give unicast its own port so broadcast actually starts #59, fix(mount): check broadcast ports before changing anything on disk #61). tcpdump and firewall steps cover both ports.unknown required flags 0x10on old builds. Inode count and region sizes come from the device (feat(superblock)!: size the bitmap, refcount and inode regions from the device #47).-skip-recoveryis marked emergency-only, and a log-line table shows whether recovery ran.fusermount3 -u.-shortskips.GOCLUSTERFS_DEBUGandGOCLUSTERFS_SECRETenv vars are gone.Checked on 172.25.1.68 at 30500b7 (rebased onto c3d2543; only #86/#95/#96 landed since, and the #96 OPERATIONS conflict is merged by hand):
-skip-recovery=5ignored on slot 0,kill -9then remount (replayed 2 txns, file kept), SIGTERM shutdown (exit 0),gfsctl stats cluster|cache|health,/metricsmemory lines, 403 for a foreign Host and for snapshots without a token, "Dashboard not built", the four startup refusals as quoted, and fsck clean plus-list-slots.-shortskips all passed in 244 s.-shorttook 915 s: 129 passed, 2 skipped, 3 failed.TestFsckRepairsOlderSharingandTestSFBPersistencehitfusermount: ... Device or resource busyon unmount, then the remount failed (likely the shared host).TestMultiNodeExtentCacheCoherency/TruncateRewriteread 262144 bytes, wanted 307200. These tests also run under-shortin CI, so they need a look.Code and doc mismatches found, not fixed here:
goclusterfs-fsck -hsays repair never frees blocks (cmd/fsck/main.go), but it has since feat(fsck): free leaked blocks in -repair #72.internal/api/doc.gosays the API is read-only.-sfbstill logsSFB: enabled.make test-integrationhas a 5 m timeout; the full suite takes about 15 m.journal.Config.FullThresholdis never read.DESIGN*.md drift at 53ef02a (not edited; for the author)
None of the DESIGN files changed since e31fa92. Each item: doc line, the claim, what the code does now.
DESIGN.md
:236-248"FENCE: fdatasync … VERIFY": a claim writes the node's own 4 KB slot in a 64-slot row, then reads the row once; lower slot wins a collision; no fsync or read-back. Backoff at:300-303is 10→25 ms (fix(lock): claim locks through per-node slots and cache released holds #29). Same CAS wording at DESIGN-cache.md:14, DESIGN-write-path.md:23,588.:227,427-442stale-node reclaim: age frees nothing; a claim holds until its registry slot reads empty (fix(lock): take over a lock only after its holder is released #90). A proven-dead node's write claims are adopted before replay and freed only after it succeeds.:39,156,159-178"512B per lock": 256 records × 64 claim slots × 4 KB (64 MB); each lock type owns fixed records (inode 128, extent 97, inode sector 16, SFB 8, others 1).:180-186,205-207lock type values and "ordered by inode number": Volume=0, BlockBitmap=6, Inode=8, Extent=9, new InodeSector=10 (fix(inode): lock each inode table sector around its read-modify-write #81); within a type, order is by record (fix(lock): give each lock type its own records #85).:19-31,572layout and "MaxFiles 1M": v3 = superblock 1 MB, registry @1 MB, locks @2 MB (64 MB), snapshot registry @66 MB, journals @67 MB, then device-sized bitmap, refcount and inode table; min 327 MB; one inode per 1 MB by default, 4096–16M (feat(superblock)!: size the bitmap, refcount and inode regions from the device #47).:49-50"GCLSTRFS", version 1: "GOCLUSTERFS", CRC32C, only version 3 with the TypedLockRecords flag mounts.:477-479mount sequence / volume lock: claims slot Active, adopts previous owner's write locks, replays, clears foreign claims; AcquireVolume has no callers.:113-119,150,447-464node states and HeartbeatRetries: states Empty/Active/Leaving/Dead; node exits when heartbeats fail >5 s or its slot is taken.:487-488,501-503inode then bitmap lock: order rejects that; writes allocate first; create claims the inode before the directory lock (fix(fs): claim a new inode before naming it, and check replayed entry changes under the directory lock #84).:515reads take a shared inode lock: reads take no cluster lock.:474(also :255,266,535) O_SYNC and buffered reads: O_DIRECT only, all I/O.:337-339Extents [4]Extent: 2 inline plus an overflow tree.:358-364fixed 254-byte names: variable-length entries in 4 KB chunks.:387-388roaming pointer, per-node zones: first free bit from sector 0; per-node start points only for inodes (perf(inode): start each node's free-inode search in its own part of the table #35).DESIGN-journal.md
:489-495,582-589mark STALE, journal lock: recovery CASes the slot to Dead with a 60 s lease, adopts write locks, replays, frees them only on success; journal lock unused.:41510 s heartbeat timeout: peers act on heartbeats >20 s old that they also saw unchanged for 15 s.:436-442,546skip corrupt records: scan stops at the first bad record; own-slot replay failure stops the mount;-skip-recovery=Ndiscards only slot N (fix(mount): make -skip-recovery name the slot whose journal it discards #43).:463-481"checks if already applied": a record applies only while its target keeps the logged MetaGeneration (fix(journal): free truncated and unlinked blocks on replay only while the inode still maps them #78, fix(journal): replay a record only while other nodes have not changed what it touches #80, fix(fs): claim a new inode before naming it, and check replayed entry changes under the directory lock #84).:506-508head moves after each txn, checkpoint at 75%: head moves at the 5 s checkpoint or when a record doesn't fit, never past an incomplete intent (fix(journal): keep the head behind intents whose operation has not completed #65); FullThreshold is never read.:414own journal dirty: new UUID each mount; replays the slot's previous owner's journal after adopting its locks.:26-35,41-49journal after registry, JournalSlotSize/JournalVersion fields: journals at 67 MB; only offset and size fields; slot = size/64.:131-137,444-4474 txn types: also IntentComplete ("INTC"); completed intents skipped.:558-572BatchDelay 10 ms, MaxIOErrors, async fsync: MaxBatchAge 100 ms, no MaxIOErrors, async never syncs, extra CommitBatch; mount uses CommitSync.:115TXN_MAGIC 0x54584E52: code is 0x524E5854 (also DESIGN-write-path.md:109).DESIGN-snapshot.md
:13,271-358O(1) creation with background refcounts: synchronous, touches every block, hidden data inode, retries if a write moved blocks (fix(snapshot): order in-place writes with snapshots on other nodes #42); worker only deletes.:197-216CoW under exclusive inode lock: in-place writes take a shared inode lock plus exclusive extent lock and recheck sharing; copies happen under the exclusive inode lock (fix(snapshot): order in-place writes with snapshots on other nodes #42, fix(snapshot): copy shared blocks under the exclusive inode lock #46).:242-259partial-write copy, free inside CoW: full block copied; old block released after the inode write commits; old uncounted sharing never released.:48,213,239,255"0 = free, 1 = exclusive": 0 and 1 both mean one owner; share sets max(n,1)+1; release frees at ≤1.:688-698all snapshot ops journaled: none are.:649-666incomplete creation becomes Active: rolled back at mount unless another node activated it.:363-505leaf/middle/root deletion: one worker path.:509-547CreateWritableClone: a clone is a normal root-directory file sharing blocks; other directories get ENOTSUP.:79-122registry header, MaxSnapshots 256, Name[64]: no header; 128-byte descriptors (~7,936 per MB) with a DataInode; names ≤47 bytes, unique per volume; IDs n×64 + slot.:124-130states Creating=0 … Orphaned=3: Invalid=0, Creating=1, Active=2, Deleting=3.:101,293directory subtrees, ParentSnapshotID: files only; parent always 0.:29,60-71RefCount header "GCLSREFC": no header; 511 entries + CRC32C per 4 KB sector.DESIGN-write-path.md
:23,78-82,465,492-495exclusive inode lock; existing block written with no lock: in-place writes take a shared inode lock plus exclusive extent lock (fix(snapshot): order in-place writes with snapshots on other nodes #42).:104-129,200-221,284-307journal before data, DataWrite/RefCountSet/BlockAlloc entries: data first; ExtentAdd + InodeUpdate only when blocks or size change; those entry types don't exist.:78-82,151-153lock CAS with fsync: write own slot, read row, no fsync; released holds kept 10 ms and reused up to 250 ms (fix(lock): claim locks through per-node slots and cache released holds #29).:444-449steal a dead node's lock, EBUSY: waiters never steal; wait for a failing holder's replay; give up after 30 s on a live holder with EIO (fix(lock): give up a lock wait after 30s and log who holds the lock #54).:528(and :487-488) MaxWriteRetries = 3: now 10 (fix(lock): claim locks through per-node slots and cache released holds #29).:136-137,301,323DataGeneration++/CoWGeneration++: only CoW bumps DataGeneration; no CoWGeneration.:196-198,312new block refcount 1: stays 0.:225lazy-zero remainder: write-allocated blocks are neither zeroed nor marked unwritten; only fallocated blocks are (unsure whether readable).:276-282partial copy then merge: full block copied first.DESIGN-v2.md
:8,52-159dynamic lock region, LockSlotSize 256: fixed 256 × 64 × 4 KB, split by type.:454-495lock hierarchy use: levels 0, 1, 3, 5 never taken; registry uses LockTypeSnapshot id 0; refcounts one global lock; order by record within a level (fix(lock): give each lock type its own records #85); new level 10.:15-26,37-48region order, ~16 MB locks, ~530 MB: device-sized regions (feat(superblock)!: size the bitmap, refcount and inode regions from the device #47); order registry, locks, snapshot registry, journals, bitmap, refcount, inodes; ≈589 MB on 1 TB.:9,163-320,501-558SFB with 512B "GCLSSFB" header: disabled; header 64 B, magic "SFBC".:327-387inode with StorageType, SFB, ProvisioningMode, 4 extents, snapshot counters: none exist (same at DESIGN-snapshot.md:179-186, DESIGN-thin.md:63-72).:668,686MaxInodes 1M, LockRetryMax 1 s: inode count scales with the device; retry max 25 ms (fix(lock): claim locks through per-node slots and cache released holds #29).DESIGN-cache.md
:5,13no external communication: UDP invalidation on by default, multicast 31337 and unicast 31338 (fix(broadcast): give unicast its own port so broadcast actually starts #59); mount fails if it cannot bind them (fix(mount): check broadcast ports before changing anything on disk #61).:132-142reads take a shared lock, read one sector: no lock; validation re-reads the whole inode.:36-42,197-213data write bumps both generations: MetaGeneration on every inode write; DataGeneration only on slot reuse and CoW.:150lock steal invalidates caches: adopting or freeing locks touches no cache.:30,57-58LastModifyLockSeq, LocalLockHeld, LocalLockMode: don't exist.DESIGN-thin.md
:161-180PUNCH_HOLE and ZERO_RANGE handled: both return EOPNOTSUPP.:81,126,162,186reads/writes take the inode lock, fallocate inode then bitmap: reads unlocked; fallocate allocates first.:278-299flush TRIMs every queued block: only blocks still free, held allocated during the discard (fix: stop TRIM discarding reallocated blocks and dedupe broadcasts per sender #37).:36-37hole = block 0, Length uint32: hole is MaxUint64, Length uint64.:30-31future compressed/encrypted bits: bit 3 SFB, bits 4-5 SFB tier, bit 6 FlagCounted.Fixed since the last list: DESIGN.md:579-581 (30 s lock timeout now enforced, #54) and DESIGN-snapshot.md:38-40 (refcount table sized from the device, #47).
🤖 Generated with Claude Code