Skip to content

Race conditions in concurrent multi-node file operations #1

Open

Description

@mattmattox

Summary

Integration tests have revealed race conditions when multiple nodes perform concurrent file operations on a shared filesystem.

Affected Tests

Two tests are currently skipped in CI (-skip flag in .github/workflows/ci.yml) because they correctly expose these bugs:

1. TestMultiNodeConcurrentDirectoryOps

Symptom: Reading node1_0.txt returns content from node0_0.txt

Root Cause (hypothesis): When multiple nodes create files simultaneously:

  • Both nodes allocate inodes and data blocks concurrently
  • The extent pointer in one inode may reference the wrong data block
  • Possible race in bitmap allocation or inode extent assignment

Log evidence:

Content mismatch: got "Content from node 0, file 0", expected "Content from node 1, file 0"

2. TestMultiNodeConcurrentSameFileWrites

Symptom: File is too small - only one node's write persisted

Root Cause (hypothesis): When two nodes write to the same file at different offsets:

  • Lost-update race condition in block allocation or extent tracking
  • One node's extent update overwrites the other's

Log evidence:

File too small: got 524286 bytes, expected at least 1048572

Investigation Areas

  1. Bitmap allocation (internal/bitmap/): Check CAS operations for ABA issues
  2. Inode extent assignment (internal/extent/): Verify extent list updates are atomic
  3. Lock ordering: Ensure no TOCTOU between reading and updating inode state
  4. Cache coherency: Verify stale cache doesn't cause incorrect extent lookups

Reproduction

# Run the specific failing tests (currently skipped in CI)
go test -v -run 'TestMultiNodeConcurrentDirectoryOps|TestMultiNodeConcurrentSameFileWrites' ./tests/integration/...

Priority

Medium - The tests correctly identify real bugs. Core single-node operations work correctly, but concurrent multi-node writes have race conditions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions