Deadlock of forked concurrent HDF5RealizationSink() calls in macOS #70

Hi,

I was working on GSVA when I stumbled upon this problem, concretely when GSVA was trying to write on multiple independent HDF5 files in parallel in macOS. I did not find this problem in my Linux workstation. Claude Code has helped me to pin down the problem and draft the rest of this message.

When several forked R processes create HDF5 datasets in the dump directory at the same time, e.g., with HDF5RealizationSink() in BiocParallel::MulticoreParam workers, the file locks that protect the dump log and the global counters can be acquired by two processes at once on macOS. From then on, a process unlocks a lock held by another one, the dump log ends up existing under both its name and its locked name, and every process trying to lock it waits forever. The process that held it fails with failed to unlock '.../HDF5Array_dump_...' file.

Minimal reproducible example

suppressPackageStartupMessages({
    library(HDF5Array)
    library(BiocParallel)
})

tempdir()  ## where HDF5Array keeps its counter, log and lock files

## many HDF5RealizationSink() calls from forked workers; each one takes the
## locks of the dump name counter, the dump log and the dataset creation
## counter
res <- bplapply(seq_len(100), function(i) {
    sink <- HDF5RealizationSink(c(10L, 10L))
    close(sink)
    i
}, BPPARAM=MulticoreParam(8))
Error: BiocParallel errors
  1 remote errors, element index: 55
  10 unevaluated and other errors
  first remote error:
Error in unlock_file(filepath): failed to unlock '/var/folders/vl/l03_ttg911bg66mfxylsh5g40000gn/T//RtmpOK0EiU/HDF5Array_dump_names_global_counter' file
res <- bplapply(seq_len(100), function(i) {
    sink <- HDF5RealizationSink(c(10L, 10L))
    close(sink)
    i
}, BPPARAM=MulticoreParam(8))
Stop worker failed with the error: wrong args for environment subassignment
Error: BiocParallel errors
  0 remote errors, element index: 
  100 unevaluated and other errors
  first remote error:

In these 2 consecutive runs, bplapply() stopped progressing and gave different errors after a few seconds, and a third run never returned. The workers stayed in HDF5Array:::lock_file(), sleeping between attempts. Listing the directory printed by tempdir() shows both files, HDF5Array_dump_log and HDF5Array_dump_log-locked:

list.files(tempdir())
[1] "HDF5Array_dataset_creation_global_counter"
[2] "HDF5Array_dump"                           
[3] "HDF5Array_dump_log"                       
[4] "HDF5Array_dump_log-locked"                
[5] "HDF5Array_dump_names_global_counter"      

What happens

Apparently, lock_file() locks a file by renaming it to <file>-locked, and unlock_file() renames it back, both through HDF5Array:::.safe_file_rename(from, to), i.e., !file.exists(to) && file.rename(from, to). To see the interleaving, we replaced HDF5Array:::.safe_file_rename() with a version recording each call on the dump log, with its process id, whether to existed, and the result. The first anomaly in the trace is two successful renames of the same file, one microsecond apart (the columns are time, pid, rename, whether to existed, result):

1791462904.642608 54102 HDF5Array_dump_log->HDF5Array_dump_log-locked exists_to=FALSE ok=TRUE
1791462904.642609 54100 HDF5Array_dump_log->HDF5Array_dump_log-locked exists_to=FALSE ok=TRUE

Both processes then believe they hold the lock. Process 54100 unlocks it, other processes lock and unlock it, and process 54102 later "unlocks" a lock taken by another process (54096):

1791462904.666834 54096 HDF5Array_dump_log->HDF5Array_dump_log-locked exists_to=FALSE ok=TRUE
1791462904.667336 54102 HDF5Array_dump_log-locked->HDF5Array_dump_log exists_to=FALSE ok=TRUE
...
1791462904.679399 54096 HDF5Array_dump_log-locked->HDF5Array_dump_log exists_to=TRUE ok=FALSE

Process 54096, still believing it holds the lock, appends its entry with cat(..., file=locked_path, append=TRUE). Since locked_path no longer exists, this creates a new HDF5Array_dump_log-locked with just that entry. Its unlock then fails, because HDF5Array_dump_log exists, and from then on lock_file() finds HDF5Array_dump_log-locked and retries forever in every process.

So there are three weaknesses, the first of which triggers the other two:

1. Two holders (on macOS). Two concurrent file.rename() calls of the same file both reported success on macOS, while on Linux only one of them succeeds. POSIX allows rename() to return success without doing anything when both names resolve to the same file, which may happen on macOS when the second call resolves its paths on either side of the first rename. The check !file.exists(to) before renaming cannot prevent it, since it is not atomic with the rename.
2. Any process can unlock. unlock_file() renames the file back whoever holds the lock, so a process that wrongly believes it holds the lock releases another one's.
3. No recovery. A failed unlock leaves the lock taken forever, and lock_file() has no timeout, so the failure becomes a deadlock of all processes.

Environment

Reproduced on macOS 26.7 (APFS), R 4.6.1, HDF5Array 1.41.3, BiocParallel 1.47.0:

sessionInfo()
R version 4.6.1 (2026-06-24)
Platform: aarch64-apple-darwin23
Running under: macOS Tahoe 26.7

Matrix products: default
BLAS:   /Library/Frameworks/R.framework/Versions/4.6/Resources/lib/libRblas.0.dylib 
LAPACK: /Library/Frameworks/R.framework/Versions/4.6/Resources/lib/libRlapack.dylib;  LAPACK version 3.12.1

locale:
[1] C.UTF-8/UTF-8/C.UTF-8/C/C.UTF-8/C.UTF-8

time zone: Europe/Madrid
tzcode source: internal

attached base packages:
[1] stats4    stats     graphics  grDevices utils     datasets  methods  
[8] base     

other attached packages:
 [1] BiocParallel_1.47.0   HDF5Array_1.41.3      h5mread_1.5.3        
 [4] rhdf5_2.57.18         DelayedArray_0.39.8   SparseArray_1.13.4   
 [7] S4Arrays_1.13.2       IRanges_2.47.5        abind_1.4-8          
[10] S4Vectors_0.51.10     MatrixGenerics_1.25.0 matrixStats_1.5.0    
[13] BiocGenerics_0.59.12  generics_0.1.4        Matrix_1.7-5         
[16] colorout_1.3-3       

loaded via a namespace (and not attached):
[1] codetools_0.2-20    lattice_0.22-9      parallel_4.6.1     
[4] rhdf5filters_1.25.4 XVector_0.53.0      Rhdf5lib_2.1.0     
[7] grid_4.6.1          compiler_4.6.1      tools_4.6.1        

Possible solutions

• OS-level locks. Lock with the advisory file locks of the operating system (fcntl()/flock(), LockFileEx() on Windows), e.g., through the CRAN package filelock. They guarantee a single holder, are released when the holding process ends, even if it is killed, and support timeouts.
• An atomic, owned lock in base R, if a new dependency is not wanted. dir.create("<file>.lock") is atomic and fails if the directory exists. The holder can write its process id inside the lock directory and only remove a lock it owns, which makes the lock owner-aware.
• Fewer shared locks.
  • Dump names could be unique per process, e.g., including the process id together with a counter of that process, instead of a single global counter.
  • The dump log could be optional, or be written per process.
  • Dataset creation could skip both the counter and the log when the caller gives the file path and dataset name explicitly. That would also allow packages to create sinks from parallel workers safely, such as HDF5RealizationSink(filepath=, name=) and TENxRealizationSink(filepath=, group=).
• Defensive checks, whatever the locking. lock_file() could give up after a timeout with an informative error, instead of waiting forever. Code holding a lock could append to the locked file only if it exists, so a lost lock raises an error instead of creating a new file.

In GSVA, we worked around the problem by writing the HDF5 files of parallel workers with rhdf5, without going through these locks.