Zstd.decompress holds the GVL, blocking other threads for the whole call #155

Summary

I've been using zstd-ruby to decompress security-advisory archives inside Sidekiq workers, which run several threads per process. Zstd.decompress on a large payload blocks every other thread in the process for the full duration of the call. It turns out the one-shot decompress path holds Ruby's GVL (Global VM Lock) throughout the libzstd call, whereas the rest of the gem — compression, streaming decompress, streaming compress — releases it.

Root cause

Zstd.decompress runs its decode loop in the C function decode_one_frame, which calls ZSTD_decompressStream directly and so keeps the GVL for the whole operation. By contrast, Zstd.compress and both streaming classes route their libzstd calls through the GVL-releasing wrappers in common.h (zstd_compress, zstd_stream_compress, zstd_stream_decompress). One-shot decompress is the only hot path that does not — the streaming code already does the right thing.

Reproduction

A heartbeat thread records a timestamp every ~0.5 ms. The main thread then runs one large Zstd.decompress, and afterwards we count how many heartbeat samples landed inside the decompress window. If the GVL is held throughout, the heartbeat is starved and no samples land.

require 'zstd-ruby'

def mono
  Process.clock_gettime(Process::CLOCK_MONOTONIC)
end

size = (ENV['MB'] || '2048').to_i * 1024 * 1024
payload = Zstd.compress("\0" * size)   # ~64 KB compressed, 2 GiB decompressed

stop = false
stamps = []
hb = Thread.new do
  until stop
    stamps << mono
    sleep 0.0005
  end
end

sleep 0.2
t0 = mono
Zstd.decompress(payload)
t1 = mono
stop = true
hb.join

inside = stamps.select { |s| s >= t0 && s <= t1 }
bounds = [t0, *inside, t1]
max_gap = bounds.each_cons(2).map { |a, b| b - a }.max
printf "decompress_time         = %.3f s\n", t1 - t0
printf "heartbeat samples in run = %d\n", inside.size
printf "largest heartbeat stall = %.3f s\n", max_gap

Environment: zstd-ruby at main (39894ff), Ruby 3.3.11, macOS arm64.

Results

Before (current main):

• decompress_time = 0.467 s
• heartbeat samples in run = 0
• largest heartbeat stall = 0.467 s — the whole call; the other thread was frozen start to finish.

After (with the one-line fix below):

• decompress_time = 0.46–0.52 s (unchanged)
• heartbeat samples in run ≈ 700
• largest heartbeat stall ≤ 0.016 s

Proposed fix

One line in decode_one_frame (ext/zstdruby/zstdruby.c): call the existing zstd_stream_decompress wrapper instead of ZSTD_decompressStream directly, so the GVL is released once per output chunk. There's no measurable throughput cost. I'll open a PR with it.