Repository navigation
Optimizing for used Connections #627
Description
Hi,
first of all, thanks for Solid Queue, we're currently using it in production and really like its simplicity and Rails-native design!
One challenge we've run into: when running IO-heavy jobs concurrently, it creates big connection spikes.
For example, with:
-
20 Solid Queue threads per worker
-
3 worker processes
On Aurora Serverless, this leads to CPU spikes, scaling churn, and connection storms, even though the jobs are barely touching the database.
We switched from sidekiq, to profit from the security solidqueue offers and from the simplicity.
But this connection storm issue is a problem for us, because we want to be able to perform many IO-heavy jobs like mailers or document generation at once, without causing CPU intensive connection spikes.
With sidekiq, we were able to handle these workloads without issue.
Sidekiq Threads share one connection pool: https://github.com/sidekiq/sidekiq/wiki/Scaling-Sidekiq/00d09b1b8a8ae5b4c6c10ccde16bf0552bafa666 and do not need a connection per thread.
Is any optimization in that direction planned / possible?
Thanks and best regards
Pauline
Activity
c0mrade commentedon Sep 3, 2025
Try RDS Proxy (or PgBouncer) to prevent connection storms with many worker threads
pauline-koch commentedon Sep 4, 2025
Thank you, RDS-Proxy would alliviate our issue, but is unfortunately not compatible with our setup. We have to fullfill special authentication processes that incompatible with the proxy.
rosa commentedon Feb 19, 2026
Hey @pauline-koch, thanks for reporting this and for your kind words. Solid Queue relies on Active Record and Rails's app executor for DB connection management, so there's no planned improvement there.
I think you could try reducing the thread count per worker. 20 is quite a big number.
Are you using a separate DB for Solid Queue?
pauline-koch commentedon Feb 25, 2026
Hi @rosa, thanks for looking into this. Yes we are using a separate DB. And you are correct that 20 x 3 threads is quite a lot for most apps. It's a configuration that worked okay for us under sidekiq, but I have already started reducing the number of threads and that currently works for us.
crmne commentedon Apr 4, 2026
FWIW, I opened #728, which adds async worker execution mode.
For cooperative I/O-bound jobs, this should help reduce queue DB pool pressure versus pushing thread counts higher, especially on Rails 7.2+.
rosa commentedon Aug 29, 2026
Hey @pauline-koch, closing the loop on this one, as two things have landed since that address it directly:
fibers: Ninstead ofthreads: N, see configuration), which run many in-flight jobs on a single reactor thread. On Rails 7.2+ a worker process typically needs only 3–5 queue DB connections regardless of how many jobs are in flight — exactly your case (mailers, document generation).Thanks for raising it! Closing now.