Repository navigation
Description
What's the optimum config for Sidekiq on Heroku with Puma?
There are quite a number of answers on the Internet, but nothing definitive, and most of them come with vague numbers and suggestions or are outdated.
Basically, these are the questions that are often asked:
- What do I exactly put in the
config/initializers/sidekiq.rbfile? - What should I set for client/server
size? - What should I set for client/server
concurrency? - How does the Puma workers and threads affect the Sidekiq settings?
- How does the number of Redis connections affect the Sidekiq settings?
- How does that number of web/worker dynos affect the Sidekiq settings?
The best (and updated) answers I can find are:
With @bryanrite's post as a reference, this is our Sidekiq config:
config/initializers/sidekiq.rb
require 'sidekiq_calculations'
Sidekiq.configure_client do |config|
sidekiq_calculations = SidekiqCalculations.new
sidekiq_calculations.raise_error_for_env!
config.redis = {
url: ENV['REDISCLOUD_URL'],
size: sidekiq_calculations.client_redis_size
}
end
Sidekiq.configure_server do |config|
sidekiq_calculations = SidekiqCalculations.new
sidekiq_calculations.raise_error_for_env!
config.options[:concurrency] = sidekiq_calculations.server_concurrency_size
config.redis = {
url: ENV['REDISCLOUD_URL']
}
endlib/sidekiq_calculations.rb
class SidekiqCalculations
DEFAULT_CLIENT_REDIS_SIZE = 2
DEFAULT_SERVER_CONCURRENCY = 25
def raise_error_for_env!
return if !Rails.env.production?
web_dynos
worker_dynos
max_redis_connection
rescue KeyError, TypeError # Integer(nil) raises TypeError
raise <<-ERROR
Sidekiq Server Configuration failed.
!!!======> Please add ENV:
- NUMBER_OF_WEB_DYNOS
- NUMBER_OF_WORKER_DYNOS
- MAX_REDIS_CONNECTION
ERROR
end
def client_redis_size
return DEFAULT_CLIENT_REDIS_SIZE if !Rails.env.production?
puma_workers * (puma_threads/2) * web_dynos
end
def server_concurrency_size
return DEFAULT_SERVER_CONCURRENCY if !Rails.env.production?
(max_redis_connection - client_redis_size - sidekiq_reserved) / worker_dynos / paranoid_divisor
end
private
def web_dynos
Integer(ENV.fetch('NUMBER_OF_WEB_DYNOS'))
end
def worker_dynos
Integer(ENV.fetch('NUMBER_OF_WORKER_DYNOS'))
end
def max_redis_connection
Integer(ENV.fetch('MAX_REDIS_CONNECTION'))
end
# ENV used in `config/puma.rb` too.
def puma_workers
Integer(ENV.fetch("WEB_CONCURRENCY", 2))
end
# ENV used in `config/puma.rb` too.
def puma_threads
Integer(ENV.fetch("WEB_MAX_THREADS", 5))
end
# https://github.com/mperham/sidekiq/blob/master/lib/sidekiq/redis_connection.rb#L12
def sidekiq_reserved
5
end
# This is added to bring down the value of Concurrency
# so that there's leeway to grow
def paranoid_divisor
2
end
endThe sidekiq_calculations.rb file is dependent on a number of ENV variables to work, so if you do scale your app (web or workers), do remember to update these ENVs:
MAX_REDIS_CONNECTIONNUMBER_OF_WEB_DYNOSNUMBER_OF_WORKER_DYNOS
At the same time, WEB_CONCURRENCY and WEB_MAX_THREADS should be the identical ENV variables used to set the number of Puma workers and threads in config/initializers/puma.rb.
Our puma.rb looks exactly like what Heroku has proposed.
The only difference to @bryanrite's calculation is that Sidekiq reserves 5 connections instead of 2 now
according to this line, and I have also added a paranoid_divisor to bring down the concurrency number and keep it below a 80% threshold.
Let me know how this config works for you. Would love to hear your feedback!
Thank you for reading.
About Jolly Good Code
We specialise in Agile practices and Ruby, and we love contributing to open source.
Speak to us about your next big idea, or check out our projects.

Activity
winston commentedon Dec 8, 2015
@shinnyx @zamakkat @dtthaison @joshteng FYI.
joshteng commentedon Dec 8, 2015
@winston thx for sharing. it's helpful!
shinnc commentedon Dec 8, 2015
@winston thanks for the update 👍
heythisisnate commentedon Dec 16, 2015
Hey @winston!
I thought I'd share our current configuration at Hired. We have pretty much the same stack! Heroku running Sidekiq (Pro) and Puma for the web server. I don't do any of the fancy dynamic sizing calculations that you do, we control everything by environment variables.
Here's our simplified Procfile:
Puma config:
Sidekiq config:
We have found that Heroku's performance dynos are phenomenally more performant than the standard ones, and they come with tons of RAM so we can fit many copies of the app in memory. This allows me to use Puma's cluster mode and currently we run 12 Puma processes (PUMA_WORKERS) per dyno, each Puma using up to 3 threads (PUMA_MAX_THREADS). I probably could increase the workers significantly and still have enough memory. For Hired production we run 2 Performance-L dynos, and an additional 1 Performance-L for admins-only (our internal admin app).
For Sidekiq, I use cheaper Standard-2X dynos and have a Hubot script that auto scales-them based on queue depth. Currently I have SIDEKIQ_CONCURRENCY=5 and we always run at least 2 or 3 worker dynos and up to 12 at peak times.
We also run a single Clock dyno for scheduled jobs. Mostly these jobs just kick off other jobs or log stuff, so it's on a Standard-1X plan.
I hope this helps!
winston commentedon Dec 17, 2015
Hey @heythisisnate, this is awesome! Thanks for sharing! Good to know what hired is using.
I haven't had a chance to use
Performancedynos yet because none of the apps I managed have reached that magnitude, but 2X have been a staple for us for quite a while now because we have noticed that the memory footprint of Ruby apps have grown quite a bit, and it hits 512MB limit (1X) pretty easily. For 2X (1GB) memory, we only turn up about 1-3PUMA_WORKERSdepending on the app, and sometimes there would be one or two gnarly memory leaks that we have to hunt down. Most of the time, we PUMA_MAX_THREADS as 5 though. Extrapolating that to 14GB Performance-L servers, numbers seem about right (our puma config is very much the same as yours).RE: sidekiq, most of what I do is through ENVs as well, except that
client_redis_sizeandserver_concurrency_sizeare derived as formulas.I noticed that you didn't set
sizeunderSidekiq.configure_clientthough and from this article it seems to suggest that the number can/should be tweaked (otherwise I think it defaults to 25? - might be wasteful? or am I wrong?).For
Sidekiq.configure_server, we set the size via the formula here, instead of through the ENV variable (${SIDEKIQ_CONCURRENCY:-5}) so that I don't have to do the math every time I change certain values. Haha.For
ENV['DATABASE_URL'] = "#{database_url}?pool=250", if you are already usingdatabase.yml, I think thepool: 250can go into database.yml` too.Thoughts? Thanks for your reply! 🙇
longkt90 commentedon Dec 22, 2015
👍
longkt90 commentedon Dec 22, 2015
hi @winston, dividing by 2 in:
puma_workers * (puma_threads / 2) * web_dynoscould causeclient_redis_size = 0whenpuma_threads = 1longkt90 commentedon Dec 22, 2015
Btw, I am in use of redis-objects which need at least one connection per app instance (web). Is it better to share the connection pool between sidekiq and redis-objects or I should use another connection pool with size equal to
client_redis_size?winston commentedon Dec 25, 2015
@longkt90 Thanks for the feedback!
That's true. Probably should modify the
puma_threadsmethod to be:So that the minimum is at least 2.
winston commentedon Dec 25, 2015
@longkt90 I haven't used redis-objects myself, but I would think this might be better?
longkt90 commentedon Dec 25, 2015
Yeah. That's what we are using. I dont think it's good idea to share the
pool between sidekiq and redis-object
On Fri, Dec 25, 2015 at 11:57 AM Winston notifications@github.com wrote:
longkt90 commentedon Dec 25, 2015
Btw I need to set our db_pool to be redis server concurrency +
puma-max-threads. Is that correct?
On Fri, Dec 25, 2015 at 12:07 PM Nguyễn Thanh Long longkt90@gmail.com
wrote:
winston commentedon Dec 25, 2015
Yes that's right. I just set it as the max number of connection that my DB allows.
longkt90 commentedon Dec 25, 2015
👍 Thank you
nitzanav commentedon Oct 29, 2016
It seems to me that this code:
And this code seems to have an issue:
It should just be:
puma_workersandweb_dynosare not relevant for the connection pool the pool is shared only within the process and puma workers and dynos are separate process to each other.The
server_concurrency_sizeis OK because if you ran bellowCONCURRENCY + 2sidekiq will fail to start, and will not heart much.However after changing
client_redis_sizeyou do need to modifyserver_concurrency_sizeto usepuma_workersandweb_dynos.You probably didn't get any error because you just reserved bigger pool size to client than you really needed.
winston commentedon Nov 6, 2016
@nitzanav The code was transcribed following the explanations detailed in http://bryanrite.com/heroku-puma-redis-sidekiq-and-connection-limits/.
Sidekiq config is often a mystery (to me) and sometimes difficult to know what's exactly "right". The config above has at least worked in all the apps I deployed so far. But do let me know your mileage on the updated config. I am sure it will be a good data point too. Thanks!
nitzanav commentedon Nov 6, 2016
@winston This blog indeed shows this formula, but the usage of it as is, is meant in order to be bale to infer the maximum connections to be expected and configured on the redis server side, rather than the ruby application side. The ruby application side is a bit different, and AFAIK should be used as I described.
What you did will work but is no optimum :)
arvindvyas commentedon Nov 20, 2017
@winston at the time of auto scale how we will update - NUMBER_OF_WEB_DYNOS
can anyone tell me please
jcohenho commentedon Sep 13, 2018
@winston I realize this is a couple years old now but I'm only now learning about and using Puma and concurrency so this may still be relevant to anyone else. To help clarify what @nitzanav is saying, I think you're misinterpreting what the blog post is saying in regards to setting the Redis client size. In fact, in the blog Bryan Rite has the following code for setting the size:
These sidekiq settings will configure the size per worker process, so you don't need to factor in the
puma_workers *or* web_dynosoperations. Thepuma_workers * (puma_threads/2) * web_dynosformula is just telling you what the expected total number of connections will be if your app dynos were to fully utilize each worker and thread.Hope this helps!