Ruby: Resque Jobs and Jitter with `resque-scheduler`

Resque is a background job processor for Ruby. Sometimes you need to do something that’ll take a long time and you don’t want that happening as part of the HTTP request lifecycle. It helps you do that.

But what happens when you want to do LOTS of things at once, you want to avoid the thundering herd problem by spreading that work out rather than doing it all at once.

Jitter is one way to do this, taking an example using Resque, say you want to recalculate something for products and you have 10,000 products. Here we can define a resque job to do the recalculation, taking in the product ID and then queue a job for each but we don’t want them to all start at the same time due to the load this would put on the DB. To fix that we add jitter, saying start these jobs somewhere between now and 120seconds time, picking a random duration per job.

To do this for Resque we can use resque-scheduler, which lets us queue work for the future, along with before_enqueue resque hook to let a job declare its jitter with, for example, @jitter_milliseconds = 10 on the job class.

Here is an example of this all wired up (note: We ended up using a slightly different code for the prod implementation due to requirements on our side, be sure to test the below works for you before adopting).