How to control queues and concurrency in Sidekiq?

Sidekiq is a great tool, but without proper configuration, it can cause a mess. As your application grows, so does the number of background tasks, and it's easy to end up in a situation where resource-intensive processes clog up those that need to be done right away. I'll show you how to take control of this chaos using Sidekiq capsules to precisely manage concurrency and queues.

Problem: different tasks, same resources

Imagine that your application sends emails, processes photos, and talks to an external API that has a query limit. By default, Sidekiq treats all these tasks equally. It puts them all in one bag and processes them at the same speed. What is the result? A long photo processing task can block the entire queue, and hundreds of tasks trying to query the API at once will quickly exhaust the limit and cause errors.

We want to avoid this. Ideally, API-related tasks should go in small groups (e.g., two at a time), and the rest, such as sending emails, should run at full speed. This is where capsules come in handy.

Solution: Sidekiq capsules in practice

Capsules are a Sidekiq feature that allows you to create separate thread pools with their own concurrency settings for specific queues. It's a bit like opening smaller, specialized processing centers inside Sidekiq.

Let's see how it works with a specific example. We want to limit the number of concurrent requests to an external API.

Step 1: configuring the capsule in Sidekiq

First, we need to define our capsule in the config/initializers/sidekiq.rb file. We add a separate section there for tasks that need restrictions.

Sidekiq.configure_server do |config|
  # We set the standard queues and general concurrency.
  config.queues = %w[default mailers]
  config.concurrency = 10

  # And here we create our capsule for jobs that require limiting.
  # We've named it 'api_calls'.
  config.capsule('api_calls') do |cap|
    # In this capsule, jobs will be processed with a maximum concurrency of 2.
    cap.concurrency = 2
    # We assign the 'api_calls' queue to it.
    cap.queues = %w[api_calls]
  end
end

What exactly happened here? We told Sidekiq: "Handle all jobs from the external_api queue using the api_calls capsule, which can run a maximum of two jobs at a time." At the same time, jobs from the default and mailers queues will run independently, with the overall concurrency set to 10.

Step 2: Create a task assigned to the capsule

Now we need to write a task (job) that will use our new, limited queue. We create a simple ExternalApiJob class and use sidekiq_options to assign it to the external_api queue.

class ExternalApiJob
  include Sidekiq::Job
  # This line tells Sidekiq: "Put all instances of this job in the 'external_api' queue".
  sidekiq_options queue: 'api_calls'

  def perform(resource_id)
    resource = Resource.find(resource_id)
    # Here we execute the logic, e.g. call a service to communicate with the API.
    # Thanks to the capsule, Sidekiq will ensure that no more than 2 such operations run at the same time.
    ExternalApiService.call(resource)
  end
end

Step 3: Calling the task

The last step is a piece of cake. All you need to do is call the task asynchronously anywhere in your application, for example in a controller or in another task.

ExternalApiJob.perform_async(123)

Once you do that, Sidekiq will put the job in the external_api queue, and the api_calls capsule will make sure it gets processed according to the set limit.

Summary

Capsules give you precise control over how Sidekiq processes background tasks. This simple but very effective solution allows you to:

• protect external APIs from being flooded with requests,
• isolate resource-intensive tasks (e.g., PDF generation, video processing) so they don't block urgent operations,
• prioritize tasks by allocating more resources to more important queues.

Instead of letting Sidekiq run blindly, you can consciously manage it, which translates into a more stable and predictable application. This is one of those small configuration changes that can save you a lot of headaches in the future.

Happy capsuling!