Sidekiq basic_fetch and super_fetch differences

Sidekiq is a popular background job processing library for Ruby with three versions: OSS, Pro, and Enterprise. While OSS is free and open source, Pro and Enterprise are closed-source and paid versions with more advanced features. However, Pro version 3.4.0 introduced a fetch strategy called super_fetch, which aims to provide a more reliable and efficient way to fetch jobs from the queue in Redis.

The basic_fetch strategy that comes with the open source version of Sidekiq uses the BRPOP Redis command to fetch a scheduled job from the queue. If the Sidekiq process crashes while processing a fetched job, the job is lost forever. super_fetch, on the other hand, uses the RPOPLPUSH Redis command to fetch a job, providing a unique approach to implementing a reliable queue.

super_fetch registers a private queue in Redis for each Sidekiq process at startup and fetches a scheduled job from the public queue in Redis. It then atomically pushes the job into the private queue using the RPOPLPUSH Redis command. When the job is finished, Sidekiq removes the job from the private queue. During a graceful shutdown, Sidekiq moves the unfinished jobs from the private queue back to the public queue. If a Sidekiq process crashes, the unfinished jobs of that process remain in the private queue, called orphaned jobs. When the sidekiq process is restarted or another sidekiq process is started, super_fetch looks for such orphaned jobs in the private queues and re-queues them for processing.
The advantages of using super_fetch are that it ensures that no jobs are lost even if the Sidekiq process crashes, provides a reliable way to recover orphaned jobs, and provides a more efficient approach to fetching jobs from the queue. However, super_fetch requires the purchase of the Sidekiq Pro gem, and its exact implementation details are not publicly available due to its closed-source nature. Also, it may take some time to recover orphaned jobs if we don't restart or start another sidekiq process.

To use this thing, all we need to do is add one line to our configuration.

Sidekiq.configure_client do |config|
  config.redis = { url: ENV['REDIS_URL'], namespace: 'myapp_sidekiq' }
  config.super_fetch! # Use super fetch
end

In summary, if job reliability and recovery are paramount, super_fetch is the recommended option. However, if job loss is not a significant concern, the basic_fetch strategy that comes with the open-source version of Sidekiq may be sufficient.