Sidekiq + Parallel performance issues

[joaopmrod]

I've been using Sidekiq + Parallel (https://github.com/grosser/parallel) together for a couple of years. Everything worked fine in in_processes paralellization. I would get basically the same performance when I ran Job.perform_now in the console and Job.perform_later.

This year I've upgraded my application to Ruby 3.0.2p107, Rails 6.1.6, Sidekiq 6.4.2 and Parallel 1.22 and I now get 1/10-20 the performance when the job runs in Sidekiq with perform_later.

class TestJob < ApplicationJob
  def perform
    Parallel.map(Model.all, in_processes: 5, progress: "Map") do |obj|
      heavy_task(obj)
    end
  end
end

The last time I could get an equal performance I was using Ruby 2.7.1, Rails 6.1.4,Sidekiq 6.2.1 and Parallel 1.20.

Extra info: peform_now in the console has the same performance in the new and old setup.

From my debug I think the 5 processes are running but I doubt they are running in parallel. I see that thing get slower and slower over the execution time.

Any ideia about what can be causing that?

[grosser]

can you try with the old parallel + sidekiq versions and then update 1 at a time to see when it breaks?