diff --git a/README.md b/README.md index 03715a1..fb00f3f 100644 --- a/README.md +++ b/README.md @@ -391,14 +391,16 @@ QUEUE=tracking rake delayed:monitor QUEUES=mailers,tasks rake delayed:monitor ``` -The following events will be emitted, grouped by priority name (e.g. "interactive") and queue name, -and the metric's "`:value`" will be available in the event's payload. **This means that there will -be one value _per_ unique combination of queue & priority**, and totals must be computed via -downstream aggregation (e.g. as a StatsD "gauge" metric). +The following events will be emitted, grouped by priority name (e.g. "interactive"), queue name, +and the values of any configured `tag_columns`. By default, job `name` is included. The +metric's "`:value`" will be available in the event's payload. **This means that there will be one +value _per_ unique combination of queue, priority, and tag column values**, and totals must be +computed via downstream aggregation (e.g. as a StatsD "gauge" metric, summed or maxed by tag). - **delayed.job.count** - the total number of jobs - **delayed.job.future_count** - jobs where run_at is in the future - **delayed.job.working_count** - jobs that are currently being worked off (excludes failed jobs) +- **delayed.job.locked_count** - jobs that are currently locked by a worker (equivalent to working_count) - **delayed.job.workable_count** - jobs that are waiting to be worked off - **delayed.job.erroring_count** - jobs where attempts > 0 - **delayed.job.failed_count** - jobs where failed_at is not nil @@ -409,8 +411,66 @@ An additional _experimental_ metric is available, intended for use with applicat - **delayed.job.alert_age_percent** - the _percent_ to which the oldest job has reached the "age alert" threshold. (See the [Alerting Threshholds](#priority-based-alerting-threshholds) section above.) -All of these events may be subscribed to via a single regular expression (again, in your application -config or in an initializer): +By default, these events are also tagged with the job's `name` (when the jobs table has a `name` +column — see [Database Setup](#database-setup)) so that downstream aggregation can answer +"_which_ job is stuck?" when `delayed.job.max_age` alerts (e.g. `max by {queue, priority, name}` +in Datadog). + +The set of tagged columns is driven by `Delayed::Monitor.tag_columns`, which defaults to +`%i(name)` when the jobs table has a `name` column (and to `[]` otherwise). You can include +columns your application adds to the jobs table (populated at enqueue time). For example, if your +jobs table has an `owner` column you wish to also monitor: + +```ruby +Delayed::Monitor.tag_columns = %i(name owner) +``` + +A few behavioral notes: + +- Rows whose value was never populated for a tagged column are reported under the value `'unset'` + (e.g. jobs enqueued before the `name` column existed, mid-upgrade). +- Configured columns must exist on the jobs table: assigning a missing column to `tag_columns` + raises an `ArgumentError` immediately, rather than the column being silently skipped. Because + the assignment validates against the schema, setting `tag_columns` in an initializer requires a + database connection at boot, in every process that loads it. See the rollout steps below. +- Tag values cannot be enumerated in advance, so a tagged series is only emitted while matching + jobs are present. Separately, an untagged zero value is always emitted for every + (priority, queue) combination, so that each metric maintains a baseline series even when no + matching jobs are enqueued. For example, `delayed.job.count` with a single enqueued job would + emit the following series: + + ```ruby + { priority: 'interactive', queue: 'default', name: 'SimpleJob', value: 1 } + { priority: 'interactive', queue: 'default', value: 0 } + { priority: 'user_visible', queue: 'default', value: 0 } + { priority: 'eventual', queue: 'default', value: 0 } + { priority: 'reporting', queue: 'default', value: 0 } + ``` +- Each column multiplies a metric's series cardinality by its number of distinct values (though in + practice a job's `name` tends to determine its `priority` and any ownership tags, making the + number of distinct job names the effective upper bound). If cardinality is a concern for your + metrics provider, tagging can be disabled entirely with `Delayed::Monitor.tag_columns = []`. + +#### Rolling out a new tag column + +Because assignment fails loudly on a missing column, a new tag column should be rolled out in +three separate deploys, each fully released before the next begins: + +1. Migrate the column onto the jobs table (nullable — no backfill required). +2. Deploy the code that populates the column at enqueue time. +3. Add the column to `Delayed::Monitor.tag_columns` in an initializer, and deploy. + +Adding the column to `tag_columns` before the migration has run everywhere would raise at boot in +every process that loads the initializer. Jobs enqueued before step 2 will report under the +`'unset'` tag value until they are worked off (or backfilled). + +The default `name` tag needs no such rollout: it applies only when the jobs table already has a +`name` column, so a monitor running against an older schema simply emits untagged metrics until +the generated migrations (see [Database Setup](#database-setup)) have run and the monitor process +has restarted (the default is resolved once per process). + +All of these events may be subscribed to via a single regular expression (again, in your +application config or in an initializer): ```ruby ActiveSupport::Notifications.subscribe(/delayed\.job\..*_(count|age|percent)/) do |*args| @@ -423,7 +483,7 @@ ActiveSupport::Notifications.subscribe(/delayed\.job\..*_(count|age|percent)/) d end ``` -Additionally, the monitor process with emit a **delayed.monitor.run** event with a duration +Additionally, the monitor process will emit a **delayed.monitor.run** event with a duration attached, so that you can monitor the time it takes to emit these aggregate metrics. ```ruby diff --git a/lib/delayed/monitor.rb b/lib/delayed/monitor.rb index c2871cb..d849063 100644 --- a/lib/delayed/monitor.rb +++ b/lib/delayed/monitor.rb @@ -17,6 +17,19 @@ class Monitor cattr_accessor :sleep_delay, instance_writer: false, default: 60 + def self.tag_columns + @tag_columns ||= (Job.column_names.include?('name') ? %i(name) : []).freeze + end + + def self.tag_columns=(columns) + if columns.any? { |column| Job.column_names.exclude?(column.to_s) } + raise ArgumentError, "Delayed::Monitor.tag_columns includes columns missing from #{Job.table_name}. " \ + "Available columns: #{Job.column_names.join(', ')}" + end + + @tag_columns = columns.map(&:to_sym).freeze + end + def initialize @jobs = Job.group(:priority, :queue) @jobs = @jobs.where(queue: Worker.queues) if Worker.queues.any? @@ -67,10 +80,13 @@ def self.parse_utc_time(string) attr_reader :jobs def emit_metric!(metric) - query_for(metric).reverse_merge(default_results).each do |(priority, queue), value| + query_for(metric) + .merge!(default_results) { |_key, existing, _default| existing } + .each do |(priority, queue, *column_values), value| + tags = column_values.zip(self.class.tag_columns).to_h { |val, column| [column, val.nil? ? 'unset' : val] } ActiveSupport::Notifications.instrument( "delayed.job.#{metric}", - default_tags.merge(priority: Priority.new(priority).to_s, queue: queue, value: value), + default_tags.merge(priority: Priority.new(priority).to_s, queue: queue, **tags, value: value), ) end end @@ -96,24 +112,30 @@ def default_tags end # This method generates a query that scans the specified scope, groups by - # priority and queue, and calculates the specified aggregates. An outer - # query is executed for priority bucketing and appending db_now_utc (to - # avoid running these computations for each tuple in the inner query). - def grouped_query(scope, include_db_time: false, **kwargs) + # priority and queue (plus any extra_group_columns), and calculates the + # specified aggregates. An outer query is executed for priority bucketing + # and appending db_now_utc (to avoid running these computations for each + # tuple in the inner query). + def grouped_query(scope, include_db_time: false, extra_group_columns: [], **kwargs) inner_selects = kwargs.map { |key, (agg, expr)| as_expression(agg, expr, key) } outer_selects = kwargs.map { |key, (agg, _)| as_expression(agg == :count ? :sum : agg, key, key) } outer_selects << "#{self.class.sql_now_in_utc} AS db_now_utc" if include_db_time Delayed::Job - .from(scope.select(:priority, :queue, *inner_selects).group(:priority, :queue)) - .group(priority_case_statement, :queue).select( + .from(scope.select(:priority, :queue, *extra_group_columns, *inner_selects).group(:priority, :queue, *extra_group_columns)) + .group(priority_case_statement, :queue, *extra_group_columns).select( *outer_selects, "#{priority_case_statement} AS priority", 'queue AS queue', - ).group_by { |j| [j.priority.to_i, j.queue] } + *extra_group_columns.map { |column| "#{column} AS #{column}" }, + ).group_by { |j| result_key(j, extra_group_columns) } .transform_values(&:first) end + def result_key(record, extra_group_columns) + [record.priority.to_i, record.queue, *extra_group_columns.map { |column| record[column] }] + end + def as_expression(aggregate_function, aggregate_expression, column_name) "#{aggregate_function.to_s.upcase}(#{aggregate_expression}) AS #{column_name}" end @@ -151,10 +173,10 @@ def max_age_grouped end def alert_age_percent_grouped - pending_counts.each_with_object({}) do |((priority, queue), j), metrics| + pending_counts.each_with_object({}) do |(key, j), metrics| max_age = time_ago(db_now(j), j.run_at) - alert_age = Priority.new(priority).alert_age - metrics[[priority, queue]] = [max_age / alert_age * 100, 100].min if alert_age + alert_age = Priority.new(key.first).alert_age + metrics[key] = [max_age / alert_age * 100, 100].min if alert_age end end @@ -175,6 +197,7 @@ def oldest_workable_job_grouped def live_counts @memo[:live_counts] ||= grouped_query( jobs.live, + extra_group_columns: self.class.tag_columns, count: [:count, '*'], future_count: [:sum, case_when(Job.future_clause.to_sql)], erroring_count: [:sum, case_when(Job.erroring_clause.to_sql)], @@ -185,6 +208,7 @@ def pending_counts @memo[:pending_counts] ||= grouped_query( jobs.pending, include_db_time: true, + extra_group_columns: self.class.tag_columns, claimed_count: [:sum, case_when(Job.claimed_clause.to_sql)], claimable_count: [:sum, case_when(Job.claimable_clause.to_sql)], locked_at: [:min, case_when(Job.claimed_clause.to_sql, 'locked_at')], @@ -193,7 +217,8 @@ def pending_counts end def failed_counts - @memo[:failed_counts] ||= grouped_query(jobs.failed, count: [:count, '*']) + @memo[:failed_counts] ||= + grouped_query(jobs.failed, extra_group_columns: self.class.tag_columns, count: [:count, '*']) end def db_now(record) diff --git a/spec/delayed/__snapshots__/monitor_spec.rb.snap b/spec/delayed/__snapshots__/monitor_spec.rb.snap index 8da11e2..b403da3 100644 --- a/spec/delayed/__snapshots__/monitor_spec.rb.snap +++ b/spec/delayed/__snapshots__/monitor_spec.rb.snap @@ -3,56 +3,61 @@ snapshots["runs the expected postgresql queries with the expected plans 1"] = << --------------------------------- SELECT SUM(count) AS count, CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END AS priority, - queue AS queue - FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", COUNT(*) AS count + queue AS queue, + name AS name + FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\", COUNT(*) AS count FROM \"delayed_jobs\" WHERE \"delayed_jobs\".\"failed_at\" IS NOT NULL - GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\") subquery - GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\" + GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\") subquery + GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\", \"name\" GroupAggregate (cost=...) - Output: sum(subquery.count), (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue - Group Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue + Output: sum(subquery.count), (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name + Group Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name -> Sort (cost=...) - Output: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.count - Sort Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue + Output: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name, subquery.count + Sort Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name -> Subquery Scan on subquery (cost=...) - Output: CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END, subquery.queue, subquery.count + Output: CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END, subquery.queue, subquery.name, subquery.count -> GroupAggregate (cost=...) - Output: delayed_jobs.priority, delayed_jobs.queue, count(*) - Group Key: delayed_jobs.priority, delayed_jobs.queue - -> Index Only Scan using idx_delayed_jobs_failed on public.delayed_jobs (cost=...) - Output: delayed_jobs.priority, delayed_jobs.queue + Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name, count(*) + Group Key: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name + -> Sort (cost=...) + Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name + Sort Key: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name + -> Index Scan using idx_delayed_jobs_failed on public.delayed_jobs (cost=...) + Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name --- SELECT SUM(count) AS count, SUM(future_count) AS future_count, SUM(erroring_count) AS erroring_count, CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END AS priority, - queue AS queue - FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", COUNT(*) AS count, + queue AS queue, + name AS name + FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\", COUNT(*) AS count, SUM(CASE WHEN \"delayed_jobs\".\"run_at\" > '2025-11-10 17:20:13' THEN 1 ELSE 0 END) AS future_count, SUM(CASE WHEN \"delayed_jobs\".\"attempts\" > 0 THEN 1 ELSE 0 END) AS erroring_count FROM \"delayed_jobs\" WHERE \"delayed_jobs\".\"failed_at\" IS NULL - GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\") subquery - GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\" + GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\") subquery + GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\", \"name\" GroupAggregate (cost=...) - Output: sum(subquery.count), sum(subquery.future_count), sum(subquery.erroring_count), (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue - Group Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue + Output: sum(subquery.count), sum(subquery.future_count), sum(subquery.erroring_count), (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name + Group Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name -> Sort (cost=...) - Output: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.count, subquery.future_count, subquery.erroring_count - Sort Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue + Output: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name, subquery.count, subquery.future_count, subquery.erroring_count + Sort Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name -> Subquery Scan on subquery (cost=...) - Output: CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END, subquery.queue, subquery.count, subquery.future_count, subquery.erroring_count + Output: CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END, subquery.queue, subquery.name, subquery.count, subquery.future_count, subquery.erroring_count -> GroupAggregate (cost=...) - Output: delayed_jobs.priority, delayed_jobs.queue, count(*), sum(CASE WHEN (delayed_jobs.run_at > '2025-11-10 17:20:13'::timestamp without time zone) THEN 1 ELSE 0 END), sum(CASE WHEN (delayed_jobs.attempts > 0) THEN 1 ELSE 0 END) - Group Key: delayed_jobs.priority, delayed_jobs.queue + Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name, count(*), sum(CASE WHEN (delayed_jobs.run_at > '2025-11-10 17:20:13'::timestamp without time zone) THEN 1 ELSE 0 END), sum(CASE WHEN (delayed_jobs.attempts > 0) THEN 1 ELSE 0 END) + Group Key: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name -> Sort (cost=...) - Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.run_at, delayed_jobs.attempts - Sort Key: delayed_jobs.priority, delayed_jobs.queue - -> Index Only Scan using idx_delayed_jobs_live on public.delayed_jobs (cost=...) - Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.run_at, delayed_jobs.attempts + Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name, delayed_jobs.run_at, delayed_jobs.attempts + Sort Key: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name + -> Index Scan using idx_delayed_jobs_live on public.delayed_jobs (cost=...) + Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name, delayed_jobs.run_at, delayed_jobs.attempts --- -- QUERIES FOR `future_count`: --------------------------------- @@ -65,8 +70,9 @@ SELECT SUM(claimed_count) AS claimed_count, MIN(run_at) AS run_at, TIMEZONE('UTC', STATEMENT_TIMESTAMP()) AS db_now_utc, CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END AS priority, - queue AS queue - FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", SUM(CASE WHEN \"delayed_jobs\".\"locked_at\" >= '2025-11-10 16:59:43' THEN 1 ELSE 0 END) AS claimed_count, + queue AS queue, + name AS name + FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\", SUM(CASE WHEN \"delayed_jobs\".\"locked_at\" >= '2025-11-10 16:59:43' THEN 1 ELSE 0 END) AS claimed_count, SUM(CASE WHEN (\"delayed_jobs\".\"locked_at\" IS NULL OR \"delayed_jobs\".\"locked_at\" < '2025-11-10 16:59:43') THEN 1 ELSE 0 END) AS claimable_count, MIN(CASE WHEN \"delayed_jobs\".\"locked_at\" >= '2025-11-10 16:59:43' THEN locked_at ELSE NULL END) AS locked_at, @@ -75,25 +81,25 @@ SELECT SUM(claimed_count) AS claimed_count, FROM \"delayed_jobs\" WHERE \"delayed_jobs\".\"failed_at\" IS NULL AND \"delayed_jobs\".\"run_at\" <= '2025-11-10 17:20:13' - GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\") subquery - GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\" + GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\") subquery + GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\", \"name\" GroupAggregate (cost=...) - Output: sum(subquery.claimed_count), sum(subquery.claimable_count), min(subquery.locked_at), min(subquery.run_at), timezone('UTC'::text, statement_timestamp()), (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue - Group Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue + Output: sum(subquery.claimed_count), sum(subquery.claimable_count), min(subquery.locked_at), min(subquery.run_at), timezone('UTC'::text, statement_timestamp()), (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name + Group Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name -> Sort (cost=...) - Output: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.claimed_count, subquery.claimable_count, subquery.locked_at, subquery.run_at - Sort Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue + Output: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name, subquery.claimed_count, subquery.claimable_count, subquery.locked_at, subquery.run_at + Sort Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name -> Subquery Scan on subquery (cost=...) - Output: CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END, subquery.queue, subquery.claimed_count, subquery.claimable_count, subquery.locked_at, subquery.run_at + Output: CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END, subquery.queue, subquery.name, subquery.claimed_count, subquery.claimable_count, subquery.locked_at, subquery.run_at -> GroupAggregate (cost=...) - Output: delayed_jobs.priority, delayed_jobs.queue, sum(CASE WHEN (delayed_jobs.locked_at >= '2025-11-10 16:59:43'::timestamp without time zone) THEN 1 ELSE 0 END), sum(CASE WHEN ((delayed_jobs.locked_at IS NULL) OR (delayed_jobs.locked_at < '2025-11-10 16:59:43'::timestamp without time zone)) THEN 1 ELSE 0 END), min(CASE WHEN (delayed_jobs.locked_at >= '2025-11-10 16:59:43'::timestamp without time zone) THEN delayed_jobs.locked_at ELSE NULL::timestamp without time zone END), min(CASE WHEN ((delayed_jobs.locked_at IS NULL) OR (delayed_jobs.locked_at < '2025-11-10 16:59:43'::timestamp without time zone)) THEN delayed_jobs.run_at ELSE NULL::timestamp without time zone END) - Group Key: delayed_jobs.priority, delayed_jobs.queue + Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name, sum(CASE WHEN (delayed_jobs.locked_at >= '2025-11-10 16:59:43'::timestamp without time zone) THEN 1 ELSE 0 END), sum(CASE WHEN ((delayed_jobs.locked_at IS NULL) OR (delayed_jobs.locked_at < '2025-11-10 16:59:43'::timestamp without time zone)) THEN 1 ELSE 0 END), min(CASE WHEN (delayed_jobs.locked_at >= '2025-11-10 16:59:43'::timestamp without time zone) THEN delayed_jobs.locked_at ELSE NULL::timestamp without time zone END), min(CASE WHEN ((delayed_jobs.locked_at IS NULL) OR (delayed_jobs.locked_at < '2025-11-10 16:59:43'::timestamp without time zone)) THEN delayed_jobs.run_at ELSE NULL::timestamp without time zone END) + Group Key: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name -> Sort (cost=...) - Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.locked_at, delayed_jobs.run_at - Sort Key: delayed_jobs.priority, delayed_jobs.queue + Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name, delayed_jobs.locked_at, delayed_jobs.run_at + Sort Key: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name -> Index Scan using idx_delayed_jobs_live on public.delayed_jobs (cost=...) - Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.locked_at, delayed_jobs.run_at + Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name, delayed_jobs.locked_at, delayed_jobs.run_at Index Cond: (delayed_jobs.run_at <= '2025-11-10 17:20:13'::timestamp without time zone) --- -- QUERIES FOR `erroring_count`: @@ -124,60 +130,62 @@ snapshots["[legacy index] runs the expected postgresql queries with the expected --------------------------------- SELECT SUM(count) AS count, CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END AS priority, - queue AS queue - FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", COUNT(*) AS count + queue AS queue, + name AS name + FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\", COUNT(*) AS count FROM \"delayed_jobs\" WHERE \"delayed_jobs\".\"failed_at\" IS NOT NULL - GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\") subquery - GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\" + GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\") subquery + GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\", \"name\" GroupAggregate (cost=...) - Output: sum(subquery.count), (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue - Group Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue + Output: sum(subquery.count), (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name + Group Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name -> Sort (cost=...) - Output: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.count - Sort Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue + Output: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name, subquery.count + Sort Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name -> Subquery Scan on subquery (cost=...) - Output: CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END, subquery.queue, subquery.count + Output: CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END, subquery.queue, subquery.name, subquery.count -> GroupAggregate (cost=...) - Output: delayed_jobs.priority, delayed_jobs.queue, count(*) - Group Key: delayed_jobs.priority, delayed_jobs.queue + Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name, count(*) + Group Key: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name -> Sort (cost=...) - Output: delayed_jobs.priority, delayed_jobs.queue - Sort Key: delayed_jobs.priority, delayed_jobs.queue + Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name + Sort Key: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name -> Index Scan using delayed_jobs_priority on public.delayed_jobs (cost=...) - Output: delayed_jobs.priority, delayed_jobs.queue + Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name Filter: (delayed_jobs.failed_at IS NOT NULL) --- SELECT SUM(count) AS count, SUM(future_count) AS future_count, SUM(erroring_count) AS erroring_count, CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END AS priority, - queue AS queue - FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", COUNT(*) AS count, + queue AS queue, + name AS name + FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\", COUNT(*) AS count, SUM(CASE WHEN \"delayed_jobs\".\"run_at\" > '2025-11-10 17:20:13' THEN 1 ELSE 0 END) AS future_count, SUM(CASE WHEN \"delayed_jobs\".\"attempts\" > 0 THEN 1 ELSE 0 END) AS erroring_count FROM \"delayed_jobs\" WHERE \"delayed_jobs\".\"failed_at\" IS NULL - GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\") subquery - GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\" + GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\") subquery + GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\", \"name\" GroupAggregate (cost=...) - Output: sum(subquery.count), sum(subquery.future_count), sum(subquery.erroring_count), (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue - Group Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue + Output: sum(subquery.count), sum(subquery.future_count), sum(subquery.erroring_count), (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name + Group Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name -> Sort (cost=...) - Output: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.count, subquery.future_count, subquery.erroring_count - Sort Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue + Output: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name, subquery.count, subquery.future_count, subquery.erroring_count + Sort Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name -> Subquery Scan on subquery (cost=...) - Output: CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END, subquery.queue, subquery.count, subquery.future_count, subquery.erroring_count + Output: CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END, subquery.queue, subquery.name, subquery.count, subquery.future_count, subquery.erroring_count -> GroupAggregate (cost=...) - Output: delayed_jobs.priority, delayed_jobs.queue, count(*), sum(CASE WHEN (delayed_jobs.run_at > '2025-11-10 17:20:13'::timestamp without time zone) THEN 1 ELSE 0 END), sum(CASE WHEN (delayed_jobs.attempts > 0) THEN 1 ELSE 0 END) - Group Key: delayed_jobs.priority, delayed_jobs.queue + Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name, count(*), sum(CASE WHEN (delayed_jobs.run_at > '2025-11-10 17:20:13'::timestamp without time zone) THEN 1 ELSE 0 END), sum(CASE WHEN (delayed_jobs.attempts > 0) THEN 1 ELSE 0 END) + Group Key: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name -> Sort (cost=...) - Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.run_at, delayed_jobs.attempts - Sort Key: delayed_jobs.priority, delayed_jobs.queue + Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name, delayed_jobs.run_at, delayed_jobs.attempts + Sort Key: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name -> Index Scan using delayed_jobs_priority on public.delayed_jobs (cost=...) - Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.run_at, delayed_jobs.attempts + Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name, delayed_jobs.run_at, delayed_jobs.attempts Filter: (delayed_jobs.failed_at IS NULL) --- -- QUERIES FOR `future_count`: @@ -191,8 +199,9 @@ SELECT SUM(claimed_count) AS claimed_count, MIN(run_at) AS run_at, TIMEZONE('UTC', STATEMENT_TIMESTAMP()) AS db_now_utc, CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END AS priority, - queue AS queue - FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", SUM(CASE WHEN \"delayed_jobs\".\"locked_at\" >= '2025-11-10 16:59:43' THEN 1 ELSE 0 END) AS claimed_count, + queue AS queue, + name AS name + FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\", SUM(CASE WHEN \"delayed_jobs\".\"locked_at\" >= '2025-11-10 16:59:43' THEN 1 ELSE 0 END) AS claimed_count, SUM(CASE WHEN (\"delayed_jobs\".\"locked_at\" IS NULL OR \"delayed_jobs\".\"locked_at\" < '2025-11-10 16:59:43') THEN 1 ELSE 0 END) AS claimable_count, MIN(CASE WHEN \"delayed_jobs\".\"locked_at\" >= '2025-11-10 16:59:43' THEN locked_at ELSE NULL END) AS locked_at, @@ -201,25 +210,25 @@ SELECT SUM(claimed_count) AS claimed_count, FROM \"delayed_jobs\" WHERE \"delayed_jobs\".\"failed_at\" IS NULL AND \"delayed_jobs\".\"run_at\" <= '2025-11-10 17:20:13' - GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\") subquery - GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\" + GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\") subquery + GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\", \"name\" GroupAggregate (cost=...) - Output: sum(subquery.claimed_count), sum(subquery.claimable_count), min(subquery.locked_at), min(subquery.run_at), timezone('UTC'::text, statement_timestamp()), (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue - Group Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue + Output: sum(subquery.claimed_count), sum(subquery.claimable_count), min(subquery.locked_at), min(subquery.run_at), timezone('UTC'::text, statement_timestamp()), (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name + Group Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name -> Sort (cost=...) - Output: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.claimed_count, subquery.claimable_count, subquery.locked_at, subquery.run_at - Sort Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue + Output: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name, subquery.claimed_count, subquery.claimable_count, subquery.locked_at, subquery.run_at + Sort Key: (CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END), subquery.queue, subquery.name -> Subquery Scan on subquery (cost=...) - Output: CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END, subquery.queue, subquery.claimed_count, subquery.claimable_count, subquery.locked_at, subquery.run_at + Output: CASE WHEN (subquery.priority < 10) THEN 0 WHEN (subquery.priority < 20) THEN 10 WHEN (subquery.priority < 30) THEN 20 WHEN (subquery.priority >= 30) THEN 30 ELSE NULL::integer END, subquery.queue, subquery.name, subquery.claimed_count, subquery.claimable_count, subquery.locked_at, subquery.run_at -> GroupAggregate (cost=...) - Output: delayed_jobs.priority, delayed_jobs.queue, sum(CASE WHEN (delayed_jobs.locked_at >= '2025-11-10 16:59:43'::timestamp without time zone) THEN 1 ELSE 0 END), sum(CASE WHEN ((delayed_jobs.locked_at IS NULL) OR (delayed_jobs.locked_at < '2025-11-10 16:59:43'::timestamp without time zone)) THEN 1 ELSE 0 END), min(CASE WHEN (delayed_jobs.locked_at >= '2025-11-10 16:59:43'::timestamp without time zone) THEN delayed_jobs.locked_at ELSE NULL::timestamp without time zone END), min(CASE WHEN ((delayed_jobs.locked_at IS NULL) OR (delayed_jobs.locked_at < '2025-11-10 16:59:43'::timestamp without time zone)) THEN delayed_jobs.run_at ELSE NULL::timestamp without time zone END) - Group Key: delayed_jobs.priority, delayed_jobs.queue + Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name, sum(CASE WHEN (delayed_jobs.locked_at >= '2025-11-10 16:59:43'::timestamp without time zone) THEN 1 ELSE 0 END), sum(CASE WHEN ((delayed_jobs.locked_at IS NULL) OR (delayed_jobs.locked_at < '2025-11-10 16:59:43'::timestamp without time zone)) THEN 1 ELSE 0 END), min(CASE WHEN (delayed_jobs.locked_at >= '2025-11-10 16:59:43'::timestamp without time zone) THEN delayed_jobs.locked_at ELSE NULL::timestamp without time zone END), min(CASE WHEN ((delayed_jobs.locked_at IS NULL) OR (delayed_jobs.locked_at < '2025-11-10 16:59:43'::timestamp without time zone)) THEN delayed_jobs.run_at ELSE NULL::timestamp without time zone END) + Group Key: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name -> Sort (cost=...) - Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.locked_at, delayed_jobs.run_at - Sort Key: delayed_jobs.priority, delayed_jobs.queue + Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name, delayed_jobs.locked_at, delayed_jobs.run_at + Sort Key: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name -> Index Scan using delayed_jobs_priority on public.delayed_jobs (cost=...) - Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.locked_at, delayed_jobs.run_at + Output: delayed_jobs.priority, delayed_jobs.queue, delayed_jobs.name, delayed_jobs.locked_at, delayed_jobs.run_at Index Cond: (delayed_jobs.run_at <= '2025-11-10 17:20:13'::timestamp without time zone) Filter: (delayed_jobs.failed_at IS NULL) --- @@ -251,15 +260,17 @@ snapshots["runs the expected sqlite3 queries with the expected plans 1"] = <<-SN --------------------------------- SELECT SUM(count) AS count, CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END AS priority, - queue AS queue - FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", COUNT(*) AS count + queue AS queue, + name AS name + FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\", COUNT(*) AS count FROM \"delayed_jobs\" WHERE \"delayed_jobs\".\"failed_at\" IS NOT NULL - GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\") subquery - GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\" + GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\") subquery + GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\", \"name\" CO-ROUTINE subquery SCAN delayed_jobs USING INDEX idx_delayed_jobs_failed +USE TEMP B-TREE FOR GROUP BY SCAN subquery USE TEMP B-TREE FOR GROUP BY --- @@ -267,14 +278,15 @@ SELECT SUM(count) AS count, SUM(future_count) AS future_count, SUM(erroring_count) AS erroring_count, CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END AS priority, - queue AS queue - FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", COUNT(*) AS count, + queue AS queue, + name AS name + FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\", COUNT(*) AS count, SUM(CASE WHEN \"delayed_jobs\".\"run_at\" > '2025-11-10 17:20:13' THEN 1 ELSE 0 END) AS future_count, SUM(CASE WHEN \"delayed_jobs\".\"attempts\" > 0 THEN 1 ELSE 0 END) AS erroring_count FROM \"delayed_jobs\" WHERE \"delayed_jobs\".\"failed_at\" IS NULL - GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\") subquery - GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\" + GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\") subquery + GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\", \"name\" CO-ROUTINE subquery SCAN delayed_jobs USING INDEX idx_delayed_jobs_live @@ -293,8 +305,9 @@ SELECT SUM(claimed_count) AS claimed_count, MIN(run_at) AS run_at, CURRENT_TIMESTAMP AS db_now_utc, CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END AS priority, - queue AS queue - FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", SUM(CASE WHEN \"delayed_jobs\".\"locked_at\" >= '2025-11-10 16:59:43' THEN 1 ELSE 0 END) AS claimed_count, + queue AS queue, + name AS name + FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\", SUM(CASE WHEN \"delayed_jobs\".\"locked_at\" >= '2025-11-10 16:59:43' THEN 1 ELSE 0 END) AS claimed_count, SUM(CASE WHEN (\"delayed_jobs\".\"locked_at\" IS NULL OR \"delayed_jobs\".\"locked_at\" < '2025-11-10 16:59:43') THEN 1 ELSE 0 END) AS claimable_count, MIN(CASE WHEN \"delayed_jobs\".\"locked_at\" >= '2025-11-10 16:59:43' THEN locked_at ELSE NULL END) AS locked_at, @@ -303,8 +316,8 @@ SELECT SUM(claimed_count) AS claimed_count, FROM \"delayed_jobs\" WHERE \"delayed_jobs\".\"failed_at\" IS NULL AND \"delayed_jobs\".\"run_at\" <= '2025-11-10 17:20:13' - GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\") subquery - GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\" + GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\") subquery + GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\", \"name\" CO-ROUTINE subquery SCAN delayed_jobs USING INDEX idx_delayed_jobs_live @@ -340,12 +353,13 @@ snapshots["[legacy index] runs the expected sqlite3 queries with the expected pl --------------------------------- SELECT SUM(count) AS count, CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END AS priority, - queue AS queue - FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", COUNT(*) AS count + queue AS queue, + name AS name + FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\", COUNT(*) AS count FROM \"delayed_jobs\" WHERE \"delayed_jobs\".\"failed_at\" IS NOT NULL - GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\") subquery - GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\" + GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\") subquery + GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\", \"name\" CO-ROUTINE subquery SCAN delayed_jobs USING INDEX delayed_jobs_priority @@ -357,14 +371,15 @@ SELECT SUM(count) AS count, SUM(future_count) AS future_count, SUM(erroring_count) AS erroring_count, CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END AS priority, - queue AS queue - FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", COUNT(*) AS count, + queue AS queue, + name AS name + FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\", COUNT(*) AS count, SUM(CASE WHEN \"delayed_jobs\".\"run_at\" > '2025-11-10 17:20:13' THEN 1 ELSE 0 END) AS future_count, SUM(CASE WHEN \"delayed_jobs\".\"attempts\" > 0 THEN 1 ELSE 0 END) AS erroring_count FROM \"delayed_jobs\" WHERE \"delayed_jobs\".\"failed_at\" IS NULL - GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\") subquery - GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\" + GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\") subquery + GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\", \"name\" CO-ROUTINE subquery SCAN delayed_jobs USING INDEX delayed_jobs_priority @@ -383,8 +398,9 @@ SELECT SUM(claimed_count) AS claimed_count, MIN(run_at) AS run_at, CURRENT_TIMESTAMP AS db_now_utc, CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END AS priority, - queue AS queue - FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", SUM(CASE WHEN \"delayed_jobs\".\"locked_at\" >= '2025-11-10 16:59:43' THEN 1 ELSE 0 END) AS claimed_count, + queue AS queue, + name AS name + FROM (SELECT \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\", SUM(CASE WHEN \"delayed_jobs\".\"locked_at\" >= '2025-11-10 16:59:43' THEN 1 ELSE 0 END) AS claimed_count, SUM(CASE WHEN (\"delayed_jobs\".\"locked_at\" IS NULL OR \"delayed_jobs\".\"locked_at\" < '2025-11-10 16:59:43') THEN 1 ELSE 0 END) AS claimable_count, MIN(CASE WHEN \"delayed_jobs\".\"locked_at\" >= '2025-11-10 16:59:43' THEN locked_at ELSE NULL END) AS locked_at, @@ -393,8 +409,8 @@ SELECT SUM(claimed_count) AS claimed_count, FROM \"delayed_jobs\" WHERE \"delayed_jobs\".\"failed_at\" IS NULL AND \"delayed_jobs\".\"run_at\" <= '2025-11-10 17:20:13' - GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\") subquery - GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\" + GROUP BY \"delayed_jobs\".\"priority\", \"delayed_jobs\".\"queue\", \"delayed_jobs\".\"name\") subquery + GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, \"queue\", \"name\" CO-ROUTINE subquery SCAN delayed_jobs USING INDEX delayed_jobs_priority @@ -430,12 +446,13 @@ snapshots["runs the expected mysql2 queries with the expected plans 1"] = <<-SNA --------------------------------- SELECT SUM(count) AS count, CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END AS priority, - queue AS queue - FROM (SELECT `delayed_jobs`.`priority`, `delayed_jobs`.`queue`, COUNT(*) AS count + queue AS queue, + name AS name + FROM (SELECT `delayed_jobs`.`priority`, `delayed_jobs`.`queue`, `delayed_jobs`.`name`, COUNT(*) AS count FROM `delayed_jobs` WHERE `delayed_jobs`.`failed_at` IS NOT NULL - GROUP BY `delayed_jobs`.`priority`, `delayed_jobs`.`queue`) subquery - GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, `queue` + GROUP BY `delayed_jobs`.`priority`, `delayed_jobs`.`queue`, `delayed_jobs`.`name`) subquery + GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, `queue`, `name` -> Table scan on -> Aggregate using temporary table @@ -444,20 +461,21 @@ SELECT SUM(count) AS count, -> Table scan on -> Aggregate using temporary table -> Filter: (delayed_jobs.failed_at is not null) (cost=...) - -> Covering index range scan on delayed_jobs using idx_delayed_jobs_live over (NULL < failed_at) (cost=...) + -> Table scan on delayed_jobs (cost=...) --- SELECT SUM(count) AS count, SUM(future_count) AS future_count, SUM(erroring_count) AS erroring_count, CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END AS priority, - queue AS queue - FROM (SELECT `delayed_jobs`.`priority`, `delayed_jobs`.`queue`, COUNT(*) AS count, + queue AS queue, + name AS name + FROM (SELECT `delayed_jobs`.`priority`, `delayed_jobs`.`queue`, `delayed_jobs`.`name`, COUNT(*) AS count, SUM(CASE WHEN `delayed_jobs`.`run_at` > '2025-11-10 17:20:13' THEN 1 ELSE 0 END) AS future_count, SUM(CASE WHEN `delayed_jobs`.`attempts` > 0 THEN 1 ELSE 0 END) AS erroring_count FROM `delayed_jobs` WHERE `delayed_jobs`.`failed_at` IS NULL - GROUP BY `delayed_jobs`.`priority`, `delayed_jobs`.`queue`) subquery - GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, `queue` + GROUP BY `delayed_jobs`.`priority`, `delayed_jobs`.`queue`, `delayed_jobs`.`name`) subquery + GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, `queue`, `name` -> Table scan on -> Aggregate using temporary table @@ -465,8 +483,7 @@ SELECT SUM(count) AS count, -> Materialize (cost=...) -> Table scan on -> Aggregate using temporary table - -> Filter: (delayed_jobs.failed_at is null) (cost=...) - -> Covering index lookup on delayed_jobs using idx_delayed_jobs_live (failed_at = NULL) (cost=...) + -> Index lookup on delayed_jobs using idx_delayed_jobs_live (failed_at = NULL), with index condition: (delayed_jobs.failed_at is null) (cost=...) --- -- QUERIES FOR `future_count`: --------------------------------- @@ -479,8 +496,9 @@ SELECT SUM(claimed_count) AS claimed_count, MIN(run_at) AS run_at, UTC_TIMESTAMP() AS db_now_utc, CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END AS priority, - queue AS queue - FROM (SELECT `delayed_jobs`.`priority`, `delayed_jobs`.`queue`, SUM(CASE WHEN `delayed_jobs`.`locked_at` >= '2025-11-10 16:59:43' THEN 1 ELSE 0 END) AS claimed_count, + queue AS queue, + name AS name + FROM (SELECT `delayed_jobs`.`priority`, `delayed_jobs`.`queue`, `delayed_jobs`.`name`, SUM(CASE WHEN `delayed_jobs`.`locked_at` >= '2025-11-10 16:59:43' THEN 1 ELSE 0 END) AS claimed_count, SUM(CASE WHEN (`delayed_jobs`.`locked_at` IS NULL OR `delayed_jobs`.`locked_at` < '2025-11-10 16:59:43') THEN 1 ELSE 0 END) AS claimable_count, MIN(CASE WHEN `delayed_jobs`.`locked_at` >= '2025-11-10 16:59:43' THEN locked_at ELSE NULL END) AS locked_at, @@ -489,8 +507,8 @@ SELECT SUM(claimed_count) AS claimed_count, FROM `delayed_jobs` WHERE `delayed_jobs`.`failed_at` IS NULL AND `delayed_jobs`.`run_at` <= '2025-11-10 17:20:13' - GROUP BY `delayed_jobs`.`priority`, `delayed_jobs`.`queue`) subquery - GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, `queue` + GROUP BY `delayed_jobs`.`priority`, `delayed_jobs`.`queue`, `delayed_jobs`.`name`) subquery + GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, `queue`, `name` -> Table scan on -> Aggregate using temporary table @@ -498,8 +516,7 @@ SELECT SUM(claimed_count) AS claimed_count, -> Materialize (cost=...) -> Table scan on -> Aggregate using temporary table - -> Filter: ((delayed_jobs.failed_at is null) and (delayed_jobs.run_at <= TIMESTAMP'2025-11-10 17:20:13')) (cost=...) - -> Covering index lookup on delayed_jobs using idx_delayed_jobs_live (failed_at = NULL) (cost=...) + -> Index lookup on delayed_jobs using idx_delayed_jobs_live (failed_at = NULL), with index condition: ((delayed_jobs.failed_at is null) and (delayed_jobs.run_at <= TIMESTAMP'2025-11-10 17:20:13')) (cost=...) --- -- QUERIES FOR `erroring_count`: --------------------------------- @@ -529,12 +546,13 @@ snapshots["[legacy index] runs the expected mysql2 queries with the expected pla --------------------------------- SELECT SUM(count) AS count, CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END AS priority, - queue AS queue - FROM (SELECT `delayed_jobs`.`priority`, `delayed_jobs`.`queue`, COUNT(*) AS count + queue AS queue, + name AS name + FROM (SELECT `delayed_jobs`.`priority`, `delayed_jobs`.`queue`, `delayed_jobs`.`name`, COUNT(*) AS count FROM `delayed_jobs` WHERE `delayed_jobs`.`failed_at` IS NOT NULL - GROUP BY `delayed_jobs`.`priority`, `delayed_jobs`.`queue`) subquery - GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, `queue` + GROUP BY `delayed_jobs`.`priority`, `delayed_jobs`.`queue`, `delayed_jobs`.`name`) subquery + GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, `queue`, `name` -> Table scan on -> Aggregate using temporary table @@ -549,14 +567,15 @@ SELECT SUM(count) AS count, SUM(future_count) AS future_count, SUM(erroring_count) AS erroring_count, CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END AS priority, - queue AS queue - FROM (SELECT `delayed_jobs`.`priority`, `delayed_jobs`.`queue`, COUNT(*) AS count, + queue AS queue, + name AS name + FROM (SELECT `delayed_jobs`.`priority`, `delayed_jobs`.`queue`, `delayed_jobs`.`name`, COUNT(*) AS count, SUM(CASE WHEN `delayed_jobs`.`run_at` > '2025-11-10 17:20:13' THEN 1 ELSE 0 END) AS future_count, SUM(CASE WHEN `delayed_jobs`.`attempts` > 0 THEN 1 ELSE 0 END) AS erroring_count FROM `delayed_jobs` WHERE `delayed_jobs`.`failed_at` IS NULL - GROUP BY `delayed_jobs`.`priority`, `delayed_jobs`.`queue`) subquery - GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, `queue` + GROUP BY `delayed_jobs`.`priority`, `delayed_jobs`.`queue`, `delayed_jobs`.`name`) subquery + GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, `queue`, `name` -> Table scan on -> Aggregate using temporary table @@ -578,8 +597,9 @@ SELECT SUM(claimed_count) AS claimed_count, MIN(run_at) AS run_at, UTC_TIMESTAMP() AS db_now_utc, CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END AS priority, - queue AS queue - FROM (SELECT `delayed_jobs`.`priority`, `delayed_jobs`.`queue`, SUM(CASE WHEN `delayed_jobs`.`locked_at` >= '2025-11-10 16:59:43' THEN 1 ELSE 0 END) AS claimed_count, + queue AS queue, + name AS name + FROM (SELECT `delayed_jobs`.`priority`, `delayed_jobs`.`queue`, `delayed_jobs`.`name`, SUM(CASE WHEN `delayed_jobs`.`locked_at` >= '2025-11-10 16:59:43' THEN 1 ELSE 0 END) AS claimed_count, SUM(CASE WHEN (`delayed_jobs`.`locked_at` IS NULL OR `delayed_jobs`.`locked_at` < '2025-11-10 16:59:43') THEN 1 ELSE 0 END) AS claimable_count, MIN(CASE WHEN `delayed_jobs`.`locked_at` >= '2025-11-10 16:59:43' THEN locked_at ELSE NULL END) AS locked_at, @@ -588,8 +608,8 @@ SELECT SUM(claimed_count) AS claimed_count, FROM `delayed_jobs` WHERE `delayed_jobs`.`failed_at` IS NULL AND `delayed_jobs`.`run_at` <= '2025-11-10 17:20:13' - GROUP BY `delayed_jobs`.`priority`, `delayed_jobs`.`queue`) subquery - GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, `queue` + GROUP BY `delayed_jobs`.`priority`, `delayed_jobs`.`queue`, `delayed_jobs`.`name`) subquery + GROUP BY CASE WHEN priority < 10 THEN 0 WHEN priority < 20 THEN 10 WHEN priority < 30 THEN 20 WHEN priority >= 30 THEN 30 END, `queue`, `name` -> Table scan on -> Aggregate using temporary table diff --git a/spec/delayed/monitor_spec.rb b/spec/delayed/monitor_spec.rb index 4e13fa9..c9ac56d 100644 --- a/spec/delayed/monitor_spec.rb +++ b/spec/delayed/monitor_spec.rb @@ -132,10 +132,10 @@ let(:p10_attributes) { job_attributes.merge(priority: 13, locked_at: now - 1.day) } let(:p20_attributes) { job_attributes.merge(priority: 23, attempts: 1) } let(:p30_attributes) { job_attributes.merge(priority: 999, locked_at: now - 1.day) } - let(:p0_payload) { default_payload.merge(priority: 'interactive') } - let(:p10_payload) { default_payload.merge(priority: 'user_visible') } - let(:p20_payload) { default_payload.merge(priority: 'eventual') } - let(:p30_payload) { default_payload.merge(priority: 'reporting') } + let(:p0_payload) { default_payload.merge(priority: 'interactive', name: 'SimpleJob') } + let(:p10_payload) { default_payload.merge(priority: 'user_visible', name: 'SimpleJob') } + let(:p20_payload) { default_payload.merge(priority: 'eventual', name: 'SimpleJob') } + let(:p30_payload) { default_payload.merge(priority: 'reporting', name: 'SimpleJob') } let!(:p0_workable_job) { Delayed::Job.create! p0_attributes.merge(run_at: now - 30.seconds) } let!(:p0_failed_job) { Delayed::Job.create! p0_attributes.merge(failed_attributes) } let!(:p0_future_job) { Delayed::Job.create! p0_attributes.merge(run_at: now + 1.hour) } @@ -201,6 +201,96 @@ .and emit_notification("delayed.job.max_age").with_payload(p30_payload.merge(queue: 'banana')).approximately.with_value(4.hours) end + context 'when multiple job names share a priority and queue' do + let!(:other_named_job) { Delayed::Job.create! p0_attributes.merge(name: 'OtherJob', run_at: now - 10.minutes) } + + it 'emits a separate series per name' do + expect { subject.run! } + .to emit_notification("delayed.job.max_age").with_payload(p0_payload).approximately.with_value(30.seconds) + .and emit_notification("delayed.job.max_age").with_payload(p0_payload.merge(name: 'OtherJob')).approximately.with_value(10.minutes) + end + end + + context 'when tag_columns is empty' do + around do |example| + described_class.tag_columns = [] + example.run + ensure + described_class.tag_columns = %i(name) + end + + it 'emits metrics without name tags' do + expect { subject.run! } + .to emit_notification("delayed.job.max_age").with_payload(p0_payload.except(:name)).approximately.with_value(30.seconds) + end + end + + context 'when the delayed_jobs table has no name column' do + before do + described_class.instance_variable_set(:@tag_columns, nil) + allow(Delayed::Job).to receive(:column_names).and_return(Delayed::Job.column_names - ['name']) + end + + after do + described_class.instance_variable_set(:@tag_columns, nil) + end + + it 'defaults to emitting metrics without name tags' do + expect { subject.run! } + .to emit_notification("delayed.job.max_age").with_payload(p0_payload.except(:name)).approximately.with_value(30.seconds) + end + end + + context 'when a job predates the name column' do + around do |example| + ValidateRunAtAndNameNotNull.migrate(:down) + AddRunAtAndNameNotNullCheck.migrate(:down) + example.run + ensure + Delayed::Job.delete_all + AddRunAtAndNameNotNullCheck.migrate(:up) + ValidateRunAtAndNameNotNull.migrate(:up) + end + + let!(:unnamed_job) { Delayed::Job.create! p0_attributes.merge(name: nil, run_at: now - 10.minutes) } + + it "emits metrics under the name 'unset'" do + expect { subject.run! } + .to emit_notification("delayed.job.max_age").with_payload(p0_payload.merge(name: 'unset')).approximately.with_value(10.minutes) + end + end + + context 'when tag_columns includes a custom column' do + around do |example| + Delayed::Job.connection.add_column :delayed_jobs, :owner, :string + Delayed::Job.reset_column_information + described_class.tag_columns = %i(name owner) + example.run + ensure + described_class.tag_columns = %i(name) + Delayed::Job.connection.remove_column :delayed_jobs, :owner + Delayed::Job.reset_column_information + end + + let!(:team_a_job) { Delayed::Job.create! p0_attributes.merge(owner: 'team_a', run_at: now - 10.minutes) } + let!(:team_b_job) { Delayed::Job.create! p0_attributes.merge(owner: 'team_b', run_at: now - 20.minutes) } + + it "tags each series with the column's value, reporting NULLs as 'unset'" do + expect { subject.run! } + .to emit_notification("delayed.job.max_age").with_payload(p0_payload.merge(owner: 'team_a')).approximately.with_value(10.minutes) + .and emit_notification("delayed.job.max_age").with_payload(p0_payload.merge(owner: 'team_b')).approximately.with_value(20.minutes) + .and emit_notification("delayed.job.max_age").with_payload(p0_payload.merge(owner: 'unset')).approximately.with_value(30.seconds) + .and emit_notification("delayed.job.failed_count").with_payload(p0_payload.merge(owner: 'unset')).with_value(1) + end + end + + context 'when tag_columns names a column that does not exist' do + it 'raises loudly rather than skipping the column' do + expect { described_class.tag_columns = %i(name owner) } + .to raise_error(ArgumentError, /tag_columns includes columns missing from delayed_jobs\. Available columns: .*\bname\b/) + end + end + context 'when named priorities are customized' do around do |example| Delayed::Priority.names = { high: 0, low: 20 } @@ -208,8 +298,8 @@ ensure Delayed::Priority.names = nil end - let(:p0_payload) { default_payload.merge(priority: 'high') } - let(:p20_payload) { default_payload.merge(priority: 'low') } + let(:p0_payload) { default_payload.merge(priority: 'high', name: 'SimpleJob') } + let(:p20_payload) { default_payload.merge(priority: 'low', name: 'SimpleJob') } it 'emits the expected results for each metric' do expect { subject.run! } @@ -223,7 +313,7 @@ .and emit_notification("delayed.job.workable_count").with_payload(p0_payload).with_value(2) .and emit_notification("delayed.job.max_age").with_payload(p0_payload).approximately.with_value(2.minutes) .and emit_notification("delayed.job.max_lock_age").with_payload(p0_payload).approximately.with_value(7.minutes) - .and emit_notification("delayed.job.alert_age_percent").with_payload(p0_payload).approximately.with_value(0) + .and emit_notification("delayed.job.alert_age_percent").with_payload(p0_payload.except(:name)).approximately.with_value(0) .and emit_notification("delayed.job.count").with_payload(p20_payload).with_value(8) .and emit_notification("delayed.job.future_count").with_payload(p20_payload).with_value(2) .and emit_notification("delayed.job.locked_count").with_payload(p20_payload).with_value(2) @@ -233,7 +323,7 @@ .and emit_notification("delayed.job.workable_count").with_payload(p20_payload).with_value(2) .and emit_notification("delayed.job.max_age").with_payload(p20_payload).approximately.with_value(6.hours) .and emit_notification("delayed.job.max_lock_age").with_payload(p20_payload).approximately.with_value(11.minutes) - .and emit_notification("delayed.job.alert_age_percent").with_payload(p20_payload).approximately.with_value(0) + .and emit_notification("delayed.job.alert_age_percent").with_payload(p20_payload.except(:name)).approximately.with_value(0) .and emit_notification("delayed.job.workable_count").with_payload(p20_payload.merge(queue: 'banana')).with_value(1) .and emit_notification("delayed.job.max_age").with_payload(p20_payload.merge(queue: 'banana')).approximately.with_value(4.hours) end @@ -264,7 +354,7 @@ Delayed::Priority.names = nil Delayed::Worker.queues = [] end - let(:banana_payload) { default_payload.merge(queue: 'banana', priority: 'interactive') } + let(:banana_payload) { default_payload.merge(queue: 'banana', priority: 'interactive', name: 'SimpleJob') } let(:gram_payload) { default_payload.merge(queue: 'gram', priority: 'interactive') } it 'emits the expected results for each queue' do @@ -274,7 +364,7 @@ .and emit_notification("delayed.job.future_count").with_payload(banana_payload).with_value(0) .and emit_notification("delayed.job.locked_count").with_payload(banana_payload).with_value(0) .and emit_notification("delayed.job.erroring_count").with_payload(banana_payload).with_value(0) - .and emit_notification("delayed.job.failed_count").with_payload(banana_payload).with_value(0) + .and emit_notification("delayed.job.failed_count").with_payload(banana_payload.except(:name)).with_value(0) .and emit_notification("delayed.job.working_count").with_payload(banana_payload).with_value(0) .and emit_notification("delayed.job.workable_count").with_payload(banana_payload).with_value(1) .and emit_notification("delayed.job.max_age").with_payload(banana_payload).approximately.with_value(4.hours) @@ -346,7 +436,7 @@ end context 'when a job is locked (in-flight)' do - let(:payload) { default_payload.merge(priority: 'interactive') } + let(:payload) { default_payload.merge(priority: 'interactive', name: 'SimpleJob') } let(:base_attributes) do { priority: 0,