Job queues (track B2)

Embed
Preview of the card
<a href="https://watchtowerops.net/docs/job-queues"><img src="https://watchtowerops.net/docs/_api/public/orgs/aa5959cb-1a0b-4a9a-bc9a-913e7ac31cd9/spaces/b7aac181-9916-4805-b253-becc5fd2ca62/pages/af6530b4-515d-41a0-a40c-26f10f538b0c/social-card.svg?v=3e7ae33eec8b" alt="Job queues (track B2)" width="600" height="315"></a>

The agent reads each job framework's own store (no change to the application, no exporter) and reports one shared metric family, so every framework feeds the same Queues page and alert pack shape. One agent module per framework; the module name is also the integration manifest (integrations/<name>/).

Metric contract (shared with the Queues page, track E)

Metric

Type

Labels

Meaning

job_queue_jobs

gauge

framework, queue, state

Jobs per state. Standard states: waiting (ready to run), active (held by a worker), scheduled (run later), retry (failed, will run again), dead (retries exhausted, parked). Framework extras: blocked, pending, cancelled. Completed jobs are never counted.

job_queue_oldest_waiting_age_seconds

gauge

framework, queue

Seconds since the oldest waiting job became ready; 0 when nothing waits. Only where the store records the time.

job_queue_stalled_jobs

gauge

framework, queue

Jobs held by a worker past the framework's lease (Laravel reserved, RQ started registry, Asynq lease, Celery visibility) or, without a lease, running longer than stall_after (default 1h).

job_queue_workers

gauge

framework, queue

Live worker processes; queue="*" when the store does not split them per queue.

job_queue_processed_total, job_queue_failed_total

counter

framework, queue

Only where the framework keeps counters.

wt.integration.up

gauge

integration

1 per healthy collection round, 0 when the store cannot be read.

queue="*" marks a value the store keeps for all queues together (for example Sidekiq's retry and dead sets). Every series a queue ever had is written each round, as 0 when the store no longer reports it, so a drained queue reads 0 instead of going stale. At most 200 queues per module. The BullMQ module (bullmq_*) predates this family and keeps its own names.

Common settings: interval (default 30s), queues (filter, or the list itself where the store has none), stall_after. Credentials (dsn, addr/password, token) live only in the agent's own config file. Postgres sessions are read-only.

Each pack: backlog (waiting > 1000 for 10 min), oldest waiting job > 5 min (where the age exists), dead jobs rising (> 10 in 10 min), stalled jobs (where reported), no workers (where reported) and unreachable.

Frameworks

Module

Store

Reads

Settings

pgboss

Postgres

<schema>.job per name and state (v9 and v10+ columns); age from start_after

dsn, schema (default pgboss)

graphile_worker

Postgres

<schema>.jobs per task identifier: locked (active), attempts exhausted (dead), future run_at (retry or scheduled); lock older than stall_after is stalled

dsn, schema (default graphile_worker), stall_after

oban

Postgres

<prefix>.oban_jobs per queue and state (available, executing, scheduled, retryable, discarded, cancelled); executing longer than stall_after is stalled

dsn, schema (Oban prefix, default public), stall_after

river

Postgres

<schema>.river_job per queue and state (available, running, scheduled, retryable, discarded, pending, cancelled); running longer than stall_after is stalled

dsn, schema (default public), stall_after

good_job

Postgres

<schema>.good_jobs unfinished jobs by queue: performed (active), errored and rescheduled (retry), future (scheduled), ready (waiting); finished with an error (dead); performed longer than stall_after is stalled

dsn, schema (default public), stall_after

solid_queue

Postgres

solid_queue_ready/scheduled/blocked/claimed/failed_executions per queue; claimed by a dead or silent (5 min) process is stalled; live Worker processes

dsn, schema (default public)

que

Postgres

<schema>.que_jobs (Que 1.x and 2.x) unfinished per queue; active = advisory-locked by a worker; expired (dead); que_lockers.worker_count

dsn, schema (default public)

sidekiq

Redis

queues, queue:<q> (depth, oldest enqueued_at), schedule/retry/dead (queue="*"), live processes and their busy, stat:processed/stat:failed

addr, password, username, tls, db, prefix (redis-namespace, e.g. myapp:)

resque

Redis

<prefix>queues, queue:<q> depth (no enqueue time in payloads, so no age), failed list (dead), workers and busy worker:<id> keys, stat:processed/stat:failed

addr, password, tls, db, prefix (default resque:)

celery

Redis

per configured queue the list and its priority lists; unacked (active) and unacked_index entries older than stall_after (stalled). On RabbitMQ use the rabbitmq module

addr, password, tls, db, queues (required), prefix (kombu global_keyprefix), stall_after (= visibility_timeout)

rq

Redis

rq:queues, rq:queue:<q> (depth, oldest job's enqueued_at), registries rq:wip (active; entries past their expiry are stalled), rq:scheduled, rq:deferred (blocked), rq:failed (dead), rq:workers:<q>

addr, password, tls, db, prefix

dramatiq

Redis

per configured queue <ns>:<q> (depth, oldest message_timestamp), .DQ (scheduled), .XQ (dead), in-flight ids of live workers (active), __heartbeats__ (workers). On RabbitMQ use the rabbitmq module

addr, password, tls, db, queues (required), prefix (default dramatiq:)

horizon

Redis

per configured queue queues:<q> (depth, oldest pushedAt/createdAt), :delayed (scheduled), :reserved (active; past retry_after is stalled); Horizon failed_jobs (dead) and supervisor process counts (workers)

addr, password, tls, db, queues (required), prefix (Laravel redis prefix, default laravel_database_), horizon_prefix (default laravel_horizon:)

asynq

Redis

asynq:queues, per queue asynq:{q}:pending (depth, oldest pending_since), active, scheduled, retry, archived (dead), expired lease entries (stalled), processed/failed totals; live asynq:servers (workers)

addr, password, tls, db, prefix

hangfire

SQL Server

JobQueue per queue (unfetched = waiting with the Enqueued state's age, fetched = active), Job in Scheduled / Failed (queue="*"), Server heartbeats in 5 min (workers), stats:succeeded (processed). Redis storage (Hangfire.Pro) is not read

dsn (sqlserver://), schema (default HangFire)

faktory

Faktory server

INFO over the Faktory protocol (HELLO with the salted password hash): queue sizes, Busy (active), Scheduled, Retries, Dead, Workers, total_processed/total_failures. INFO has no job ages

addr (default port 7419), password, tls

temporal

Temporal HTTP API

DescribeTaskQueue with reportStats per configured task queue, workflow and activity summed: backlog (waiting), backlog age, distinct poller identities (workers). Needs a server that reports task queue stats (1.27+)

url (frontend HTTP port 7243), namespace (default default), token (Temporal Cloud API key), queues (required)

Not covered

  • Throughput for SQL-backed frameworks (they keep no counters and counting finished rows scans history); use the SDK spans for rates there.

  • Celery and Dramatiq on RabbitMQ: their queues are RabbitMQ queues, read by the rabbitmq module.

  • Hangfire on Redis (Hangfire.Pro.Redis), Solid Queue on MySQL or SQLite, Que before 1.0, resque-scheduler's delayed jobs (stored per timestamp, not countable without a scan).

  • Per-queue active counts where the store keeps them only per process (Sidekiq <process>:work, Resque workers); those report queue="*".

Updated