Message brokers in the data-streams view

Embed
Preview of the card
<a href="https://watchtowerops.net/docs/queue-brokers"><img src="https://watchtowerops.net/docs/_api/public/orgs/aa5959cb-1a0b-4a9a-bc9a-913e7ac31cd9/spaces/b7aac181-9916-4805-b253-becc5fd2ca62/pages/c9ab1066-fc61-4737-9eb5-4dac0e48feaf/social-card.svg?v=3e7ae33eec8b" alt="Message brokers in the data-streams view" width="600" height="315"></a>

The Queues area (GET /v1/projects/{prj}/queues, Infrastructure, Queues and streams) reads one metric contract whatever the broker: messaging.queue.depth{queue, framework, state} (ready, unacked, dead), messaging.queue.oldest_age_seconds and messaging.consumer.lag_seconds. Every collector below keeps its own series and also writes these, so a broker shows next to the spans of the same destination (spans join on the queue name; framework is the engine).

Engine

How it is read

Queue name

ready

unacked

oldest age / lag

RabbitMQ

management API (rabbitmq module, url, username, password)

queue (+ vhost)

messages_ready

messages_unacknowledged

head message timestamp

ActiveMQ classic

Jolokia (activemq module)

queue (+ broker)

QueueSize

InFlightCount

none

ActiveMQ Artemis

Jolokia (artemis module)

queue (+ broker, address)

MessageCount

DeliveringCount

FirstMessageAge

NATS JetStream

monitoring /jsz (nats module)

stream (+ consumer)

consumer pending

ack pending

none

Pulsar

admin topic stats (pulsar module)

topic (+ subscription)

msgBacklog

unackedMessages

earliest message in backlog

Kafka

consumer group lag (kafka module)

topic (+ group)

group lag

none

lag time

Redis streams

redis_streams

stream (+ group)

group lag

group pending

oldest pending, lag time

Amazon SQS

CloudWatch (aws, preset data_streams)

queue name

messages visible

messages not visible

oldest message age

Amazon Kinesis

CloudWatch (aws, preset data_streams)

stream name

none

none

iterator age (lag)

Azure Service Bus

Azure Monitor (azure, preset azure_service_bus)

queue or topic (entity), else namespace

ActiveMessages

none

none (dead: DeadletteredMessages)

Google Pub/Sub

Cloud Monitoring (gcp, metrics below)

subscription id

num_undelivered_messages

none

oldest_unacked_message_age

Notes:

  • Service Bus is asked for one series per entity (EntityName eq '*'); the dead-letter count lands in state dead.

  • Pub/Sub: list pubsub.googleapis.com/subscription/num_undelivered_messages, .../oldest_unacked_message_age and .../dead_letter_message_count in the gcp pull's metrics.

  • Amazon SNS has no backlog (it pushes); its pathways come from span links, and failed deliveries stay in the raw CloudWatch series.

  • A dead-letter queue is recognised by name (*.dlq, *-dlq, *.dead, *-dead, or the project's queue_dlq_patterns) or by state dead.

  • Modules started from pod annotations get the same series.

Updated