Notification delivery

The Go notifier (go/notifier) delivers the notification_outbox written by the router. Delivery is at-least-once; webhook receivers dedupe on X-Watchtower-Idempotency-Key. Webhooks are always HMAC-signed (X-Watchtower-Signature); the API generates the signing secret when none is supplied and shows it once.

notification_failed alerts

When a message ends failed (5th failed attempt or a permanent error), the notifier opens one alert per receiver: alertname=notification_failed, source=notifier, receiver_id, project_id, severity high. The next successful delivery to that receiver resolves it in the same transaction as the delivery is recorded.

The alert reaches people only through the routing tree: add a route matching source=notifier (for example to the team that owns the receivers). The router has no project-wide default receiver, so with no matching route the alert is visible in the alerts list but is not sent anywhere. A fallback to a default receiver is not built.

Send critical alerts straight to a channel as well as paging

A route stops matching its later siblings once it matches: the first matching sibling wins. Set "continue": true on a route to keep matching the siblings after it, so an alert can reach more than one receiver. Use this to send critical alerts to a chat channel at once while they still page on-call: add a continue route that matches the severity and points at the channel receiver before the route that pages, so matching falls through to the paging route too.

{
  "matchers": [{ "name": "severity", "op": "eq", "value": "critical" }],
  "receiver": "rcv_...",
  "continue": true
}

continue defaults to false. Ordering is by sibling position; put the continue route first. Matcher op is one of eq, ne, re, nre (=, !=, =~, !~); a missing label matches the empty string. The channel receiver is any kind in docs/product/notifications/channels.md; the paging route points at an escalation_policy receiver.

Folded messages

At most one message per minute per (receiver, route, group) is sent; the rest wait in state folded and are settled together with the digest that reports their count.

Did an alert reach anyone?

Every alert carries delivery, null while pending:

GET /v1/projects/{prj}/alerts?delivery=undelivered lists every alert that reached no one, and the console shows a red banner while any are firing. A critical alert that reaches no one also goes, once, to the org owner's login email (the person who created the org; an org admin's when the owner has none). Creating or updating a critical rule while no deliverable receiver gets critical alerts succeeds with a warnings[] entry saying so.

GET /receivers and GET /routing show receivers that deliver to no one: not_delivering is disabled or unverified (an escalation_policy receiver whose people cannot be reached). GET /alert-coverage lists the severities with no deliverable receiver (uncovered_severities) and the on-call people with no verified push, voice or SMS method (unreachable_oncall); the on-call screen flags them too.

Contact methods and the login email

A person's login email is a contact method from day one, and high and critical pages fall back to it when they have no verified push or voice method. Adding an email method whose address is the login email of a member of the organization needs no code: it is verified on create (verification: member_email). Any other address gets a 6-digit code. An audible rule set whose first step is email only is saved with a warning (warnings[]) instead of being refused; a first step with neither push, a verified voice method nor email is refused (422).