Anomaly insights and the sweep

Embed
Preview of the card
<a href="https://watchtowerops.net/docs/insights"><img src="https://watchtowerops.net/docs/_api/public/orgs/aa5959cb-1a0b-4a9a-bc9a-913e7ac31cd9/spaces/b7aac181-9916-4805-b253-becc5fd2ca62/pages/597f925d-393f-4c36-9467-643bd0f0b951/social-card.svg?v=3e7ae33eec8b" alt="Anomaly insights and the sweep" width="600" height="315"></a>

Watchtower looks for anomalies in every project without any setup. A background sweep in the worker runs every five minutes and opens insights; related insights are folded into problems with a root-cause candidate.

What the sweep looks at

Signal

Source

Opens

red.error_rate, red.request_rate

the per-service RED rollup, last 48 h at 5-minute steps

spike, dip, level_shift; multi_metric when both move; population (an enabled population job)

logs.volume

log volume per service and level

spike, dip, level_shift

logs.new_pattern

log patterns first seen in the last 15 minutes

new_pattern

forecasts

saved forecasts (/forecasts)

forecast_breach

deploys

deployments that started 10 minutes to 2 hours ago

faulty_deploy

An insight opens only for an anomaly in the newest two buckets, and only once per service, metric and kind while it stays open, so a repeated pass opens nothing new.

Learning period

A series is judged once it has 48 points (four hours of five-minute buckets). Until then the project is warming up: the sweep runs, but nothing opens for that series.

Faulty deployments

For each deployment that did not fail, the error rate of the deployed service (every service for a deployment without a service) in the 30 minutes after it started is compared with the two hours before. Both windows need at least 50 requests. The deployment is flagged when the rate after is at least 5% and at least twice the rate before plus two points. The insight (kind: faulty_deploy, metric: error_rate) has the rate after as value, the rate before as expected, and details.deployment_id and details.version. One insight per deployment and service.

Status

GET /v1/projects/{prj}/insights/status (telemetry read access):

{
  "last_sweep_at": "2026-11-03T10:05:00Z",
  "interval_s": 300,
  "signals_covered": [{"signal": "red.error_rate", "series": 12, "warm": 10}],
  "first_data_at": "2026-11-02T08:00:00Z",
  "warming_up_until": "2026-11-02T12:00:00Z",
  "warming_up": false,
  "opened_last": 1,
  "error": null,
  "kinds": ["spike", "dip", "level_shift", "multi_metric", "population", "forecast_breach",
            "new_pattern", "faulty_deploy"]
}

series counts what the last pass saw (services, log series, saved forecasts, recent deployments), warm how many have enough history. last_sweep_at is null before the sweep has visited the project. error is a fixed text when the last pass could not read the telemetry. The services page shows this status above the service list.

Updated