Forecast alerts

Embed
Preview of the card
<a href="https://watchtowerops.net/docs/forecast"><img src="https://watchtowerops.net/docs/_api/public/orgs/aa5959cb-1a0b-4a9a-bc9a-913e7ac31cd9/spaces/b7aac181-9916-4805-b253-becc5fd2ca62/pages/81aa9dec-59b7-4ca8-9519-bd4a977d4115/social-card.svg?v=3e7ae33eec8b" alt="Forecast alerts" width="600" height="315"></a>

The forecast rule kind

A forecast rule fires when a metric is predicted to cross a threshold within a horizon, for example "a disk is full within 3 days":

{"name": "Disk full in 3 days", "kind": "forecast",
 "query": {"forecast": {"expr": "max by (host_name, mountpoint) (system_filesystem_utilization)",
                        "algorithm": "linear", "horizon_s": 259200}},
 "condition": {"threshold": {"op": "ge", "value": 0.95}}, "every_s": 300}

Field

Meaning

expr

PromQL returning series; each series is one alert group.

algorithm

linear (least-squares trend, default) or seasonal (additive Holt-Winters).

seasonality

daily (default) or weekly; seasonal only.

horizon_s

How far ahead, 60 s to 7 days.

history_s

History read, 1 hour to 31 days. Defaults: 1 day for linear, 4 days for daily, 28 days for weekly.

The condition must be a threshold with gt, ge, lt or le. The rule's value per series is the forecast's highest point within the horizon for gt/ge and its lowest for lt/le, so the threshold holds exactly when the forecast crosses it before the horizon. warn tiers, for_s, recovery hysteresis, no-data handling and routing work as for any metric rule.

History is read at 5 minute steps up to 7 days, hourly beyond. A seasonal forecast needs three full seasons of history (3 days for daily, 3 weeks for weekly); with less, a weekly season falls back to a daily one and a daily one to linear. A series with fewer than two points is no data for its group, never a firing. A forecast rule evaluates only its latest due tick after an outage.

Preview in the rule editor

Choosing the forecast kind shows a forecast preview under the condition: per series (first five) the history, the forecast with a band of two residual standard deviations, the threshold line, and the time the forecast first crosses it. The preview calls GET /v1/projects/{prj}/forecast with the rule's method and history.

Forecast condition on a metric rule

The older form, a metric rule with "condition": {"forecast": {"method", "horizon_s", "op", "value"}}, keeps working; capacity_forecast templates use it. New rules should use the forecast kind, which has a seasonal choice, its own history and the editor preview.

Updated