Metrics without Limits (tag configs)

Embed
Preview of the card
<a href="https://watchtowerops.net/docs/metrics-without-limits"><img src="https://watchtowerops.net/docs/_api/public/orgs/aa5959cb-1a0b-4a9a-bc9a-913e7ac31cd9/spaces/b7aac181-9916-4805-b253-becc5fd2ca62/pages/d72f71b6-57b5-4a63-8ad0-33a3c23c2aa0/social-card.svg?v=3e7ae33eec8b" alt="Metrics without Limits (tag configs)" width="600" height="315"></a>

Choose, per metric, which tags (labels) stay queryable. Everything else is dropped before storage, so high-cardinality tags nobody queries stop costing series. Console: Metrics → Tag configs.

Tag configs

A tag config is a metric rule of kind tags (/v1/projects/{prj}/metric-rules):

Spec

Meaning

{"keep": ["route", "status"]}

Allowlist: only these tags are indexed.

{"drop": ["pod", "user_id"]}

Denylist: these tags are dropped.

metric_match is an exact metric name or a prefix*. service and env are always kept. The worker applies a changed config within 30 s (its config cache), at ingest: series that become equal are merged (sums add, gauges keep the last value). Data already stored is not rewritten. POST /v1/projects/{prj}/metric-rules/preview shows series before and after, dollars saved, and the alert rules, rule groups and dashboards that use a dropped tag.

Ingested against indexed cardinality

  • Indexed series: stored in metric_series_v1, active in the last 24 h.

  • Ingested series: as received in the last 24 h, before the tag config. The worker counts them for every metric a tags rule matches (metric_ingested_card_1h, hourly uniq sketches, kept 35 days). A metric without a tag config ingests what it indexes.

GET /v1/projects/{prj}/metrics/usage?days=30&limit=100 lists metrics by ingested series with series_indexed, series_ingested, series_saved, queries, last_queried and tag_rule_id. ?unqueried_days=30|60|90 is the unqueried report: only metrics no query, alert rule or dashboard read in that many days.

GET /v1/projects/{prj}/metrics/tag-cardinality?metric=http_requests_total&days=30 lists every tag of one metric with values_ingested, values_indexed, queries, sources (query, alert, dashboard, notebook, slo), indexed (the tag config keeps it) and queried.

Query usage

Every executed PromQL query records which tags it used with which metric (metric_label_queries_1d, source query, flushed every 60 s). A query over several metrics credits each of its tags to all of them, so suggestions may keep a tag more than needed, never less. The janitor's refs_refresh re-reads alert rules, dashboards, notebooks and SLOs daily and writes their tags under their own source; those rows count only while less than two days old, so a removed reference clears after the next refresh.

Suggested configs and savings

GET /v1/projects/{prj}/metric-rules/suggestions?days=30|60|90&limit=100 suggests, per metric, an allowlist of the tags used in the window, ranked by series removed. Each row has series_before, series_after, keep, drop, est_bytes_saved_monthly, est_usd_saved_monthly (the metric's stored bytes over the last 30 days times the share of series removed, at the project's storage rate), tag_rule_id when it would replace an existing config, and rule, the body that applies it. A metric no query used gets an empty allowlist.

Bulk apply

POST /v1/projects/{prj}/metric-rules/bulk with {"rules": [{metric_match, kind, spec}, ...]} (1 to 100) applies many configs in one transaction: a rule whose metric_match and kind exist replaces that rule's spec (its version increases), others are created. Every rule is validated first; an invalid one or passing the 100-rule project cap writes nothing (422). The response is {data, created, updated}. The console's Apply button sends the selected suggestions.

Limits and estimates

Ingested counts are uniq estimates (about 2% error) and run low when one worker holds more than 500,000 hashes between flushes. Histogram metrics are counted under their base name. Savings are estimates from series share, not measured bytes.

Updated