Metric query dialects (MET-016)

Embed
Preview of the card
<a href="https://watchtowerops.net/docs/metric-dialects"><img src="https://watchtowerops.net/docs/_api/public/orgs/aa5959cb-1a0b-4a9a-bc9a-913e7ac31cd9/spaces/b7aac181-9916-4805-b253-becc5fd2ca62/pages/286a2607-a9a7-4c18-8ace-2a47ebd6c765/social-card.svg?v=3e7ae33eec8b" alt="Metric query dialects (MET-016)" width="600" height="315"></a>

Watchtower's metric engine runs PromQL. Queries written in other vendors' metric languages are translated to PromQL first, by our own parsers in crates/wt-promql/src/dialects/ (Datadog in crates/wt-promql/src/dd.rs). There are two ways to use them:

  • dialect=<name> on the Prometheus query endpoints (/v1/projects/{prj}/prometheus/api/v1/query and query_range): the query text is translated and run.

  • POST /v1/translate/metric-query {"dialect", "query"} returns {"promql"}, so you can see and edit the result. The Metrics page has a "Translate from another metric language" panel.

Names are datadog (or dd), signalflow, dql, ppl, apl, uql and badgerql.

Anything outside the subsets below is refused, never dropped. The translate endpoint answers 422 application/problem+json with detail: "unsupported construct: <dialect> <construct>", for example unsupported construct: uql table operation join. The Prometheus endpoints answer their usual 422 unsupported_promql with the same name. Metric and tag names keep their spelling, except that . and - become _.

Every dialect maps to the same shape: one metric selector with label matchers, an optional per-series window function, an optional aggregation across series with by labels, and an optional top or bottom k. Time-bucketed aggregations (stats avg(..) by span(1m)) become a window function over the bucket, followed by the same aggregation across series. A per-bucket count is summed across series.

OTLP metric names (W17)

The dialects above translate query languages. A separate question is how an OTLP metric or attribute name appears to PromQL (alert rules, dashboards, the Prometheus API). otlp-metric-names.md has the full rules (the project's otlp.translation_strategy, the UCUM unit table, resource-attribute promotion, target_info); this is the short version for writing a selector.

OTLP name and type

PromQL

container.cpu.usage (gauge, from the agent)

container_cpu_usage

http.server.duration (histogram, ms)

http_server_duration_milliseconds_bucket (and _sum, _count)

counter jobs.done

jobs_done_total

  • Dots and dashes. Any character outside [a-zA-Z0-9_:] becomes _; runs of _ collapse, and a leading digit gets an _ prefix (2xx -> _2xx). Attribute names escape the same way but to [a-zA-Z0-9_] (no :), with a key_ prefix for a leading digit.

  • Units. The UCUM unit is appended as a word: s -> _seconds, ms -> _milliseconds, By -> _bytes, a rate By/s -> _bytes_per_second; a gauge whose unit is 1 gets _ratio. It is added only when the name doesn't already contain that word.

  • _total. A monotonic sum (counter) ends in _total, added after the unit suffix and never doubled (a trailing _total already on the name is stripped first).

  • Histograms expose _bucket (with an le label), _sum and _count series.

  • Resource attributes -> labels. service.name -> service, service.instance.id -> instance, host.name -> host_name, and the ingest key's environment -> env (plus env_id). Other resource attributes are not labels unless promoted (promote_resource_attributes); otherwise find them on the target_info series. Data-point attributes keep their name (escaped the same way under an escaping strategy).

The escaping and the unit/_total suffixes are what the UnderscoreEscapingWithSuffixes strategy does. The default strategy is NoTranslation: names and attribute keys are stored exactly as sent (dots kept) and queried quoted — {"http.server.duration_bucket"}, {"http.route"="/users"}. So the http_server_duration_milliseconds_bucket and jobs_done_total rows above assume an escaping-with-suffixes strategy; service.name -> service and the other resource-attribute labels hold under every strategy.

The one metric-name mapping that holds under any strategy is the agent's: it emits container CPU and memory under both the dotted OTLP name and its underscored alias (container.cpu.usage and container_cpu_usage, container.memory.usage and container_memory_usage), so container_cpu_usage{service="hermetica-api"} works out of the box. These dotted/underscored duplicates are kept on purpose and have no scheduled removal: the dotted name is the canonical OTLP series and the underscored one is a convenience alias for PromQL.

SignalFlow

data('cpu.utilization', filter=filter('host', 'a', 'b') and not filter('env', 'dev'), rollup='rate').mean(by=['host']).top(count=5).publish()

  • An optional stream name: A = data(..).

  • data(metric, filter=.., rollup=..). Filters are filter(key, value, ..) terms (* wildcards, several values match any of them), not filter(..), and and. or is unsupported.

  • rollup=: average/mean, sum, min, max, count, latest, rate, delta, over 60 s.

  • .mean/.sum/.min/.max/.count(by=[..]): one aggregation across series.

  • .mean(over='5m') (and the other aggregations): a moving window per series, before any aggregation.

  • .top(count=n) and .bottom(count=n), then .publish(..).

Dynatrace DQL

timeseries cpu = avg(dt.host.cpu.usage), by: {dt.entity.host}, filter: {env == "prod" and host != "h2"}, interval: 5m

  • Only the timeseries command, with one aggregation: avg, sum, min, max or count, and an optional alias.

  • rate: 1m inside the aggregation: a counter's rate per minute (rate(..) * 60).

  • by: {..}, filter: with ==, !=, =~, !~, in (..) and and (or and not are unsupported), and interval: (60 s by default).

  • Pipe commands after timeseries are unsupported.

OpenSearch PPL

source = cpu.usage | where host = 'a' and env != 'dev' | stats avg(value) by span(@timestamp, 1m), host

  • source = <metric>, then where and one stats with one aggregation.

  • The aggregated field is either value (the source metric's samples) or a metric name.

  • by takes span(<time field>, <duration>) and labels. Other commands are unsupported.

Axiom APL

['cpu.usage'] | where host == "a" | summarize count() by bin(_time, 5m), host

  • The dataset is the metric, then where and one summarize. bin(_time, d) sets the bucket, and bin_auto(_time) uses 60 s. The aggregated field follows the PPL rule.

ServiceNow Cloud Observability UQL

metric cpu.usage | filter host == "a" && env != "dev" | rate 1m | group_by [host], sum | top 5

  • metric <name>, filter (==, !=, =~, !~, &&), the aligners rate, delta, latest, align <fn>[, window] and reduce <fn>[, window], one group_by [..], <agg>, then top n or bottom n. Joins and other table operations are unsupported.

Honeybadger BadgerQL

filter metric::str == "http.duration" and route::str == "/api" | stats avg(value::float) by bin(1h), route::str

  • ::type suffixes are ignored. A metric or name equality in filter names the metric. An optional leading fields projection is accepted. There is one stats stage, with bin(d) as the bucket.

Datadog

See the module docs of dd.rs: space aggregators, tag filters with wildcards, by {..}, .rollup(), .as_count(), .as_rate(), .timeshift(), .fill(), .ewma_N(), .median_N(), .cumsum(), .diff(), .per_second(), .top(), .outliers() and .anomalies().

Updated