Integrations

An integration is a directory integrations/<name>/manifest.yaml (shape: integrations/schema.json) that says how to recognize a product, which agent module collects it, what the metrics mean, which dashboard to show and which alerts to start with. Manifests are compiled into the API and read by the agent from disk.

v1 set (INF-002)

apache, clickhouse, coredns, elasticsearch, etcd, haproxy, host, kafka, memcached, mongodb, mysql, nats, nginx, postgres, rabbitmq, redis (16).

Collected by Watchtower modules: postgres, mysql, redis, memcached, mongodb, kafka, rabbitmq, nats (their own modules), nginx, apache, haproxy, elasticsearch (the http_status module, kind selects the page format), host (the host module). ClickHouse and CoreDNS use the existing prometheus module against the product's own metrics endpoint; the clickhouse module adds statement and table depth and etcd has its own etcd module (see datastores-nosql.md).

Target list

Ranked by how many reference vendors ship an integration for it (harvest frequency, most common first). The v1 set above covers the head of this list; the rest are added as manifests plus, where no Prometheus endpoint exists, a module.

1. Linux and Windows host, Docker, Kubernetes (host and container modules)

2. Postgres, MySQL and MariaDB, Redis, MongoDB, Elasticsearch and OpenSearch

3. nginx, Apache, HAProxy, Kafka, RabbitMQ, memcached

4. NATS, etcd, ClickHouse, CoreDNS, Consul, Vault

5. SQL Server, Oracle, Cassandra, Couchbase, Traefik, Envoy

6. Jenkins, GitLab, Prometheus exporters in general (the prometheus module covers them)

Manifest

name: postgres
version: "1.0.0"
discovery: {process_names: [postgres], ports: [5432], k8s_labels: {...}}
module: {kind: postgres, config_defaults: {dsn: "...", interval: "30s"}}
metrics: [{name, unit, description}]
dashboards: [dashboard spec v1]
alerts: [{name, rule, threshold_rationale, runbook_url, default_enabled}]
depends_on: [{parent_kind: host, equal_labels: [host_name]}]

A rule uses the metric names and labels as the worker stores them: dotted OTel names selected with {__name__="postgresql.database.xid_age"}, point attributes as labels, and the resource attribute host.name as host_name. Every module also reports wt.integration.up{integration=<name>} (1 per healthy round, 0 when the product cannot be reached), which the "unreachable" alert of each pack watches.

Modules and credentials

Addresses, DSNs, passwords and tokens live only in the agent's own config file (modules: section). Remote config can switch a module on or off and set its interval, nothing else, and cannot start a module the local file does not define. The agent never stamps wt.* attributes.

Job frameworks (pg-boss, Sidekiq, Oban and the rest) have one module each, documented with their shared job_queue_* metrics in job-queues.md.

Postgres reports the connected database's tables; configure one DSN per database to cover several. The XID wraparound alert fires above 1.5e9 (the freeze limit is 2e9).

Postgres Database Monitoring

Everything below runs in the agent, inside your network, on a read-only session; nothing needs the database to be reachable from Watchtower.

• dbm: true: per-queryid series from pg_stat_statements deltas (PostgreSQL 13+), the 200 busiest statements per round: postgresql.query.calls, .total_time (ms), .rows, .blks_hit, .blks_read (running totals kept by the agent, so pg_stat_statements_reset() does not break rate()), and postgresql.query.mean_time (ms, this round), labelled database, queryid and the bind-safe query.

• collect_samples: true: pg_stat_activity every second, aggregated every 10 s into db.activity.sessions{db.namespace,db.session.state,db.wait_event_type,db.wait_event} (average sessions; CPU when not waiting) and one db.query.sample log record per statement and wait event (at most 100), with bind-safe text and db.query.id.

• explain: true with collect_samples: sampled SELECTs are planned through the function below (at most 5 per 10 s, each statement once per 10 minutes, 2 s timeout). Literals are removed from the plan before it is sent; plans arrive as db.query.plan log records. Without the function, explain uses EXPLAIN (GENERIC_PLAN) on PostgreSQL 16+, and the API's own EXPLAIN (POST .../db/queries/{hash}/explain) remains the fallback for databases it can reach.

Create the function as a role that can read the tables (it runs EXPLAIN only, never ANALYZE), and let only the monitoring role call it:

CREATE SCHEMA IF NOT EXISTS watchtower;
CREATE OR REPLACE FUNCTION watchtower.explain_statement(l_query text, OUT explain json)
RETURNS SETOF json AS $$
DECLARE curs refcursor; plan json;
BEGIN
  OPEN curs FOR EXECUTE pg_catalog.concat('EXPLAIN (FORMAT JSON) ', l_query);
  FETCH curs INTO plan;
  CLOSE curs;
  RETURN QUERY SELECT plan;
END;
$$ LANGUAGE plpgsql RETURNS NULL ON NULL INPUT SECURITY DEFINER SET statement_timeout = '2s';
REVOKE ALL ON FUNCTION watchtower.explain_statement(text) FROM PUBLIC;
GRANT USAGE ON SCHEMA watchtower TO watchtower_agent;
GRANT EXECUTE ON FUNCTION watchtower.explain_statement(text) TO watchtower_agent;

explain_function names another schema.function with the same signature.

Service discovery (INF-021)

The listen module reports up to 200 listening TCP sockets with their process names. GET /v1/projects/{p}/integrations/detected matches them against every manifest's discovery (a process-name match wins over a port match) and answers rows such as "postgres on :5432 on host X", each with the remote config that switches the module on and the local config the agent file needs. Credentials are filled in on the host.

Alert packs (INF-029)

POST /v1/projects/{p}/integrations/{name}:install-pack creates one alert rule per pack alert, labelled pack=<name>, enabled as default_enabled says, with the rationale and runbook in the annotations. Packs are copy-to-edit: the install remembers the hash of each rule it wrote, so running it again (the pack update) rewrites rules still as the pack left them and skips any a person edited (outcome skipped_edited). Deleting a rule and installing again recreates it.

Trigger dependencies (INF-030)

depends_on becomes inhibition rules owned by the pack, evaluated by the router's ordinary inhibition (see docs/product/notification-delivery.md). The host pack has an alert "Host down" labelled down="true": hosts that sent the agent heartbeat in the last 7 days but not in the last five minutes. A host silent for more than 7 days resolves as stale, and the inhibition of its alerts ends with it. Installing a pack with depends_on: [{parent_kind: host, equal_labels: [host_name]}] adds the rule "source pack=host, down=true; target pack=<name>; equal host_name". Installing the host pack also adds a rule that holds back every other alert carrying the downed host's host_name, so container and service alerts of that host go quiet without a pack of their own. A re-install replaces the pack's managed rules; the N2 routes answer 409 managed by pack <name>; uninstall or re-install the pack for PUT and DELETE on them (managed_by shows in the list), unless an org admin passes ?force=true; a forced change detaches the rule from the pack.

The listen module needs a local opt-in (modules.listen), so detection is empty until it is set. For a product the agent reaches over the network, host_name is the agent's host, not the product's, so the host-down inhibition follows the agent's host. Several http_status targets in one file are written by hand into the one targets list; the detected guidance shows one target to merge in.