Authoring an agent integration

Watchtower ships 113 integration manifests (integrations/<name>/manifest.yaml), up from the 32 the early spec counted: databases, queues, job frameworks, web and application servers, service meshes, Kubernetes add-ons, CDNs, SaaS status pages and host checks. This page is how to add one.

The manifest

Shape: integrations/schema.json; the agent (go/agent/integrations/loader.go) and the API (crates/wt-api-platform-sources) both refuse unknown keys.

• discovery: process names, ports and Kubernetes labels that make the API suggest the integration. Leave ports empty when the port is a common one (8080) or the product is SaaS.

• module: the agent module (kind) and its local config_defaults (credentials are placeholders).

• metrics: every series the module emits, with unit and description.

• dashboards: dashboard spec v1. alerts: stock monitors with a threshold_rationale; installed with POST /v1/projects/{prj}/integrations/{name}:install-pack.

• depends_on: the host pack, so a down host silences the product's alerts.

• logs (optional): paths (file globs, or pseudo sources like eventlog://Application) and a pipeline (pipeline config v1, stages parse_json|parse_grok|parse_regex|parse_kv|remap|enrich|redact|filter). The pipeline applies to records the agent tags agent.integration=<name>; every stage should keep on_fail: keep so an unparsable line passes unchanged. Installing the pack (:install-pack) or enabling the integration on an agent (PUT /agents/{id}/integrations/{name}) merges the pipeline into the project's logs pipeline (created if absent) as a new activated version: a v1 pipeline gets the stages guarded by attrs.agent.integration == "<name>", a v2 pipeline one group integration:<name> with that filter; running it again replaces only what it owns. Needs manage_pipelines (else outcome skipped_no_permission). Enabling also puts the name in the agent's remote overlay (logs.integrations); the agent maps it to the manifest's file paths compiled into it (go/agent/integrations/logpaths_gen.go, regenerate with scripts/gen-logpaths.py) and tails them with the label agent.integration=<name>. Remote config never carries a path. For agents without fleet remote config, the install response's logs.agent_config is the logs.files entry to put in the agent's own file.

Choosing a collector

1. Product exposes Prometheus text: add a row to exporterPackSpecs in go/agent/integrations/platform_packs_more.go (prefixes to forward, path, optional rename for unprefixed names). Or, with no Go at all, use kind: prometheus with a targets list.

2. Java server with Jolokia: add a jolokiaPacks entry (MBean reads and an attribute-to-metric table).

3. StatsD/DogStatsD emitter: kind: dogstatsd (the agent's listener aggregates it).

4. Anything else: a collector file in go/agent/integrations/ using Loop (reports wt.integration.up), registered through PackCollectors, with a fixture test.

The kit

wtctl integration new acme                       # scaffold integrations/acme/manifest.yaml + fixture.txt
wtctl integration validate integrations/acme     # schema, config defaults, logs pipeline (also a root dir)
wtctl integration validate integrations --strict # undocumented metrics in queries become errors
wtctl integration test integrations/acme --fixture payload.txt

test replays a recorded payload (Prometheus text, a Jolokia response, a status document) through the manifest's collector against a local server and fails when a documented metric is not emitted or the collector does not run clean. Kinds that need a connection (databases) are tested with a Go fixture test next to their collector instead.

Checklist for a new integration: manifest with logs where the product logs; add the name to v1Set in loader_test.go and to SOURCES in crates/wt-api-platform-sources/src/domain/product/integrations.rs; a fixture test.

What enabling installs

PUT /v1/projects/{prj}/agents/{id}/integrations/{name} switches the module on and, when the caller may manage alerts and edit dashboards, also installs the manifest's recommended monitors (rules labelled pack=<name>, the same path as POST .../integrations/{name}:install-pack) and its dashboard (titled <name> integration; skipped when a dashboard of that title exists). The response's pack field says what was created.

Testing multi-endpoint and tool-driven kinds

wtctl integration test <dir> --fixture <dir>/fixtures takes a fixture directory instead of one file. A local server answers each request from the file named METHOD_path (slashes as underscores, any extension; POST_root.json for a POST to /), and the collector's url (or endpoint, for cloud pulls) is pointed at it. A request with no fixture fails the test. The gpu kind reads nvidia-smi.csv and nvidia-smi-ecc.csv from the directory. Snowflake, databricks, vercel, cloudflare and dcgm ship their fixtures this way. The vercel, cloudflare and dcgm collectors report wt.integration.up, which their "unreachable" monitors read.