Instrumentation coverage

Embed
Preview of the card
<a href="https://watchtowerops.net/docs/instrumentation-coverage"><img src="https://watchtowerops.net/docs/_api/public/orgs/aa5959cb-1a0b-4a9a-bc9a-913e7ac31cd9/spaces/b7aac181-9916-4805-b253-becc5fd2ca62/pages/2604c272-8919-45a4-9c8c-61519171c451/social-card.svg?v=3e7ae33eec8b" alt="Instrumentation coverage" width="600" height="315"></a>

GET /v1/projects/{prj}/instrumentation/coverage scores each service 0 to 100 and lists the next steps that would raise it (console: Instrumentation > Coverage).

SDK freshness

sdk.latest comes from a catalog the worker syncs every 24 hours from the package registries: npm (@opentelemetry/core, for nodejs and webjs), PyPI (opentelemetry-sdk), Maven Central (io.opentelemetry:opentelemetry-sdk), the Go module proxy (go.opentelemetry.io/otel/sdk), NuGet (OpenTelemetry), RubyGems (opentelemetry-sdk), Packagist (open-telemetry/sdk), crates.io (opentelemetry_sdk) and Hex (opentelemetry). Only stable releases count. The rows live in the global sdk_releases table; a registry that fails keeps its last row.

Servers without egress (or with WT_SDK_RELEASE_SYNC=off) use the built-in table compiled into the release (crates/wt-query/src/coverage_sdk.rs, LATEST), which also covers SDKs no registry describes (Swift, C++, the Watchtower game and device SDKs). The sync uses the worker's guarded client and contacts only the registry hosts above over https.

Agent integrations

Each service lists agent_integrations: what the host agents report through the wt.integration.up gauge (host-level integrations, plus status-page integrations tagged with the service), with up, last_seen_ms and a kind (database, cache, queue) for datastores.

  • datastore_spans: a service with SERVER spans but no client spans carrying db.system or messaging.system is a gap when the agents report a datastore integration for it (the step names them: "The agent reports postgres, redis here, but no span reaches them") or when the project has no agent integrations at all. A project whose agents report integrations but no datastore gets no such step.

  • agent_integration: an integration whose newest check failed is a step, naming it.

The service map draws the same datastores as dashed "expected but unseen" nodes (see the service map).

Updated