Trace sampling, exact RED and span-based metrics

Ingest and retain are separate layers

• The gateway counts every span as it arrives, before head sampling: per service and endpoint by hour (span_ingest_stats: ingested, kept, calls, errors) and per service by minute (span_ingest_minute). Calls and errors come from SERVER and CONSUMER spans, the same spans as the service_red_1m rollup.

• GET /v1/projects/{prj}/apm/ingest-stats returns ingested versus kept per service and endpoint (Settings, Tracing usage).

• The service RED chart (GET .../apm/services/{service}/red) replaces each bucket's request and error counts with the gateway's per-minute totals, so they are exact whatever is retained. Percentiles still come from the stored spans. The totals are kept per service and environment (span_ingest_minute, 32 days) and per endpoint (span_ingest_endpoint_minute, 8 days; the endpoint is http.route, else url.template, else the span name, as the worker stamps it by default). So the env_id and endpoint filters are exact too. Cold-start and flag filters have no counter and stay extrapolated, as do projects whose endpoint_rules rename endpoints or services (the gateway counts before those rules), and buckets older than the counters; extrapolated in the response and the chart badge say which.

• Counts are pruned by the janitor job span_stats_prune (32 days hourly and per minute, 8 days per endpoint).

Retention and ingestion reason

The gateway applies the sampling rules (Settings, Sampling); the worker's tail sampler keeps errors, slow traces, rare paths and, by default, K traces per service, endpoint, status class and latency bucket per window. Every kept span carries wt.ingestion_reason: head, rule, manual, error, rare, slow, policy or retention_filter. A span attribute sampling.priority (above 0 keeps, 0 drops) or watchtower.sampling.keep / .drop forces the decision; keep beats drop.

Sampling inside the SDK (remote sampling)

Off by default in every SDK. Turn it on with WT_REMOTE_SAMPLING=1 (or the option: Node remoteSampling, Python remote_sampling, Go Options.RemoteSampling, Java Options.remoteSampling or watchtower.remote-sampling, .NET the env var). The SDK polls GET /v1/sampling?service=&env= with an ETag every WT_SAMPLING_INTERVAL_S seconds (default 30), applies the project's head rules to root spans with ParentBased, and keeps the last good config when a poll fails. It decides by fnv64a of the trace id, the gateway's hash, so every service and the gateway agree on a trace.

Why off: a span dropped in the SDK never reaches the gateway, so its pre-sampling counts, and the exact RED above, see only what the SDK kept. Turn it on for cost (network and CPU in the process), not for exactness. Rules that need duration or status (tail decisions) are left to the gateway and worker.

Code origin

Opt-in (WT_CODE_ORIGIN=1, or Node codeOrigin, Python code_origin, Go Options.CodeOrigin, Java Options.codeOrigin / watchtower.code-origin). Each span gets code.filepath, code.lineno and code.function (Java and .NET also code.namespace) of the first application frame, skipping the OpenTelemetry and Watchtower packages. It walks the stack per span, so it costs CPU. The trace view links the file and line to the repository (Settings, Repo links).

Span-based metrics

Settings, Span-based metrics (or POST /v1/projects/{prj}/span-metrics): a filter (service, name prefix, errors only, attributes) and up to 5 group-by attributes become traces.span.custom.<name>.calls and .duration, kept 450 days by a metrics retention rule. The traces worker adds the definitions' metric stages to the project's traces pipeline (reloaded within its config TTL), so they are computed from every span that reaches the worker, and written like any derived metric. At most 10 definitions per project. The counts are of spans after the gateway's head sampling, so with sampling on they are estimates; use the RED chart for exact request counts.