Dynamic Probes

<!-- status: beta language: probes -->

Dynamic probes

Add a log line, a snapshot or a metric to running code without redeploying, from an issue's stack frame ("Add probe here") or the API. Points are compiled into the service; nothing patches running code. Runtime agents (see Clients) attach without compiled-in points.

Governance

• Off by default: PUT /v1/projects/{prj}/probe-settings {"enabled": true} (project admin).

• Create: POST /v1/projects/{prj}/probes with service, env, kind (log, snapshot, metric, span, span_decoration), location ({file, line} or {function}), condition, message, capture (locals_depth up to 3, max_bytes up to 8192), max_rate_per_min up to 60 and ttl_s up to 86400 (default 3600).

• span needs {function} and wraps each matching call in a span probe <function>, made current so the call's own spans are its children. span_decoration needs capture.tags (1 to 10 {name, value}; value is a template like message, wt.* names are refused) and sets them on the span active at the location.

• A probe made by an engineer in production (or with no environment) is pending until a project admin calls POST .../probes/{id}/approve.

• At most 25 pending or active probes per project. Expiry is enforced when serving, so a late janitor never serves a dead probe. Audit events: probe.create, probe.approve, probe.delete, probe.expire.

Conditions

Comparisons (==, !=, <, <=, >, >=), &&, ||, !, in [..], string contains. Function calls are rejected.

Delivery and snapshots

SDKs poll GET /v1/probes/active?service&env&version on the edge every 30 s with If-None-Match, using an ingestion key with scope probes:read. An agent lists the kinds it applies in kinds= (comma-separated); without it only log, snapshot and metric are served, so older agents never get a span probe. A hit emits probe.id, code.filepath, code.lineno, code.function and code.snapshot.* (values cut to 1 KiB) with the active trace and span ids. The worker redacts them with the project's pipeline before storage, applies the probe's max_rate_per_min again and drops snapshots of expired probes.

Clients

Go: dyninst.Point("Pay", func(v dyninst.Vars) { v["order"] = o.ID }); the closure runs only while a probe targets Pay (or file:line). Browser: wt.probe("pay", () => ({ n })) with probes: { key } in the config.

Python (watchtower.probes.start(...)) attaches every kind at runtime through sys.settrace, for functions and lines. PHP (Watchtower\Probes::start($endpoint, $key, $service) at bootstrap) attaches through the OpenTelemetry PHP extension's hook(): functions and methods (Class::method) only, because PHP has no line hooks, and the probe set is cached per host for 30 s so a request makes no API call.

Java (Probes.start(...), JVM started with a loopback JDWP agent) applies span and span_decoration through JDI: the span starts at the method's entry breakpoint and ends at its method-exit event, or with status error when an exception leaves the frame. JDI cannot run code in the target thread, so the agent reads and sets the suspended thread's OTel context directly (needs --add-opens java.base/java.lang=ALL-UNNAMED; without it spans are roots, not current, and decorations do nothing): the parent is the thread's context, and the probe span is current for the call (restored at its exit), so spans the method starts nest under it. .NET (ProbeSet.Start(...)) patches the method with Harmony: the span is an Activity of source Watchtower, current for the call and ended by a finalizer (error on throw); an async method's span ends when it returns its Task.

Node (createProbes(...)) applies span by wrapping the named export of a loaded CommonJS module (location.function is path/to/file.js:fn or fn): the span is current for the call, ends when the call returns or its promise settles, and gets error status on a throw or rejection; removing the probe restores the original function. ES module exports are read-only, so they cannot be wrapped yet. A span_decoration on a function wraps it the same way; one on file:line is judged in the inspector worker and applied with a fixed expression that only calls setAttributes on the active span. Ruby (Probes.start(...)) uses the same TracePoint: :call opens the span and makes it current, :return ends it (error status when the method exits by exception); span_decoration works on methods and lines.

Go has no runtime attach. A Go binary has no interpreter hook, and patching machine code (uprobes, a delve-style debugger) needs privileges (CAP_SYS_ADMIN/ptrace) that a shared agent should not ask for, and it breaks under inlining and the Go stack model. So Go probes fire only at compiled-in dyninst.Point() calls. Place points at the functions you may want to probe; they cost one atomic load while no probe targets them. For the span kinds, wrap the body in dyninst.Span(ctx, "Pay", func(ctx context.Context) error { ... }), which opens probe Pay only while a span probe targets Pay (its condition sees no variables), and use dyninst.PointContext(ctx, ...), which applies span_decoration probes to the span in ctx (plain Point has no span to decorate).