Serverless (AWS Lambda, Cloud Run, Azure Functions)

Explorer

GET /v1/projects/{prj}/serverless/functions?platform=&region=&range= lists every function with its invocations, errors, throttles, p95 duration, cold-start rate, p95 init duration and memory, unified across platforms. platform is one of aws_lambda, gcp_cloud_run, azure_functions (exact match); omit it for all. Results are cursor-paginated by name (?limit=&cursor= -> {data, next_cursor}).

GET /v1/projects/{prj}/serverless/functions/{name} returns one function's cold-vs-warm series by minute plus the links to its traces (service.name:{name}), logs (faas.name:{name}) and the cold-start filter. A function with no data in the range is a 404.

Counters and latency come from the metrics tables: the CloudWatch amazonaws.com/AWS/Lambda/* series (counters use the Sum stat, duration the Average), the GCP Cloud Run metrics (cloud_run preset) and the Azure Functions metrics (azure_functions preset), all keyed by faas.name or the Lambda FunctionName dimension. The cold-start rate and the init-duration proxy (the p95 total duration of cold spans) come from the OTel Lambda layer's spans, since the metric rollups have no faas.coldstart dimension. faas.invoke_duration/faas.init_duration are histograms and are not yet read for the p95 latency columns.

Azure Functions

Add the azure_functions preset to the Azure Monitor pull (cloud-pull.md) to collect FunctionExecutionCount, FunctionExecutionUnits and Http5xx. The execution count and 5xx count are also emitted as faas.invocations and faas.errors with cloud.platform=azure_functions, so the function appears in the explorer.

modules:
  azure_monitor:
    presets: ["azure_functions"]
    discover: { resource_types: ["microsoft.web/sites"] }

Metrics

Lambda metrics (Invocations, Errors, Duration, Throttles, ...) arrive through the CloudWatch Metric Stream and the Firehose endpoint described in cloud.md. Include the AWS/Lambda namespace in the stream.

Traces and logs

Use the OpenTelemetry Lambda layer for your runtime and point it at the gateway:

Environment:
  Variables:
    AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
    OTEL_EXPORTER_OTLP_ENDPOINT: https://gateway.example.com:4318
    OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
    OTEL_EXPORTER_OTLP_HEADERS: x-watchtower-key=wti_...
    OTEL_SERVICE_NAME: checkout-fn

Keep the key in Secrets Manager or SSM and inject it at deploy time rather than writing it in the template. Logs go through a CloudWatch Logs subscription into the same Firehose stream.

Step Functions

The agent's aws_stepfunctions module turns finished executions of standard state machines into traces: one root span per execution (status, error and cause of a failed run), a span per state from its StateEntered to its StateExited event, nested under the Parallel or Map state that encloses it, and a client span per task, Lambda or activity call. Pairing follows the history's previousEventId, so concurrent branches and Map iterations nest correctly. The service name is the state machine's name. It reads ListStateMachines, ListExecutions and GetExecutionHistory (without execution input or output) with the standard AWS credential chain, so the role needs states:ListStateMachines, states:ListExecutions and states:GetExecutionHistory. It is off unless enabled: true and a region are set; calls fail (and are logged once) without credentials. Express workflows have no execution history in this API and are skipped.

modules:
  aws_stepfunctions:
    enabled: true
    region: eu-west-1
    interval: 1m              # poll interval (at least 30s); executions that stopped within two intervals are picked up
    max_executions: 50        # per state machine and poll (at most 200)
    # state_machines: ["arn:aws:states:eu-west-1:123456789012:stateMachine:orders"]   # default: all standard ones (at most 100)

Cold starts

The layer sets faas.coldstart=true on the first invocation span of an execution environment. The RED view and the service page take a cold-start filter: GET /v1/projects/{prj}/apm/services/{service}/red?cold_start=true shows only cold invocations, cold_start=false only warm ones. Omit it for both. The same settings as a JSON environment block (for example a Terraform environment map or a SAM parameter file):

{
  "AWS_LAMBDA_EXEC_WRAPPER": "/opt/otel-instrument",
  "OTEL_EXPORTER_OTLP_ENDPOINT": "https://gateway.example.com:4318",
  "OTEL_SERVICE_NAME": "checkout-fn"
}

And as a dotenv file for local runs of the function:

OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_PROPAGATORS=tracecontext,baggage
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.5