Azure Monitor (Event Hubs)

Receive Azure resource logs, activity logs and platform metrics by streaming them from Azure Monitor into an Event Hub and reading that hub with the azure_eventhub module of wt-agent. The agent connects to the hub's Kafka-compatible endpoint (port 9093, TLS, SASL PLAIN), so no Azure SDK or Function is involved. Needs an ingestion key with ingest:logs and ingest:metrics for the agent (see the agent guide).

Azure resource ──diagnostic setting──▶ Event Hub ──Kafka 9093──▶ wt-agent ──OTLP/HTTP──▶ Watchtower

1. Create the Event Hub and a listen-only rule

Use a Standard (or higher) namespace: the Basic tier has no Kafka endpoint. Give the agent its own consumer group and a policy that can only listen.

az eventhubs namespace create --resource-group obs --name wt-obs-ns --location westeurope --sku Standard
az eventhubs eventhub create --resource-group obs --namespace-name wt-obs-ns --name diagnostics --partition-count 4 --cleanup-policy Delete --retention-time-in-hours 24
az eventhubs eventhub consumer-group create --resource-group obs --namespace-name wt-obs-ns --eventhub-name diagnostics --name wt-agent
az eventhubs namespace authorization-rule create --resource-group obs --namespace-name wt-obs-ns --name wt-agent-listen --rights Listen
az eventhubs namespace authorization-rule create --resource-group obs --namespace-name wt-obs-ns --name azmon-send --rights Send
az eventhubs namespace authorization-rule keys list --resource-group obs --namespace-name wt-obs-ns --name wt-agent-listen --query primaryConnectionString --output tsv

The last command prints the connection string for the agent. It is a secret: put it in the agent's environment (AZURE_EVENTHUB_CS below), never in the config file or in a ticket.

2. Send Azure Monitor data to the hub (diagnostic settings)

Each resource that should report gets a diagnostic setting that streams its log categories and AllMetrics to the hub. The activity log (subscription-level control plane events) uses a subscription diagnostic setting.

az monitor diagnostic-settings create --name to-watchtower --resource /subscriptions/<sub>/resourceGroups/app/providers/Microsoft.Web/sites/checkout --event-hub diagnostics --event-hub-rule /subscriptions/<sub>/resourceGroups/obs/providers/Microsoft.EventHub/namespaces/wt-obs-ns/authorizationRules/azmon-send --logs '[{"categoryGroup":"allLogs","enabled":true}]' --metrics '[{"category":"AllMetrics","enabled":true}]'
az monitor diagnostic-settings subscription create --name to-watchtower --location westeurope --event-hub-name diagnostics --event-hub-auth-rule /subscriptions/<sub>/resourceGroups/obs/providers/Microsoft.EventHub/namespaces/wt-obs-ns/authorizationRules/azmon-send --logs '[{"category":"Administrative","enabled":true},{"category":"ServiceHealth","enabled":true},{"category":"Alert","enabled":true}]'

To cover a whole subscription at once, assign the built-in Azure Policy initiative "Enable allLogs category group resource logging for supported resources to Event Hub" with the same hub and rule: it creates the setting on every existing and new resource.

3. Configure wt-agent

Add the module to /etc/wt-agent/agent.yaml (Windows: %ProgramData%\Watchtower\agent\agent.yaml). One agent per consumer group is enough; for more throughput run more agents with the same group, and Event Hubs spreads the partitions across them.

endpoint: https://ingest.example.com
token_file: /etc/wt-agent/token
modules:
  azure_eventhub:
    topics: [diagnostics]
    connection_string_env: AZURE_EVENTHUB_CS
    group: wt-agent
    start_at_earliest: false

brokers defaults to <namespace>.servicebus.windows.net:9093, taken from the connection string's Endpoint. start_at_earliest: true makes a new consumer group read the hub's whole retention once instead of starting at now. Offsets are committed only after the records were exported, so delivery is at-least-once: a restart can resend the last batch.

What arrives

Every record carries cloud.provider=azure and cloud.resource_id (lower-cased), so it attaches to the resource's cloud entity in the infrastructure inventory. Do not set wt.* attributes; the gateway deletes them.

Azure Monitor Agent (AMA) on VMs

AMA cannot send to a non-Azure endpoint directly. Two options:

1. Run wt-agent on the VM instead of (or next to) AMA. It collects host metrics, logs, journald and Windows Event Log itself; see the agent guide.

2. Keep AMA and route its data to the hub: send AMA to a Log Analytics workspace with a data collection rule, then add a workspace data export rule to the same Event Hub (az monitor log-analytics workspace data-export create --resource-group obs --workspace-name wt-law --name to-watchtower --tables Syslog Event Perf --destination /subscriptions/<sub>/resourceGroups/obs/providers/Microsoft.EventHub/namespaces/wt-obs-ns). Data export writes one hub per table (am-<table>), so list those hubs under topics.

Troubleshooting

• need a connection string: the variable named by connection_string_env is empty in the agent's environment (for systemd, set it with Environment= or an EnvironmentFile= in a drop-in).

• Nothing arrives, no error: the diagnostic setting points at another hub, or the consumer group already committed offsets past the data. Check the hub's "Incoming messages" chart in the portal.

• SASL authentication failed: the policy lacks Listen, or the connection string is the hub-level string of a different hub.

Status: structure-checked (the config blocks above are parsed by go/gateway/docs_config_test.go), and the record mapping is covered by the agent's cloud package tests. Not yet run against a live Azure subscription.