Remote Upgrades

<!-- status: beta language: java, python, node, dotnet, ruby, php -->

Remote SDK upgrades

Fleet > Packages > "Upgrade SDKs" applies an auto-instrumentation version to chosen services without touching their code. A job names a language and a target version; each service is one target, delivered by the Kubernetes operator or by a host agent. Status per target is queued, applying, done or failed (with the error). "Roll back" creates a new job that re-applies the version the language ran before (previous_version).

API (ManageFleet to write, ReadTelemetry to read): POST|GET /v1/projects/{prj}/instrumentation/sdk-upgrade-jobs, GET .../sdk-upgrade-jobs/{id}, POST .../sdk-upgrade-jobs/{id}/rollback. A newer job for the same service and language fails the older unfinished target as superseded.

Delivery uses the fleet channel (edge, agent:fleet key): GET /v1/sdk-upgrades?kind=kubernetes|host and POST /v1/sdk-upgrades/report.

Kubernetes

Enable operator.remoteUpgrades in the Helm chart (endpoint = the edge URL, keySecret = a Secret holding an agent:fleet key; adds a ClusterRole with get/list/patch on workloads). Every operator replica's webhook then injects the desired tag of the language's init-container image; the leader rolls each target's Deployments, StatefulSets and DaemonSets (matched by name or the app.kubernetes.io/name / app label) that opted into the language, by stamping the pod template with watchtower.dev/sdk-upgrade-<lang>: <version>. The target is done only when every such workload has observed the new generation, has all replicas updated and available; it is failed with the reason on ProgressDeadlineExceeded or when a rollout has not completed in 15 minutes.

Hosts

The agent polls every minute when its own config defines the autoinst module. It first reports the (language, service) pairs its rules cover; the server keeps one target per (service, host), so a service on N hosts is finished only when all N hosts have reported (a host that appears later gets the service's latest job too). If the requested version is not installed under <agents_dir>/versions/<version>/<lang>, the agent downloads the bundle registered for it (POST /v1/projects/{prj}/fleet/packages with version = autoinst-<lang>@<version>, a tar.gz of the language directory, signed like an agent package), verifies the Ed25519 release signature with the key compiled into the agent and the SHA-256, and unpacks it (no links, no paths outside the directory). Only a failed download or verification, or a failed switch, reports failed. The agent then switches the preload shim's per-language directory and stores the version in state_file (default /var/lib/watchtower/autoinst-versions.json). Running processes keep the old agent until they restart. Services are restarted only when the job asks for it AND the machine's own config sets allow_restart: true and names the rule's systemd unit:

modules:
  autoinst:
    enabled: true
    allow_restart: true
    rules:
      - { exe: /usr/bin/java, lang: java, service: shop, unit: shop.service }