Recommended disaster-recovery settings

Embed
Preview of the card
<a href="https://watchtowerops.net/docs/recommended-dr-settings"><img src="https://watchtowerops.net/docs/_api/public/orgs/aa5959cb-1a0b-4a9a-bc9a-913e7ac31cd9/spaces/b7aac181-9916-4805-b253-becc5fd2ca62/pages/037576b7-c3de-4001-81c6-2e2b6789d232/social-card.svg?v=3e7ae33eec8b" alt="Recommended disaster-recovery settings" width="600" height="315"></a>

Who does what. The operator of the platform backs up and restores the control plane (Postgres) and the telemetry store (ClickHouse); see the backup policy. You cannot configure those. What you control is how long your data lives, and whether a copy exists outside the platform.

Objectives (platform backups)

Store

RPO

RTO

Control plane (projects, dashboards, alert rules, members)

up to 24 h

minutes to an hour

Telemetry (logs, traces, metrics)

last backup, then Kafka retention (24 h) refills the gap

restore plus replay

Backups are daily and kept 35 days. Data you delete or erase stays in backups until they expire (at most 36 days). Delivery is at-least-once; a replay after a restore is deduplicated.

What you configure

Setting

Recommendation

Where

Retention

At least 30 days for logs, traces and metrics, longer for anything you must investigate after the fact. Changes apply to new data only.

Settings, Retention and archive

Archive to your own bucket

Turn on for logs and traces. Closed days are copied as Parquet to your S3-compatible bucket before retention removes them. Use a bucket in a different region from the platform, with versioning on.

Settings, Retention and archive (PUT /v1/projects/{prj}/archive/{signal})

Scheduled export

One schedule per signal you cannot lose (S3, GCS, Azure Blob, Iceberg, BigQuery or Snowflake). A destination you own is your independent copy and your cross-region copy.

Settings, Export schedules (/v1/projects/{prj}/export-schedules)

Data plane

Projects are pinned to a data plane at creation. For regional failover, keep a second project on a plane in another region and export to it; moving a project is create, export, import.

Settings, Organization

As code

Keep dashboards, alert rules and monitors in Terraform or wtctl manifests so the control plane can be rebuilt from your repo.

as code

Your own RPO for telemetry is the export schedule interval (hourly by default) plus the archive delay. Your own RTO is the time to point readers at the archive or re-import it. Credentials for archive and export are write-only: store them in your secret manager too.

What the operator runs

Daily Postgres and ClickHouse backups, encryption with the backup key, 35-day pruning, post-backup verification and the restore drill (runbook). Ask your operator for the date of the last tested restore.

The Settings page shows your current values against this table under Retention and archive.

Updated