Object Storage Monitoring

Watchtower turns an S3 Inventory report into a daily, prefix-level view of what a bucket holds: bytes and object counts per prefix, the mix of storage classes, how old the data is, an estimated monthly bill, and a recommendation (with projected savings) for cold data that should move to a cheaper class. This is the Datadog-parity item D6.

It builds on what the platform already has: the AWS agent pulls CloudWatch bucket metrics (go/agent/cloud/cloudwatch*.go), and billing exports arrive through cost-sources. D6 adds the missing pieces: S3 Inventory ingest and the prefix breakdown, joined to real S3 spend.

How it works

1. You register a storage source that points at an S3 Inventory report and a cross-account IAM role the daily worker assumes to read it.

2. Once a day the worker reads the inventory manifest.json, parses its CSV data files, and rolls the objects up by bucket and key prefix to depth 3 (prefix "" with depth 0 is the whole bucket). Each prefix records its bytes, object count, bytes per storage class, and the median object age. One snapshot per day lands in ClickHouse (storage_inventory_v1).

3. You read the breakdown per bucket. Each prefix carries an estimated monthly bill. When the project has S3 billing lines from cost-sources, the real month-to-date spend is apportioned across prefixes by their share of the estimated bill; otherwise the estimate uses list prices.

4. For prefixes with a lot of aged Standard data, the response recommends a transition (Standard-IA, Glacier Instant Retrieval, or Intelligent-Tiering) and the projected saving.

Register a source

POST /v1/projects/{prj}/storage-sources
{
  "provider": "aws",
  "inventory_bucket": "my-inventory-bucket",
  "prefix": "reports/daily",
  "role_arn": "arn:aws:iam::123456789012:role/WatchtowerInventoryRead",
  "region": "us-east-1"
}

Needs manage_credentials. No secrets are stored or returned — access is by the role_arn your account trusts. GET lists sources (cursor pagination ?limit=&cursor=), DELETE /v1/projects/{prj}/storage-sources/{id} removes one (ingested snapshots stay until their retention ends). A source the project cannot see returns 404, never 403. A project may hold at most 20 storage sources.

Read the prefix breakdown

GET /v1/projects/{prj}/storage/prefixes?bucket=my-data&depth=3

Returns the newest snapshot's prefixes (biggest first), each:

{
  "prefix": "logs/2025/",
  "bytes": 734003200,
  "objects": 1240,
  "by_class": {"STANDARD": 734003200},
  "p50_age_days": 190.0,
  "est_monthly_usd": 15.7,
  "recommendation": "transition_to_glacier_ir",
  "est_savings_usd": 12.9
}

depth (0–3, default 3) is the deepest prefix level returned. Needs read_telemetry.

Alert on growth

The storage_growth rule kind fires on the percent change in a bucket or prefix's stored bytes over a number of days, one group per bucket and prefix:

{"storage_growth": {"days": 7}}

Pair it with a threshold (e.g. gt 25 for 25% weekly growth). Omit bucket to watch every bucket the project has a source for; set it to watch one. A prefix with no baseline snapshot in the window is not a group.

Data and retention

• ClickHouse table storage_inventory_v1 (migration 0126), one row per (project, source, bucket, prefix, snapshot_date); re-running a day replaces that day's rows. Snapshots are kept 400 days.

• Storage sources live in Postgres (storage_sources).

Limits

• CSV inventory formats only for now; ORC/Parquet inventory is not yet read.

• Estimated bills use hard-coded us-east-1 list prices when no real S3 spend is available; the join to cost-sources makes the numbers accurate once billing exports are configured.

• Prefix roll-up stops at depth 3.