Embedded Datastores

Embed
Preview of the card
<a href="https://watchtowerops.net/docs/embedded-datastores"><img src="https://watchtowerops.net/docs/_api/public/orgs/aa5959cb-1a0b-4a9a-bc9a-913e7ac31cd9/spaces/b7aac181-9916-4805-b253-becc5fd2ca62/pages/2fd89f4c-bef8-4baa-9914-b6caca76933d/social-card.svg?v=3e7ae33eec8b" alt="Embedded Datastores" width="600" height="315"></a>

<!-- status: beta language: embedded-datastores -->

Embedded and in-memory datastores (server SDKs)

Stores with no server (SQLite, LMDB, RocksDB, bbolt, ETS, caches) are monitored from inside the app. Every SDK uses one model, so the Datastores area treats them alike. Node is in node.md.

Model

  • Spans follow OTel db semconv: db.system.name, db.operation.name, db.namespace (file path or logical name), db.collection.name (bucket, table, tree). Keys, values and raw SQL are never recorded. SQL spans carry db.query.hash and db.query.summary (also the span name). db.query.text is opt-in and sanitized. The hash algorithm is in sql-normalization.md, and all SDKs run the shared vectors.

  • Metrics are wt.embedded.*, tagged {db.system.name, db.namespace} (the datastore entity, with the resource host/service). Counters are cumulative totals; the SDK exporters send deltas.

    • Gauges: file.size, wal.size, map.size, map.used, memory.used, cache.size (bytes), keys, compaction.pending (bytes), transactions.open, free_pages, levels.files, snapshots.open, iterators.open, readers.

    • Counters: cache.hits, cache.misses, cache.evictions, cache.rejections, lock.waits, lock.wait_time (ms), compactions, compaction.duration (ms), flushes, transactions, bytes.written, bytes.read.

    • A store reports only what it knows. Names outside the list and negative values are dropped.

Coverage

SDK

Spans

Metrics

Go

sql (any driver incl. mattn/modernc SQLite, DuckDB); separate module datastores: bbolt, badger, pebble, leveldb (goleveldb), buntdb

the same packages plus datastores/ristretto, sql.ObservePool, embedded.ObserveFiles

Python

sqlite3, aiosqlite, apsw, duckdb, lmdb, plyvel, rocksdict, tinydb

the same plus diskcache, cachetools, functools.lru_cache

Rust (feature datastores)

rusqlite (Connection::profile hook), any store via datastore::span

sled, redb, heed, moka, RocksDB (property closure), files

Java

upstream OTel JDBC (H2, HSQLDB, SQLite JDBC)

Caffeine, RocksDB JNI (property reader), files (H2, HSQLDB, MapDB)

.NET

-

MemoryCache, files (Microsoft.Data.Sqlite, LiteDB, DuckDB)

PHP

-

APCu, files (SQLite3, PDO sqlite)

Elixir

Mnesia transactions

ETS, DETS, Mnesia tables and node stats, Cachex

Ruby

sqlite3 (Watchtower::SQLite3.instrument)

-

Job frameworks: Celery (Python), Sidekiq (Ruby) and Oban (Elixir) already have SDK integrations over the upstream OTel instrumentations. River (Go) ships rivercontrib/otelriver upstream, so the SDKs add nothing there.

Updated