Data Jobs (OpenLineage + Spark/Databricks)

Watchtower monitors data pipelines (Airflow, Spark, dbt, Dagster) through two paths that land in the same data_job_runs_v1 table and surface in GET /v1/projects/{prj}/data-jobs and the data_job_failed / data_job_duration alert kinds.

1. OpenLineage HTTP (Airflow, dbt, Dagster, Spark's OL listener)

Point any OpenLineage producer at POST /v1/projects/{prj}/lineage with a project-scoped API key (the Authorization header is the tenant; wt.* fields in the body are ignored). Each OpenLineage RunEvent is stored as one row.

Mapping (crates/wt-api-datajobs/src/lineage.rs):

Airflow: set AIRFLOW__OPENLINEAGE__TRANSPORT='{"type":"http","url":"https://<wt>","endpoint":"/v1/projects/<prj>/lineage","auth":{"type":"api_key","apiKey":"<key>"}}' (OpenLineage provider ≥ 1.0). A DAG then appears in /data-jobs with its tasks, and a failed task fires data_job_failed carrying the DAG id and run id.

2. Spark / Databricks OTLP spans (native)

For Spark jobs we also accept OTLP spans from a listener, so a job shows up even without the OL integration and its stages/tasks nest under the application trace.

Spans (OTel span per unit, service.name = spark.app.name):

• spark.application — one per SparkContext; attributes spark.app.id, spark.app.name.

• spark.job — child of the application; spark.job.id, spark.job.description, spark.job.result (succeeded/failed).

• spark.stage — child of the job; spark.stage.id, spark.stage.attempt, spark.stage.num_tasks, spark.stage.failure (set on failure).

• spark.task — child of the stage (sampled); spark.task.id, spark.task.executor_id, spark.task.failed.

Ingest mapping: a spark.job span becomes a data_job_runs_v1 row with platform = spark, job = spark.app.name, run_id = spark.app.id, task = spark.job.id, status from spark.job.result, trace_id = the span's trace id (so the data-job row links to the full trace), and error from the failed stage's spark.stage.failure. Stage and task spans stay as spans under the same trace.

Databricks init script (cluster-scoped)

#!/bin/bash
# /databricks/scripts/wt-spark-otlp.sh — add the Watchtower Spark listener + OTLP exporter.
cp /dbfs/FileStore/wt/wt-spark-listener.jar /databricks/jars/
cat >> /databricks/driver/conf/wt.conf <<'EOF'
spark.extraListeners com.watchtower.spark.OtlpSparkListener
spark.watchtower.otlp.endpoint https://<wt-collector>:4317
spark.watchtower.otlp.headers  authorization=Bearer <project-key>
EOF

shortcut: the wt-spark-listener.jar itself (a SparkListener that maps the events above to OTLP spans) is a follow-up; this doc specifies the span schema and the ingest mapping so the jar and the ingest side can be built independently. Until the jar ships, use the OpenLineage Spark listener (path 1), which Databricks also supports via the same init-script mechanism.