Test visibility and Test Impact Analysis

Watchtower stores every test result your CI uploads. From those results it shows flaky tests (with quarantine), each test's history, the slowest tests, failures grouped by message, and per-branch or per-PR views. With per-test coverage it also tells CI which tests a change can skip (Test Impact Analysis, TIA).

Everything below uses the project API with an API key (WT_API_KEY, sent as Authorization: Bearer). Uploading results and coverage needs a key that can manage the project. Reading needs read access.

1. Send results

All frameworks end up at POST /v1/projects/{prj}/test-runs. This is a multipart upload of JUnit XML with the fields run_id, branch, sha, optional suite and optional coverage_pct. It accepts up to 20 MiB, and posting the same file again changes nothing. Test ids are suite::classname.name (suite::name when there is no class). Use the same suite name everywhere so ids stay stable.

Common environment for the examples (GitHub Actions):

env:
  WT_API_URL: https://watchtower.example.com
  WT_API_KEY: ${{ secrets.WT_API_KEY }}
  WT_PROJECT_ID: prj_...
  WT_CI_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}

Upload in an if: always() step so that failing runs are recorded too.

2. Read results

Every read takes from/to (RFC 3339, default the last 7 days) and branch. Set branch to a PR's head branch for the PR view, or to main for the default-branch view. In the UI, the branch filter applies to every view.

Flaky tests can be quarantined (POST /tests/{id}/quarantine with a ticket and expiry). A quarantined test still runs, but its failures do not fail the build when you use wtctl ci junit --respect-quarantine, the Vitest/Jest test plan or the pytest plugin. Auto-quarantine and early flake detection are set in PUT /tests/flaky-policy.

3. Test Impact Analysis

TIA skips tests whose recorded files a change does not touch. It works in two steps.

Record per-test coverage (on the default branch)

Run the full suite with coverage recording, for example nightly or on every push to main. Each recording replaces the previous one per test.

Paths must be repository-relative, the same paths git diff --name-only prints. wtctl converts absolute and working-directory-relative paths for you.

Select tests in a PR

- uses: actions/checkout@v4
  with: { fetch-depth: 0 }
- run: wtctl tests select --base origin/${{ github.base_ref }} --api "$WT_API_URL" --project "$WT_PROJECT_ID" --format skip > skip.txt

POST /v1/projects/{prj}/tia/select with {"changed_files": [...], "tests": [...]} returns {run_all, reasons, run, skip, unknown_files}. Feed the skip list to your runner. --format go-skip prints a go test -skip regexp: go test -skip "$(wtctl tests select ... --format go-skip)" ./.... The Vitest/Jest test plan and the pytest plugin apply the same decisions themselves when WT_TEST_OPTIMIZATION=1 and WT_GIT_BASE are set.

Safety rules. When any of these applies, every test runs and skip is empty:

• The diff is empty or unknown.

• No per-test coverage has been recorded yet.

• A configuration, dependency or build file changed. This covers lockfiles, manifests (package.json, Cargo.toml, go.mod, pyproject.toml, ...), test-runner configuration (vitest.config.*, jest.config.*, conftest.py, pytest.ini, tsconfig*, ...), Makefile, Dockerfile, and .github/workflows/ and other CI configuration.

• A changed file is one that no recorded test executed, such as a new module, a fixture, a data file or a rename. Documentation (*.md, *.rst, LICENSE, ...) is the exception.

Tests that were never recorded always run. So do tests recorded with no files. TIA does not handle disabled or quarantined tests. Those decisions come from POST /tests/plan.

Delivery is at-least-once: uploading results or coverage again is safe.