Managed LLM evaluators

Every project gets six evaluators and an online rule for each (migration 0355; new projects through insert_defaults): managed_prompt_injection and managed_pii_leak (guards, free, on) and managed_faithfulness, managed_toxicity, managed_topic_adherence, managed_failure_to_answer (judges, off by default, 10% sample, 500 jobs a day). The Online rules tab lists them with their state.

Model key

Judges call a model. They run with the organization's provider key (AI settings) or, failing that, ANTHROPIC_API_KEY on wt-evald. With neither:

• GET /v1/projects/{prj}/llm/managed-evaluators returns judge: "no_model_key" and the UI shows a banner; judge rows read blocked_no_model_key; guards keep running.

• Enabling a judge rule (PATCH .../eval-rules/{id}) and POST .../llm/judges/generate answer 503 no_model_key.

• POST .../llm/managed-evaluators/install restores any managed evaluator that was deleted.

The API asks wt-evald (WT_EVALD_URL); if that is unset the state is unknown and nothing is blocked.

Images and audio

A judge spec may set "media": ["image"] and/or ["audio"]. Captured parts (wt-media:<sha256> in the payload) are fetched with the job's key and attached, up to four. Images go to Anthropic and OpenAI-compatible vision models; audio only to OpenAI-compatible audio models (wav, mp3). Templates: image_relevance, image_unsafe_content, audio_response_quality. A full-mode payload needs a key allowed to read media.

Agent frameworks

Python: instrument_crewai, instrument_bedrock_agents, and LangGraph nodes through the LangChain callback (watchtower.integrations.genai); OpenAI Agents SDK through genai_agents.

TypeScript (@watchtowerobservability/node, .../genai): langchainCallbackHandler() for LangChain.js and LangGraph.js (graph and nodes become invoke_agent spans, models chat, tools execute_tool), instrumentVercelAI({ generateText, streamText }) for the Vercel AI SDK agent loop (a chat span per step, execute_tool per tool call), instrumentMastra(agent), plus instrumentOpenAIAgents and instrumentLlamaIndex. CrewAI exists only for Python, so there is no CrewAI hook in the TypeScript SDK.

Open

• Live eval numbers (G-54) need ANTHROPIC_API_KEY from the owner: run make eval once.

• Price rows (G-55), checked 2026-11-08 against machine-readable vendor sources (migration 0450, infra/data/model_prices.json). Of the 60 rows that were unverified: 26 are now verified (Azure OpenAI 8 via the Azure Retail Prices API, Bedrock 8 via the AWS Price List, Vertex 4, Cohere 3, Mistral 2, OpenAI 1), 22 are retired models closed with effective_to (4 Claude 3.x, 9 Mistral, 3 Gemini, 6 DeepSeek legacy names; their old prices are not re-verifiable), and 12 stay unverified: Bedrock Llama 3.1 405B (only batch pricing is published) and the 4 Bedrock Claude rows (not in the Bedrock price list), Cohere embeds (no per-token price published), and 5 xAI rows (the docs now list only grok-4.3 and later; grok-code-fast-1 now reads $1/$2, not our $0.20/$1.50). Catalog now: 102 rows, 67 verified, 23 closed and unverified, 12 open (plus 1 verified and closed: gpt-5-chat-latest).