Opik

Opik is an open-source LLM observability and evaluation platform built by Comet, offering AI agent tracing, LLM evaluation, prompt management, and production monitoring. It records every step of an agent, from user interactions to context retrieval and tool calls. It runs LLM-as-a-judge evaluations with more than 30 metrics, and evaluates production traces in real time, alerting you when they fall short of your criteria. The core features are free and open source, and can be downloaded from GitHub and run locally.

• Agent tracing and debugging (trace visualization)
• LLM-as-a-judge evaluation with more than 30 metrics
• Pass/fail testing with Test Suites and plain-text assertions
• Ollie coding assistant analyzes traces and implements code fixes
• Test entire agents with Agent Playground
• 6 prompt optimization algorithms (Prompt Optimizer)
• Production monitoring and guardrails (blocking PII exposure)
• Track coding agent costs with Cost Intelligence

• One of 27 AI tools in the Observability & Evals field.
• Connected to 5 other tools — #6 in its field.