DeepEval
Pytest-style unit testing for LLM apps, 50+ metrics, Apache 2.0
Overview
DeepEval is an open-source (Apache-2.0) Python framework for unit-testing LLM applications with 50+ research-backed metrics, custom LLM-as-judge metrics, synthetic data and agent evaluation, from Confident AI, whose cloud platform adds tracing, datasets and online evals. Library free; Confident AI from $200 a month per organisation.
Pricing
Pricing shown for reference only. These figures reflect RECATOOLS research as of 1 Oct 2026 and may be out of date or incomplete. This is not financial or purchasing advice — always confirm the current price on the provider’s official website before making any decision.
- Unit and regression testing in dev and CI
- LLM tracing and prompt versioning
- 2 seats, 1 project
- 5 test runs a week
- Unlimited seats, 5 projects
- No-code eval workflows and custom metrics
- Online evals on live traffic
- Annotation queues and alerting
- Unlimited projects
- Metric and dataset versioning
- Custom RBAC, SSO and SOC 2
- Dedicated support channel
- On-premises deployment
- Custom data residency incl. Japan and Australia
- HIPAA
- 24x7 support
What you can produce with DeepEval
- Open-source LLM evaluation framework (Apache-2.0)
- 50+ research-backed metrics
- Custom LLM-as-judge metrics
- Synthetic test-data generation
- Confident AI tracing, datasets and online evals
- DeepTeam red-teaming framework
DeepEval is one of the most complete open-source toolkits for testing LLM applications, and its unit-test style fits naturally into engineering workflows. The library is free, but Confident AI's cloud plans now jump from free to $200 and then $2,000 a month per organisation, a big step for small teams. Its adoption figures are its own. An excellent choice for developers adding evaluation to CI; teams needing hosted dashboards should compare cloud costs first.
What we found
DeepEval is an open-source Python framework for testing and evaluating large-language-model applications, written in the style of unit tests so that checks run in development and in CI/CD pipelines. It offers more than 50 research-backed metrics — answer relevance, faithfulness, hallucination, bias, toxicity, agent task completion and multi-turn conversation quality among them — plus custom metrics built with LLM-as-judge methods, synthetic test-data generation and agent evaluation. It is licensed under Apache 2.0 and released frequently, most recently in late September 2026, and its maker also publishes DeepTeam, an open-source red-teaming framework. DeepEval says it is used by more than 150,000 developers and by over half the Fortune 500, and its site shows logos including Microsoft, AXA, Google, NVIDIA and Mastercard; those are its own claims.
DeepEval is made by Confident AI, Inc., a San Francisco company founded by Jeffrey Ip, its chief executive, and Kritin Vongthongsri, its technical chief, which went through Y Combinator's winter 2025 batch. It raised a $2.2 million seed round from investors including YC, Flex Capital and the Singapore venture firm January Capital. Its commercial platform, Confident AI, adds a cloud dashboard for test runs, tracing of LLM calls, prompt versioning, datasets, online evaluation of live traffic, annotation queues and alerts; the company says its tracing costs at least three times less than alternatives.
The DeepEval library is free. Confident AI's free plan allows two seats, one project, five test runs a week and a gigabyte-month of trace data. Paid plans are priced per organisation, not per user: Starter costs $200 a month with unlimited seats, five projects, no-code workflows, online evals and alerts, and Team costs $2,000 a month with unlimited projects, versioning, RBAC, SSO and SOC 2, with extra trace storage at $1 per gigabyte-month. Enterprise adds on-premises deployment, HIPAA and data residency options including Japan and Australia.
We could not read independent review sites. DeepEval suits teams who want rigorous, code-first LLM testing.
Compiled by RECATOOLS from public sources: vendor documentation and pricing, code repositories and published reporting.
About this listing
This entry was compiled from publicly available data including DeepEval's official website, press releases, documentation, and reputable third-party publications. RECATOOLS is not affiliated with DeepEval unless explicitly stated.
Third-party AI tools update their pricing, features, availability, and policies frequently. Information here may be outdated by the time you read this — we make reasonable efforts to keep listings current, but cannot guarantee absolute accuracy.
For the latest details, please refer to DeepEval directly →
Spotted something out of date? Suggest an update →
Alternatives to DeepEval
More in Code & Dev Tools
