DeepEval Overview DeepEval is a pytest-based framework for testing LLM applications. It provides 50+ evaluation metrics covering RAG pipelines, conversational AI, agents, safety, and custom criteria. DeepEval integrates into development workflows through pytest, supports multiple LLM providers, and includes component-level tracing with the @observe decorator. Repository: https://github.com/confident-ai/deepeval Documentation: https://deepeval.com Installation pip install -U deepeval Requires Python 3.9+. Quick Start Basic pytest test import pytest from deepeval import assert_test from deepeval . test_case import LLMTestCase from deepeval . metrics import AnswerRelevancyMetric def test_chatbot ( ) : metric = AnswerRelevancyMetric ( threshold = 0.7 , model = "athropic-claude-sonnet-4-5" ) test_case = LLMTestCase ( input = "What if these shoes don't fit?" , actual_output = "You have 30 days for full refund" ) assert_test ( test_case , [ metric ] ) Run with: deepeval test run test_chatbot.py Environment setup DeepEval automatically loads .env.local then .env :
.env
OPENAI_API_KEY
"sk-..." Core Workflows RAG Evaluation Evaluate both retrieval and generation phases: from deepeval . metrics import ( ContextualPrecisionMetric , ContextualRecallMetric , ContextualRelevancyMetric , AnswerRelevancyMetric , FaithfulnessMetric )
Retrieval metrics
contextual_precision
ContextualPrecisionMetric ( threshold = 0.7 ) contextual_recall = ContextualRecallMetric ( threshold = 0.7 ) contextual_relevancy = ContextualRelevancyMetric ( threshold = 0.7 )
Generation metrics
answer_relevancy
AnswerRelevancyMetric ( threshold = 0.7 ) faithfulness = FaithfulnessMetric ( threshold = 0.8 ) test_case = LLMTestCase ( input = "What are the side effects of aspirin?" , actual_output = "Common side effects include stomach upset and nausea." , expected_output = "Aspirin side effects include gastrointestinal issues." , retrieval_context = [ "Aspirin common side effects: stomach upset, nausea, vomiting." , "Serious aspirin side effects: gastrointestinal bleeding." , ] ) evaluate ( test_cases = [ test_case ] , metrics = [ contextual_precision , contextual_recall , contextual_relevancy , answer_relevancy , faithfulness ] ) Component-level tracing: from deepeval . tracing import observe , update_current_span @observe ( metrics = [ contextual_relevancy ] ) def retriever ( query : str ) : chunks = your_vector_db . search ( query ) update_current_span ( test_case = LLMTestCase ( input = query , retrieval_context = chunks ) ) return chunks @observe ( metrics = [ answer_relevancy , faithfulness ] ) def generator ( query : str , chunks : list ) : response = your_llm . generate ( query , chunks ) update_current_span ( test_case = LLMTestCase ( input = query , actual_output = response , retrieval_context = chunks ) ) return response @observe def rag_pipeline ( query : str ) : chunks = retriever ( query ) return generator ( query , chunks ) Conversational AI Evaluation Test multi-turn dialogues: from deepeval . test_case import Turn , ConversationalTestCase from deepeval . metrics import ( RoleAdherenceMetric , KnowledgeRetentionMetric , ConversationCompletenessMetric , TurnRelevancyMetric ) convo_test_case = ConversationalTestCase ( chatbot_role = "professional, empathetic medical assistant" , turns = [ Turn ( role = "user" , content = "I have a persistent cough" ) , Turn ( role = "assistant" , content = "How long have you had this cough?" ) , Turn ( role = "user" , content = "About a week now" ) , Turn ( role = "assistant" , content = "A week-long cough should be evaluated." ) ] ) metrics = [ RoleAdherenceMetric ( threshold = 0.7 ) , KnowledgeRetentionMetric ( threshold = 0.7 ) , ConversationCompletenessMetric ( threshold = 0.6 ) , TurnRelevancyMetric ( threshold = 0.7 ) ] evaluate ( test_cases = [ convo_test_case ] , metrics = metrics ) Agent Evaluation Test tool usage and task completion: from deepeval . test_case import ToolCall from deepeval . metrics import ( TaskCompletionMetric , ToolUseMetric , ArgumentCorrectnessMetric ) agent_test_case = ConversationalTestCase ( turns = [ Turn ( role = "user" , content = "When did Trump first raise tariffs?" ) , Turn ( role = "assistant" , content = "Let me search for that information." , tools_called = [ ToolCall ( name = "WebSearch" , arguments = { "query" : "Trump first raised tariffs year" } ) ] ) , Turn ( role = "assistant" , content = "Trump first raised tariffs in 2018." ) ] ) evaluate ( test_cases = [ agent_test_case ] , metrics = [ TaskCompletionMetric ( threshold = 0.7 ) , ToolUseMetric ( threshold = 0.7 ) , ArgumentCorrectnessMetric ( threshold = 0.7 ) ] ) Safety Evaluation Check for harmful content: from deepeval . metrics import ( ToxicityMetric , BiasMetric , PIILeakageMetric , HallucinationMetric ) def safety_gate ( output : str , input : str ) -
tuple [ bool , list ] : """Returns (passed, reasons) tuple""" test_case = LLMTestCase ( input = input , actual_output = output ) safety_metrics = [ ToxicityMetric ( threshold = 0.5 ) , BiasMetric ( threshold = 0.5 ) , PIILeakageMetric ( threshold = 0.5 ) ] failures = [ ] for metric in safety_metrics : metric . measure ( test_case ) if not metric . is_successful ( ) : failures . append ( f" { metric . name } : { metric . reason } " ) return len ( failures ) == 0 , failures Metric Selection Guide RAG Metrics Retrieval Phase: ContextualPrecisionMetric - Relevant chunks ranked higher than irrelevant ones ContextualRecallMetric - All necessary information retrieved ContextualRelevancyMetric - Retrieved chunks relevant to input Generation Phase: AnswerRelevancyMetric - Output addresses the input query FaithfulnessMetric - Output grounded in retrieval context Conversational Metrics TurnRelevancyMetric - Each turn relevant to conversation KnowledgeRetentionMetric - Information retained across turns ConversationCompletenessMetric - All aspects addressed RoleAdherenceMetric - Chatbot maintains assigned role TopicAdherenceMetric - Conversation stays on topic Agent Metrics TaskCompletionMetric - Task successfully completed ToolUseMetric - Correct tools selected ArgumentCorrectnessMetric - Tool arguments correct MCPUseMetric - MCP correctly used Safety Metrics ToxicityMetric - Harmful content detection BiasMetric - Biased outputs identification HallucinationMetric - Fabricated information PIILeakageMetric - Personal information leakage Custom Metrics G-Eval (LLM-based): from deepeval . metrics import GEval from deepeval . test_case import LLMTestCaseParams custom_metric = GEval ( name = "Professional Tone" , criteria = "Determine if response maintains professional, empathetic tone" , evaluation_params = [ LLMTestCaseParams . ACTUAL_OUTPUT ] , threshold = 0.7 , model = "anthropic-claude-sonnet-4-5" ) BaseMetric subclass: See references/custom_metrics.md for complete guide on creating custom metrics with BaseMetric subclassing and deterministic scorers (ROUGE, BLEU, BERTScore). Configuration LLM Provider Setup DeepEval supports OpenAI, Anthropic Claude, Google Gemini, AWS Bedrock, and 100+ providers via LiteLLM. Anthropic models are preferred. CLI configuration (global): deepeval set-azure-openai --openai-endpoint = .. . --openai-api-key = .. . --deployment-name = .. . deepeval set-ollama deepseek-r1:1.5b Python configuration (per-metric): from deepeval . models import AnthropicModel , OllamaModel anthropic_model = AnthropicModel ( model_id = settings . anthropic_model_id , client_args = { "api_key" : settings . anthropic_api_key } , temperature = settings . agent_temperature ) metric = AnswerRelevancyMetric ( model = anthropic_model ) See references/model_providers.md for complete provider configuration guide. Performance Optimisation Async mode is enabled by default. Configure with AsyncConfig and CacheConfig : from deepeval import evaluate , AsyncConfig , CacheConfig evaluate ( test_cases = [ . . . ] , metrics = [ . . . ] , async_config = AsyncConfig ( run_async = True , max_concurrent = 20 ,
Reduce if rate limited
throttle_value
0
Delay between test cases (seconds)
) , cache_config = CacheConfig ( use_cache = True ,
Read from cache
write_cache
True
Write to cache
) ) CLI parallelisation: deepeval test run -n 4 -c -i
4 processes, cached, ignore errors
Best practices: Limit to 5 metrics maximum (2-3 generic + 1-2 custom) Use the latest available Anthropic Claude Sonnet or Haiku models Reduce max_concurrent to 5 if hitting rate limits Use evaluate() function over individual measure() calls See references/async_performance.md for detailed performance optimisation guide. Dataset Management Loading datasets from deepeval . dataset import EvaluationDataset , Golden dataset = EvaluationDataset ( )
From CSV
dataset . add_goldens_from_csv_file ( file_path = "./test_data.csv" , input_col_name = "question" , expected_output_col_name = "answer" , context_col_name = "context" , context_col_delimiter = "|" )
From JSON
dataset . add_goldens_from_json_file ( file_path = "./test_data.json" , input_key_name = "query" , expected_output_key_name = "response" ) Synthetic generation from deepeval . synthesizer import Synthesizer synthesizer = Synthesizer ( )
From documents
goldens
synthesizer . generate_goldens_from_docs ( document_paths = [ "./docs/knowledge_base.pdf" ] , max_goldens_per_document = 10 , evolution_types = [ "REASONING" , "MULTICONTEXT" , "COMPARATIVE" ] )
From scratch
goldens
synthesizer . generate_goldens_from_scratch ( subject = "customer support for SaaS product" , task = "answer user questions about billing" , max_goldens = 20 ) Evolution types: REASONING, MULTICONTEXT, CONCRETISING, CONSTRAINED, COMPARATIVE, HYPOTHETICAL, IN_BREADTH See references/dataset_management.md for complete dataset guide including versioning and cloud integration. Test Case Types Single-turn (LLMTestCase) from deepeval . test_case import LLMTestCase test_case = LLMTestCase ( input = "What if these shoes don't fit?" , actual_output = "You have 30 days for full refund" , expected_output = "We offer 30-day full refund" , retrieval_context = [ "All customers eligible for 30 day refund" ] , tools_called = [ ToolCall ( name = "..." , arguments = { "..." : "..." } ) ] ) Multi-turn (ConversationalTestCase) from deepeval . test_case import Turn , ConversationalTestCase convo_test_case = ConversationalTestCase ( chatbot_role = "helpful customer service agent" , turns = [ Turn ( role = "user" , content = "I need help with my order" ) , Turn ( role = "assistant" , content = "I'd be happy to help" ) , Turn ( role = "user" , content = "It hasn't arrived yet" ) ] ) Multimodal (MLLMTestCase) from deepeval . test_case import MLLMTestCase , MLLMImage m_test_case = MLLMTestCase ( input = [ "Describe this image" , MLLMImage ( url = "./photo.png" , local = True ) ] , actual_output = [ "A red bicycle leaning against a wall" ] ) CI/CD Integration
.github/workflows/test.yml
name : LLM Tests on : [ push , pull_request ] jobs : evaluate : runs-on : ubuntu - latest steps : - uses : actions/checkout@v5 - name : Install dependencies run : pip install deepeval - name : Run evaluations env : OPENAI_API_KEY : $ { { secrets.OPENAI_API_KEY } } run : deepeval test run tests/ References Detailed implementation guides: references/model_providers.md - Complete guide for configuring OpenAI, Anthropic, Gemini, Bedrock, and local models. Includes provider-specific considerations, cost analysis, and troubleshooting. references/custom_metrics.md - Complete guide for creating custom metrics by subclassing BaseMetric. Includes deterministic scorers (ROUGE, BLEU, BERTScore) and LLM-based evaluation patterns. references/async_performance.md - Complete guide for optimising evaluation performance with async mode, caching, concurrency tuning, and rate limit handling. references/dataset_management.md - Complete guide for dataset loading, saving, synthetic generation, versioning, and cloud integration with Confident AI. Best Practices Metric Selection Match metrics to use case (RAG systems need retrieval + generation metrics) Start with 2-3 essential metrics, expand as needed Use appropriate thresholds (0.7-0.8 for production, 0.5-0.6 for development) Combine complementary metrics (answer relevancy + faithfulness) Test Case Design Create representative examples covering common queries and edge cases Include context when needed ( retrieval_context for RAG, expected_output for G-Eval) Use datasets for scale testing Version test cases over time Evaluation Workflow Component-level first - Use @observe for individual parts End-to-end validation before deployment Automate in CI/CD with deepeval test run Track results over time with Confident AI cloud Testing Anti-Patterns Avoid: Testing only happy paths Using unrealistic inputs Ignoring metric reasons Setting thresholds too high initially Running full test suite on every change Do: Test edge cases and failure modes Use real user queries as test inputs Read and analyse metric reasons Adjust thresholds based on empirical results Use component-level tests during development Separate config and eval content from code