{"slug": "ai-evaluation-series-06-deepeval-in-practice-enterprise-agent-evaluation-suite", "title": "AI Evaluation Series (06): DeepEval in Practice — Enterprise Agent Evaluation Suite", "summary": "A developer demonstrates using DeepEval to evaluate an enterprise agent, contrasting its test-case-first paradigm with RAGAS's batch evaluation. The implementation uses a custom judge LLM (glm-4-flash) and reveals that low Faithfulness scores can be a trap when agents skip tool calls, leaving an empty retrieval context.", "body_md": "Article 04 used RAGAS to batch-evaluate a RAG system, producing four metric scores. This article uses DeepEval on the same Agent — with a completely different paradigm.\n\n**RAGAS paradigm:** batch evaluation, DataFrame input, outputs per-metric averages. Good for analyzing quality trends and comparing system versions.\n\n**DeepEval paradigm:** test-case-first, each case has an explicit Pass/Fail verdict, native pytest integration. Good for CI quality gates — giving a clear go/no-go before each merge.\n\nRAGAS answers \"how was system quality this week?\" DeepEval answers \"can this commit go to production?\" Different questions, different tools.\n\nDeepEval defaults to OpenAI as its Judge LLM. To use glm-4-flash, subclass `DeepEvalBaseLLM`\n\n:\n\n``` python\nfrom deepeval.models.base_model import DeepEvalBaseLLM\n\nclass GlmFlashEval(DeepEvalBaseLLM):\n    def __init__(self):\n        self._llm = ChatOpenAI(\n            model=\"glm-4-flash\",\n            api_key=os.environ[\"LLM_API_KEY\"],\n            base_url=\"https://open.bigmodel.cn/api/paas/v4\",\n            temperature=0.0,\n        )\n\n    def load_model(self): return self._llm\n\n    def generate(self, prompt: str, *args, **kwargs) -> str:\n        return str(self._llm.invoke([HumanMessage(content=prompt)]).content)\n\n    async def a_generate(self, prompt: str, *args, **kwargs) -> str:\n        return self.generate(prompt)\n\n    def get_model_name(self) -> str:\n        return \"glm-4-flash\"\n\njudge_llm = GlmFlashEval()\n```\n\nPass `judge_llm`\n\nwhen instantiating each metric:\n\n```\nAnswerRelevancyMetric(threshold=0.7, model=judge_llm)\nFaithfulnessMetric(threshold=0.7, model=judge_llm)\nToolCorrectnessMetric(model=judge_llm)\n```\n\nDeepEval's unit of work is `LLMTestCase`\n\n. Each case holds:\n\n``` python\nfrom deepeval.test_case import LLMTestCase, ToolCall\n\ncase = LLMTestCase(\n    input=\"What's your refund policy?\",\n    actual_output=answer,                           # Agent's actual response\n    expected_tools=[ToolCall(name=\"search_faq\")],   # expected tool sequence\n    tools_called=[ToolCall(name=\"search_faq\")],     # actual tool sequence\n    retrieval_context=[\"Refund policy: full refund within 7 days...\"],\n)\n```\n\n`tools_called`\n\nand `expected_tools`\n\nrequire `ToolCall`\n\nobjects — not plain strings.\n\n```\nQuestion                                AnsRel   Faith   ToolOK\n────────────────────────────────────── ───────  ──────  ───────\nWhat's your refund policy?              1.00 ✓   0.50 ✗   ✗\nDid order ORD-001 ship?                 0.33 ✗   0.50 ✗   ✗\nHow much refund for ORD-004?            1.00 ✓   1.00 ✓   ✗\nWhat payment methods do you support?    0.50 ✗   0.00 ✗   ✗\nBought 299¥ item 3 days ago, refund?    1.00 ✓   1.00 ✓   ✓\n\nAggregate:\n  AnswerRelevancy   avg=0.767  pass_rate=60%\n  Faithfulness      avg=0.600  pass_rate=40%\n  ToolCorrectness   avg=0.200  pass_rate=20%\n```\n\n**AnswerRelevancy (avg 0.767, 60% pass)**\n\nQ2 (\"Did order ORD-001 ship?\") scored 0.33. The Agent skipped `get_order_status`\n\nand produced a vague \"Regarding your order...\" non-answer. Low Answer Relevancy is downstream of tool triggering failure, not LLM generation quality.\n\n**Faithfulness (avg 0.600, 40% pass)**\n\nQ1 scored 0.50: the Agent answered from its own knowledge without calling a tool, and some details differed from what's in the FAQ database.\n\nQ4 scored 0.00 — an extreme case: the Agent answered \"we support WeChat Pay, Alipay, bank cards...\" but `retrieval_context`\n\nwas \"No context retrieved\" (no tool was called). The framework considers an answer completely unsupported when context is empty.\n\n**This 0.0 exposes an evaluation trap.** When an Agent skips tool calls and answers directly, the context is empty, and Faithfulness scores zero — even if the factual content is correct. Faithfulness measures \"does the answer go beyond the context,\" but that only makes sense when context exists.\n\n**ToolCorrectness (avg 0.200, 20% pass)**\n\nOnly the last case passed — the one where the user provided amount and days directly, triggering `calculate_refund`\n\n. The other 4 cases: Agent didn't call the expected tools. ToolCorrectness is DeepEval's most distinctive advantage over RAGAS: it directly evaluates whether the tool call sequence matched what was expected.\n\nDeepEval's strength is native pytest integration. In CI:\n\n``` python\n# tests/test_agent_quality.py\nimport pytest\nfrom deepeval import assert_test\nfrom deepeval.test_case import LLMTestCase, ToolCall\nfrom deepeval.metrics import AnswerRelevancyMetric, ToolCorrectnessMetric\n\n@pytest.mark.parametrize(\"case\", build_test_cases())\ndef test_agent_response(case):\n    assert_test(case, metrics=[\n        AnswerRelevancyMetric(threshold=0.7, model=judge_llm),\n        ToolCorrectnessMetric(model=judge_llm),\n    ])\n# .github/workflows/eval.yml\nname: Agent Quality Gate\non: [pull_request]\njobs:\n  eval:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - run: pip install deepeval\n      - run: pytest tests/test_agent_quality.py -v\n        # any test case below threshold → CI fails\n```\n\n**Threshold guidance:**\n\n```\nDimension          RAGAS                      DeepEval\n─────────────────────────────────────────────────────────────────\nParadigm           Metric-first (batch)       Test-case-first (pytest)\nInput format       Dataset (DataFrame)        LLMTestCase objects\nOutput             Score per metric           Pass/Fail + score + reason\nCI integration     Needs wrapper              Native pytest, assert_test()\nTool eval          No built-in                ToolCorrectnessMetric\nCustom metrics     Via custom scorers         Via BaseMetric subclass\nBest for           Trend analysis, comparison CI gates, regression tests\n─────────────────────────────────────────────────────────────────\nUse RAGAS when:    Analyzing quality trends over time; comparing v1 vs v2\nUse DeepEval when: Need clear pass/fail before PR merge; CI gate enforcement\n```\n\n**Use both, not one.** They're complementary:\n\n*Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.*\n\n*Find more useful knowledge and interesting products on my Homepage*", "url": "https://wpnews.pro/news/ai-evaluation-series-06-deepeval-in-practice-enterprise-agent-evaluation-suite", "canonical_source": "https://dev.to/wonderlab/ai-evaluation-series-06-deepeval-in-practice-enterprise-agent-evaluation-suite-22c", "published_at": "2026-07-25 13:20:43+00:00", "updated_at": "2026-07-25 13:32:41.458232+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "developer-tools"], "entities": ["DeepEval", "RAGAS", "OpenAI", "glm-4-flash", "ChatOpenAI"], "alternates": {"html": "https://wpnews.pro/news/ai-evaluation-series-06-deepeval-in-practice-enterprise-agent-evaluation-suite", "markdown": "https://wpnews.pro/news/ai-evaluation-series-06-deepeval-in-practice-enterprise-agent-evaluation-suite.md", "text": "https://wpnews.pro/news/ai-evaluation-series-06-deepeval-in-practice-enterprise-agent-evaluation-suite.txt", "jsonld": "https://wpnews.pro/news/ai-evaluation-series-06-deepeval-in-practice-enterprise-agent-evaluation-suite.jsonld"}}