AI Evaluation Series (06): DeepEval in Practice — Enterprise Agent Evaluation Suite

A developer demonstrates using DeepEval to evaluate an enterprise agent, contrasting its test-case-first paradigm with RAGAS's batch evaluation. The implementation uses a custom judge LLM (glm-4-flash) and reveals that low Faithfulness scores can be a trap when agents skip tool calls, leaving an empty retrieval context.

Article 04 used RAGAS to batch-evaluate a RAG system, producing four metric scores. This article uses DeepEval on the same Agent — with a completely different paradigm. RAGAS paradigm: batch evaluation, DataFrame input, outputs per-metric averages. Good for analyzing quality trends and comparing system versions. DeepEval paradigm: test-case-first, each case has an explicit Pass/Fail verdict, native pytest integration. Good for CI quality gates — giving a clear go/no-go before each merge. RAGAS answers "how was system quality this week?" DeepEval answers "can this commit go to production?" Different questions, different tools. DeepEval defaults to OpenAI as its Judge LLM. To use glm-4-flash, subclass DeepEvalBaseLLM : python from deepeval.models.base model import DeepEvalBaseLLM class GlmFlashEval DeepEvalBaseLLM : def init self : self. llm = ChatOpenAI model="glm-4-flash", api key=os.environ "LLM API KEY" , base url="https://open.bigmodel.cn/api/paas/v4", temperature=0.0, def load model self : return self. llm def generate self, prompt: str, args, kwargs - str: return str self. llm.invoke HumanMessage content=prompt .content async def a generate self, prompt: str, args, kwargs - str: return self.generate prompt def get model name self - str: return "glm-4-flash" judge llm = GlmFlashEval Pass judge llm when instantiating each metric: AnswerRelevancyMetric threshold=0.7, model=judge llm FaithfulnessMetric threshold=0.7, model=judge llm ToolCorrectnessMetric model=judge llm DeepEval's unit of work is LLMTestCase . Each case holds: python from deepeval.test case import LLMTestCase, ToolCall case = LLMTestCase input="What's your refund policy?", actual output=answer, Agent's actual response expected tools= ToolCall name="search faq" , expected tool sequence tools called= ToolCall name="search faq" , actual tool sequence retrieval context= "Refund policy: full refund within 7 days..." , tools called and expected tools require ToolCall objects — not plain strings. Question AnsRel Faith ToolOK ────────────────────────────────────── ─────── ────── ─────── What's your refund policy? 1.00 ✓ 0.50 ✗ ✗ Did order ORD-001 ship? 0.33 ✗ 0.50 ✗ ✗ How much refund for ORD-004? 1.00 ✓ 1.00 ✓ ✗ What payment methods do you support? 0.50 ✗ 0.00 ✗ ✗ Bought 299¥ item 3 days ago, refund? 1.00 ✓ 1.00 ✓ ✓ Aggregate: AnswerRelevancy avg=0.767 pass rate=60% Faithfulness avg=0.600 pass rate=40% ToolCorrectness avg=0.200 pass rate=20% AnswerRelevancy avg 0.767, 60% pass Q2 "Did order ORD-001 ship?" scored 0.33. The Agent skipped get order status and produced a vague "Regarding your order..." non-answer. Low Answer Relevancy is downstream of tool triggering failure, not LLM generation quality. Faithfulness avg 0.600, 40% pass Q1 scored 0.50: the Agent answered from its own knowledge without calling a tool, and some details differed from what's in the FAQ database. Q4 scored 0.00 — an extreme case: the Agent answered "we support WeChat Pay, Alipay, bank cards..." but retrieval context was "No context retrieved" no tool was called . The framework considers an answer completely unsupported when context is empty. This 0.0 exposes an evaluation trap. When an Agent skips tool calls and answers directly, the context is empty, and Faithfulness scores zero — even if the factual content is correct. Faithfulness measures "does the answer go beyond the context," but that only makes sense when context exists. ToolCorrectness avg 0.200, 20% pass Only the last case passed — the one where the user provided amount and days directly, triggering calculate refund . The other 4 cases: Agent didn't call the expected tools. ToolCorrectness is DeepEval's most distinctive advantage over RAGAS: it directly evaluates whether the tool call sequence matched what was expected. DeepEval's strength is native pytest integration. In CI: python tests/test agent quality.py import pytest from deepeval import assert test from deepeval.test case import LLMTestCase, ToolCall from deepeval.metrics import AnswerRelevancyMetric, ToolCorrectnessMetric @pytest.mark.parametrize "case", build test cases def test agent response case : assert test case, metrics= AnswerRelevancyMetric threshold=0.7, model=judge llm , ToolCorrectnessMetric model=judge llm , .github/workflows/eval.yml name: Agent Quality Gate on: pull request jobs: eval: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: pip install deepeval - run: pytest tests/test agent quality.py -v any test case below threshold → CI fails Threshold guidance: Dimension RAGAS DeepEval ───────────────────────────────────────────────────────────────── Paradigm Metric-first batch Test-case-first pytest Input format Dataset DataFrame LLMTestCase objects Output Score per metric Pass/Fail + score + reason CI integration Needs wrapper Native pytest, assert test Tool eval No built-in ToolCorrectnessMetric Custom metrics Via custom scorers Via BaseMetric subclass Best for Trend analysis, comparison CI gates, regression tests ───────────────────────────────────────────────────────────────── Use RAGAS when: Analyzing quality trends over time; comparing v1 vs v2 Use DeepEval when: Need clear pass/fail before PR merge; CI gate enforcement Use both, not one. They're complementary: Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works. Find more useful knowledge and interesting products on my Homepage