AI Evaluation Series (06): DeepEval in Practice — Enterprise Agent Evaluation Suite

Article 04 used RAGAS to batch-evaluate a RAG system, producing four metric scores. This article uses DeepEval on the same Agent — with a completely different paradigm.

RAGAS paradigm: batch evaluation, DataFrame input, outputs per-metric averages. Good for analyzing quality trends and comparing system versions.

DeepEval paradigm: test-case-first, each case has an explicit Pass/Fail verdict, native pytest integration. Good for CI quality gates — giving a clear go/no-go before each merge.

RAGAS answers "how was system quality this week?" DeepEval answers "can this commit go to production?" Different questions, different tools.

DeepEval defaults to OpenAI as its Judge LLM. To use glm-4-flash, subclass DeepEvalBaseLLM

:

from deepeval.models.base_model import DeepEvalBaseLLM

class GlmFlashEval(DeepEvalBaseLLM):
    def __init__(self):
        self._llm = ChatOpenAI(
            model="glm-4-flash",
            api_key=os.environ["LLM_API_KEY"],
            base_url="https://open.bigmodel.cn/api/paas/v4",
            temperature=0.0,
        )

    def load_model(self): return self._llm

    def generate(self, prompt: str, *args, **kwargs) -> str:
        return str(self._llm.invoke([HumanMessage(content=prompt)]).content)

    async def a_generate(self, prompt: str, *args, **kwargs) -> str:
        return self.generate(prompt)

    def get_model_name(self) -> str:
        return "glm-4-flash"

judge_llm = GlmFlashEval()

Pass judge_llm

when instantiating each metric:

AnswerRelevancyMetric(threshold=0.7, model=judge_llm)
FaithfulnessMetric(threshold=0.7, model=judge_llm)
ToolCorrectnessMetric(model=judge_llm)

DeepEval's unit of work is LLMTestCase

. Each case holds:

from deepeval.test_case import LLMTestCase, ToolCall

case = LLMTestCase(
    input="What's your refund policy?",
    actual_output=answer,                           # Agent's actual response
    expected_tools=[ToolCall(name="search_faq")],   # expected tool sequence
    tools_called=[ToolCall(name="search_faq")],     # actual tool sequence
    retrieval_context=["Refund policy: full refund within 7 days..."],
)

tools_called

and expected_tools

require ToolCall

objects — not plain strings.

Question                                AnsRel   Faith   ToolOK
────────────────────────────────────── ───────  ──────  ───────
What's your refund policy?              1.00 ✓   0.50 ✗   ✗
Did order ORD-001 ship?                 0.33 ✗   0.50 ✗   ✗
How much refund for ORD-004?            1.00 ✓   1.00 ✓   ✗
What payment methods do you support?    0.50 ✗   0.00 ✗   ✗
Bought 299¥ item 3 days ago, refund?    1.00 ✓   1.00 ✓   ✓

Aggregate:
  AnswerRelevancy   avg=0.767  pass_rate=60%
  Faithfulness      avg=0.600  pass_rate=40%
  ToolCorrectness   avg=0.200  pass_rate=20%

AnswerRelevancy (avg 0.767, 60% pass)

Q2 ("Did order ORD-001 ship?") scored 0.33. The Agent skipped get_order_status

and produced a vague "Regarding your order..." non-answer. Low Answer Relevancy is downstream of tool triggering failure, not LLM generation quality.

Faithfulness (avg 0.600, 40% pass)

Q1 scored 0.50: the Agent answered from its own knowledge without calling a tool, and some details differed from what's in the FAQ database.

Q4 scored 0.00 — an extreme case: the Agent answered "we support WeChat Pay, Alipay, bank cards..." but retrieval_context

was "No context retrieved" (no tool was called). The framework considers an answer completely unsupported when context is empty.

This 0.0 exposes an evaluation trap. When an Agent skips tool calls and answers directly, the context is empty, and Faithfulness scores zero — even if the factual content is correct. Faithfulness measures "does the answer go beyond the context," but that only makes sense when context exists.

ToolCorrectness (avg 0.200, 20% pass)

Only the last case passed — the one where the user provided amount and days directly, triggering calculate_refund

. The other 4 cases: Agent didn't call the expected tools. ToolCorrectness is DeepEval's most distinctive advantage over RAGAS: it directly evaluates whether the tool call sequence matched what was expected.

DeepEval's strength is native pytest integration. In CI:

import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase, ToolCall
from deepeval.metrics import AnswerRelevancyMetric, ToolCorrectnessMetric

@pytest.mark.parametrize("case", build_test_cases())
def test_agent_response(case):
    assert_test(case, metrics=[
        AnswerRelevancyMetric(threshold=0.7, model=judge_llm),
        ToolCorrectnessMetric(model=judge_llm),
    ])
name: Agent Quality Gate
on: [pull_request]
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install deepeval
      - run: pytest tests/test_agent_quality.py -v

Threshold guidance:

Dimension          RAGAS                      DeepEval
─────────────────────────────────────────────────────────────────
Paradigm           Metric-first (batch)       Test-case-first (pytest)
Input format       Dataset (DataFrame)        LLMTestCase objects
Output             Score per metric           Pass/Fail + score + reason
CI integration     Needs wrapper              Native pytest, assert_test()
Tool eval          No built-in                ToolCorrectnessMetric
Custom metrics     Via custom scorers         Via BaseMetric subclass
Best for           Trend analysis, comparison CI gates, regression tests
─────────────────────────────────────────────────────────────────
Use RAGAS when:    Analyzing quality trends over time; comparing v1 vs v2
Use DeepEval when: Need clear pass/fail before PR merge; CI gate enforcement

Use both, not one. They're complementary:

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.

Find more useful knowledge and interesting products on my Homepage