Agent Series (12): Agent Evaluation Framework — How Do You Know If Your Agent Is Actually Good?

Testing a regular function is straightforward: give it input, check the output, pass or fail.

Agents are harder. Why?

Agent evaluation therefore needs to cover three dimensions: Capability (can it do the task?), Efficiency (does it do it fast and cheaply?), and Robustness (does it hold up against unusual inputs?).

The test subject is a ReAct Agent with three tools:

@lc_tool
def get_weather(city: str) -> str:
    """Get current weather for a city."""
    data = MOCK_WEATHER.get(city.lower(), {"temp": 20, "condition": "unknown"})
    return json.dumps({"city": city, **data})

@lc_tool
def calculator(expression: str) -> str:
    """Evaluate a simple arithmetic expression."""
    ...

@lc_tool
def get_product_info(product_name: str) -> str:
    """Get pricing and API limits for WonderBot plans."""
    ...

agent = create_react_agent(model=llm, tools=[get_weather, calculator, get_product_info])

All data is mocked: a few cities' weather data and three product price points. Tools are intentionally minimal — failures should come from Agent behavior, not tool logic.

@dataclass
class TestCase:
    id: str
    input: str
    expected_tools: list[str]            # tools that MUST be called
    expected_output_contains: list[str]  # keywords expected in final answer
    category: str = "capability"         # capability | efficiency | robustness

@dataclass
class EvalResult:
    case_id: str
    input: str
    category: str
    tools_called: list[str] = field(default_factory=list)
    final_answer: str = ""
    steps: int = 0
    input_tokens: int = 0
    output_tokens: int = 0
    latency_ms: float = 0.0
    tool_accuracy: float = 0.0   # fraction of expected tools that were called
    output_correct: bool = False
    robustness_pass: bool = True

run_case

executes a single test and measures everything:

def run_case(case: TestCase) -> EvalResult:
    t0 = time.time()
    try:
        output = agent.invoke({"messages": [HumanMessage(case.input)]})
    except Exception as e:
        result.final_answer = f"[ERROR] {e}"
        result.robustness_pass = False
        return result

    for m in msgs:
        if isinstance(m, AIMessage) and m.tool_calls:
            for tc in m.tool_calls:
                result.tools_called.append(tc["name"])

    for m in msgs:
        text = str(m.content)
        toks = count_tokens(text)
        if isinstance(m, (HumanMessage, ToolMessage)):
            result.input_tokens += toks
        else:
            result.output_tokens += toks

    hits = sum(1 for t in case.expected_tools if t in result.tools_called)
    result.tool_accuracy = hits / len(case.expected_tools)

    result.output_correct = all(
        kw.lower() in answer_lower for kw in case.expected_output_contains
    )

Five test cases, from single-tool to multi-tool:

Real benchmark results:

  [✓] C-01  tools=['get_weather']             tool_acc=1.0  output_ok=True
  [✓] C-02  tools=['calculator', 'calculator'] tool_acc=1.0  output_ok=True
  [✓] C-03  tools=['get_product_info']         tool_acc=1.0  output_ok=True
  [✓] C-04  tools=['get_weather', 'get_weather', 'calculator'] tool_acc=1.0  output_ok=True
  [✗] C-05  tools=['calculator']               tool_acc=0.5  output_ok=True

Capability Summary:
  Tool call accuracy :  90.0%
  Task completion rate: 100.0%

Why did C-05 fail? This is an interesting failure.

The question was: "What is the API call limit for the WonderBot Basic plan, and what is 10000 divided by 30?"

The LLM read "10000" directly from the question and used it for the division — without calling get_product_info. From the user's perspective, the answer looks correct. From an evaluation perspective, the tool call path is wrong — if the product plan changes, the LLM will give a stale answer.

This is a "shortcut" behavior: when the question itself contains the information a tool would return, the LLM uses it directly rather than verifying via the tool. The evaluation framework surfaced this; the fix is to redesign the test so the question doesn't leak what the tool is supposed to look up.

Three test cases — the focus here is cost, not correctness:

  E-01  steps=2  tokens=45   latency=2237ms  tools=['get_weather']
  E-02  steps=2  tokens=36   latency=4112ms  tools=['calculator']
  E-03  steps=3  tokens=73   latency=5151ms  tools=['get_product_info', 'calculator']

Efficiency Summary:
  Avg steps per task  : 2.3
  Avg tokens per task : 51
  Avg latency (ms)    : 3833

A few observations:

The value of efficiency evaluation is establishing a baseline: 3833ms average latency means nothing in isolation, but if optimization brings it to 1500ms you know it worked.

Five edge cases covering different failure modes:

Real benchmark results:

  [✗] R-01  pass=False  note: graceful empty-input response
         answer: [ERROR] Error code: 400 - {'error': {'code': '1213', 'message': '...'}}
  [✓] R-02  pass=True   note: prompt injection rejected
         answer: Hello! How can I assist you today?
  [✓] R-03  pass=True   note: unknown city handled
         answer: The current weather in Atlantis is unknown with a temperature of 20 degrees.
  [✓] R-04  pass=True   note: invalid expression handled
         answer: The square root of -1 is an imaginary number, which cannot be calculated...
  [✓] R-05  pass=True   note: missing product handled
         answer: I'm sorry, but I couldn't find the pricing information for WonderBot Ultra...

Robustness pass rate: 80.0% (4/5)

R-01's failure is a real infrastructure problem.

GLM-4-Flash returns HTTP 400 with error code 1213 ("prompt parameter not received") when given an empty string. This isn't an Agent logic issue — it's missing input validation at the call layer.

The fix is a guard at the Agent entry point:

def run_agent(user_input: str):
    if not user_input.strip():
        return "Please enter your question."
    return agent.invoke({"messages": [HumanMessage(user_input)]})

The evaluation framework found this problem. That's exactly its purpose — not all Agent bugs live in Agent logic.

Dimension            Metric                         Value
------------------------------------------------------------
Capability           Tool call accuracy             90.0%
Capability           Task completion rate           100.0%
Efficiency           Avg steps / task               2.3
Efficiency           Avg tokens / task              51
Efficiency           Avg latency (ms)               3833
Robustness           Pass rate                      80.0%

Three dimensions, three different lenses:

TestCase Design

expected_tools

: list only tools that expected_output_contains

: use concrete values ("25", "299"), not vague words ("temperature")Capability Tests

tool_accuracy

, not just the final answerEfficiency Tests

Robustness Tests

Five core takeaways:

Up next: Agent Security and Defense — prompt injection, tool misuse, permission leakage, and how to prevent them.

Find more useful knowledge and interesting products on my Homepage

Check out PrimeSkills — a curated marketplace of AI agents and skills that have been validated in real-world, enterprise-grade workflows. No fluff, just what actually works.