What our GLM vs Opus experiment taught us about agent harnesses.
At Funnel, we've been experimenting with different language models powering Reef, our internal data agent platform.
Reef operates across three layers of company knowledge - a company world model represented as a graph, structured operational data containing carefully modelled financial, SaaS and CRM metrics, and unstructured customer conversations from which we extract structured claims. It's a fairly sophisticated environment for an agent to navigate.
Recently, we ran a randomised A/B test comparing GLM-5.2, an open-weight model, against Claude Opus 4.8. Each new user task was randomly assigned a model, which remained the same throughout subsequent interactions, including deeper investigations and the creation of interactive dashboards we call explorations.
The initial results looked promising - across our observation window, GLM averaged 7.8 seconds per model call, compared to 18.1 seconds for Opus - more than twice as fast.
But when we looked at end-to-end task execution, a different picture emerged. GLM made 66% more tool calls, averaging 19 compared to Opus's 11. Despite faster individual responses, GLM took 14% longer to complete those tasks on average.
In two separate runs, GLM executed the same successful SQL query 113 and 254 times.
In one case, the model was attempting to retrieve more data from a tool that only exposed a 20-row preview. Rather than recognising the limitation, it repeatedly executed the same query. Every call succeeded but the agent obviously wasn't making any progress. A successful tool call is not necessarily a useful tool call.
We've observed similar tendencies with Kimi K2.6 and Qwen3.8 27B; Qwen is twice as aggressive on tool usage compared to GLM.
Now, it's tempting to blame the model, but that's missing the big picture. Some of our saved agent instructions referenced Athena tables and fields that didn't actually exist. Other problems revealed ambiguous tool contracts, missing safeguards against repetitive calls, or insufficiently explicit instructions about data retrieval. Opus was able to overcome most of these issues, but they are issues nonetheless.
Agent harnesses matter as much as model intelligence.
Different models interpret the same instructions, tools, and stopping conditions differently. Replacing a model can expose architectural assumptions that were previously invisible.
For engineers integrating open-weight models into existing agentic platforms, I'd suggest three things:
- Make tool contracts explicit. Distinguish previews from complete results, expose pagination clearly, and communicate what a successful response actually means.
- Bound agent behaviour. Detect repeated equivalent calls, enforce sensible execution budgets, and validate saved instructions against real schemas.
- Benchmark the whole system. Measure useful task completion, correctness, latency, tool efficiency, billed costs and the effects of input caching.
Open-weight models offer exciting opportunities for cost, performance and control. Realising those benefits may require more than swapping out an API endpoint.
PS. GLM-5.2 is a happy little camper now (actually, we now use GLM-5.3).