cd /news/ai-agents/back-from-openai-devday-exchange-in-… · home › topics › ai-agents › article
[ARTICLE · art-172646] src=deepsenseai.substack.com ↗ pub= topic=ai-agents verified=true sentiment=· neutral

Back from OpenAI DevDay Exchange in London: what we’d tell any team building agents for 2027

Engineers from deepsense.ai reported that agent reliability measured in a product harness diverged sharply from test-loop results, with GPT-6 Luna scoring 0.88 versus GPT-5-mini's 0.43 in production-like conditions compared with 0.74 and 0.64 in testing, and 21% of answers flipping between pass and fail across repeated runs. The team also built an MCP server on Azure with OAuth and dataset-scoped access for a spirits producer's Microsoft Fabric data, which survived OpenAI's connector-to-apps-to-plugins renaming without changes, and consolidated several supply-chain agent prototypes for an agricultural machinery manufacturer, cutting tool calls per question by 75%. They advised teams to evaluate agents where they run, test partial-failure recovery, and measure cost per completed outcome rather than per token.

recommendation

Run your agent evals inside the production harness rather than a test loop, and repeat runs to catch the roughly 21% of answers that flip between pass and fail.

more in playbook: Evals and testing for LLM apps
read3 min views4 publishedNov 19, 2026
Back from OpenAI DevDay Exchange in London: what we’d tell any team building agents for 2027
Image: Deepsenseai (auto-discovered)

Three of us from deepsense.ai, Maciej Domagała, Michał Lis and Artur Zygadło, were at OpenAI’s DevDay Exchange in London. We caught up with OpenAI’s EMEA team and swapped notes with people from the Agents API product team. It wasn’t our first DevDay: over the past few years we’ve been to several, in San Francisco and in London.

After every DevDay, our clients ask us the same thing: which of these announcements should we actually build on? This time Artur answered it in a detailed piece on our blog, based on our early access, client deployments and a few uncomfortable test results. Here are the parts we’d pass on to any team planning agents for 2027.

The most worrying number #

In the test loop, GPT-6 Luna scored 0.74 and GPT-5-mini scored 0.64. A 10-point gap. Pick the bigger model, move on.

In the product harness, Luna went up to 0.88. GPT-5-mini went down to 0.43. A 10-point gap in testing became 45 points in the product.

It gets messier. Across repeated runs, 21% of answers flipped between pass and fail.

If we had to put the lesson on a sticky note, it would say: evaluate agents where they will run.

Our integration outlived the product name #

One of our clients, a spirits producer, wanted ChatGPT connected to its Microsoft Fabric data. In January and February we built an MCP server on Azure, with OAuth and access limited to approved datasets. It shipped as a ChatGPT connector.

Then the product got renamed. Connectors became apps. Apps became plugins. The MCP server didn’t need a single change.

The reason is simple: the integration lives apart from the user interface.

We heard about Agent Builder at DevDay in London last year. A year later, it sits in legacy APIs with a migration guide. That’s why we look at every new announcement with some distance now. In the full piece we track what happened to last year’s announcements, and we’d look at that table before betting an architecture on this year’s.

We tested what happens when it breaks halfway #

We had early access to the Agents API (in public beta since 10 September) and we gave it four multi-step workloads: a clinical-trial protocol synopsis, synthetic insurance claims, recovery from a faulty software release, and a presentation built from a brand brief.

The interesting questions weren’t about whether the agent finishes. They were about what happens when it doesn’t. Can you tell which external actions already happened? Which steps are safe to rerun? Can an operator see the difference between failed, still running, and completed-but-interrupted?

Test what happens when execution stops halfway. If your pilot never did, that’s probably where your first production incident will come from.

Cheaper tokens don’t mean cheaper agents #

GPT-6.1 Sol costs a fifth of GPT-6 Astra per token. That’s a good reason to dust off use cases rejected on cost a year ago. It isn’t a business case on its own.

What we look at is the cost of a finished task: tool calls, retries, human review, keeping the thing running. Architecture moves that number more than most people expect. For an agricultural machinery manufacturer, we consolidated several supply-chain agent prototypes into one platform. Tool calls per question dropped by 75%, and it now handles hundreds of requests a day.

Measure cost per outcome, not per token.

What we left for the full article #

We kept back the parts that matter most if you’re planning next year:

  • what in OpenAI’s 2026 stack is actually beta, preview or still a proposal,
  • how the supply-chain platform was built by June after a three-month advisory phase,
  • a three-week expense-reporting PoC that’s now becoming a ChatGPT plugin,
  • our six-step plan for a 2027 agent roadmap.
── more in #ai-agents 4 stories · sorted by recency
── more on @deepsense.ai 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/back-from-openai-dev…] indexed:0 read:3min 2026-11-19 · —