# Back from OpenAI DevDay Exchange in London: what we’d tell any team building agents for 2027

> Source: <https://deepsenseai.substack.com/p/back-from-openai-devday-exchange>
> Published: 2026-11-19 15:31:44+00:00

**Recommendation:** Run your agent evals inside the production harness rather than a test loop, and repeat runs to catch the roughly 21% of answers that flip between pass and fail.
More in the playbook "Evals and testing for LLM apps": <https://wpnews.pro/playbooks/evals-testing>

Three of us from deepsense.ai, Maciej Domagała, Michał Lis and Artur Zygadło, were at [OpenAI’s DevDay Exchange](https://www.linkedin.com/feed/update/urn:li:activity:7523723396111867904) in London. We caught up with OpenAI’s EMEA team and swapped notes with people from the Agents API product team. It wasn’t our first DevDay: over the past few years we’ve been to several, in San Francisco and in London.

After every DevDay, our clients ask us the same thing: **which of these announcements should we actually build on?** This time Artur answered it in a [detailed piece on our blog](https://deepsense.ai/blog/openai-for-enterprise-agents-architecture-reliability-and-cost-in-2027/?utm_source=Substack&utm_medium=Post_19_11_2026&utm_campaign=LLM), based on our early access, client deployments and a few uncomfortable test results. Here are the parts we’d pass on to any team planning agents for 2027.

## The most worrying number

In the test loop, GPT-6 Luna scored 0.74 and GPT-5-mini scored 0.64. A 10-point gap. Pick the bigger model, move on.

In the product harness, Luna went up to 0.88. GPT-5-mini went down to 0.43. A 10-point gap in testing became 45 points in the product.

It gets messier. Across repeated runs, 21% of answers flipped between pass and fail.

If we had to put the lesson on a sticky note, it would say: evaluate agents where they will run.

## Our integration outlived the product name

One of our clients, a spirits producer, wanted ChatGPT connected to its Microsoft Fabric data. In January and February we built an MCP server on Azure, with OAuth and access limited to approved datasets. It shipped as a ChatGPT connector.

Then the product got renamed. Connectors became apps. Apps became plugins. The MCP server didn’t need a single change.

The reason is simple: the integration lives apart from the user interface.

We heard about Agent Builder at DevDay in London last year. A year later, it sits in legacy APIs with a migration guide. That’s why we look at every new announcement with some distance now. In the full piece we track what happened to last year’s announcements, and we’d look at that table before betting an architecture on this year’s.

## We tested what happens when it breaks halfway

We had [early access to the Agents API](https://deepsense.ai/blog/early-access-to-openais-agents-api-what-we-learned-from-testing-it-on-enterprise-workflows/?utm_source=Substack&utm_medium=Post_19_11_2026&utm_campaign=LLM) (in public beta since 10 September) and we gave it four multi-step workloads: a clinical-trial protocol synopsis, synthetic insurance claims, recovery from a faulty software release, and a presentation built from a brand brief.

The interesting questions weren’t about whether the agent finishes. They were about what happens when it doesn’t. Can you tell which external actions already happened? Which steps are safe to rerun? Can an operator see the difference between failed, still running, and completed-but-interrupted?

Test what happens when execution stops halfway. If your pilot never did, that’s probably where your first production incident will come from.

## Cheaper tokens don’t mean cheaper agents

GPT-6.1 Sol costs a fifth of GPT-6 Astra per token. That’s a good reason to dust off use cases rejected on cost a year ago. It isn’t a business case on its own.

What we look at is the cost of a finished task: tool calls, retries, human review, keeping the thing running. Architecture moves that number more than most people expect. For an agricultural machinery manufacturer, we consolidated several supply-chain agent prototypes into one platform. Tool calls per question dropped by 75%, and it now handles hundreds of requests a day.

Measure cost per outcome, not per token.

## What we left for the full article

We kept back the parts that matter most if you’re planning next year:

- what in OpenAI’s 2026 stack is actually beta, preview or still a proposal,
- how the supply-chain platform was built by June after a three-month advisory phase,
- a three-week expense-reporting PoC that’s now becoming a ChatGPT plugin,
- our six-step plan for a 2027 agent roadmap.
