In this episode, Stack Overflow Director of Platform Engineering Peter O'Connor continues his conversation with Andi Gutmans, this time digging into what changes when you're running dozens or hundreds of agents instead of a handful of humans clicking buttons. Andi explains why he thinks the model itself is rarely the bottleneck anymore. In his view, the harder problem is figuring out the least amount of context needed to get a reliable outcome at the lowest cost, and why "token maxing" is the wrong goal entirely. He also shares how a weekend prototype changed the calculus on de-risking new ideas, and lays out what platform engineering teams need to build to treat "the agent" as a persona in its own right.
The discussion also:
- Breaks down why cost governance matters more at agent scale, where a single employee might be responsible for dozens of agents working around the clock.
- Covers Google's approach to "hill climbing" model and data together, and why Andi is skeptical of vendors claiming to have solved the context problem.
- Explores what new tooling and observability platform teams need to build to serve "the agent" as a persona alongside developers, data scientists, and business users.
- Closes with Andi's advice for someone starting a computer science degree today: fundamentals still matter, agentic fluency is now essential, and business judgment may matter as much as either.
Notes
TRANSCRIPT
Eira May:
Hi, and welcome back to Leaders of Code. This is a segment on the Stack Overflow Podcast where we get senior engineering leaders together to talk about the work they're doing, how they build great teams, and the biggest challenges they face in the market right now. My name is Eira May. I'm the B2B editor at Stack Overflow, and this is the second half of our conversation with Peter O'Connor, Director of Platform Engineering at Stack Overflow, and Andi Gutmans, who is Head of Agentic Data Cloud at Google. I hope you really enjoy the second half of their chat.
Peter O'Connor:
You discussed about token efficiency, and that's becoming a very, very big thing lately. Where do you think... I agree, I don't think models need to get much better. I think people still opine about Opus 4.6, about how amazing it is. So where do you think models have to go from here? Is it really an improvement? Do you think it's now becoming more in-house? Are we always going to be using frontier models? Where's the next step you think we're going to see you taking in the model industry?
Andi Gutmans:
Look, I think models will definitely keep on getting better, and I think we're seeing that every model that's coming out of DeepMind, models that are coming out of other players, right, they are getting consistently better. So I'm not saying that they shouldn't get better and they won't get better. But what I am saying is that I think for a lot of tasks that we're trying to automate today, a lot of the models we have today are actually good enough. Now, I think about this problem more as a system as a whole problem. It's not just about the model, right? It's the model, it's the harness, it's the context, it's the personalization. And the way I think about this is you want the least sophisticated model that gets the job done at the lowest cost. So it's very use case specific, right? In some use cases, it may be a super lightweight model.
I mean like Gemini 3.5 Flash is awesome and it's super low cost, right? But you're going to have situations where you're going to want to use Gemini Pro. And so I think we're going to get better at figuring out what model to use when, how to make sure that we are limiting the context to only the context that matters. So we actually need to do less token processing, plus the models have more accurate trajectories. So you're going to have less of a reasoning loop, which is also going to reduce the amount of tokens. So I think a lot of the prompts shift towards that system as a whole problem. That's where I think Google is very fortunate because we're actually the only hyperscaler who has both in-house model development and has a differentiated data platform and understands information really well. So that gives us the opportunity to hill climb in partnership with DeepMind model and data platform together, really thinking about this problem of the least amount of context that drives outcomes you need at the lowest cost.
Peter O'Connor:
Yeah, that's super valuable. I think I'm with you on trying to figure out how data plays in. The systemization's a really smart thought experiment when we think about how this works. I was talking to a friend the other day and saying the one worry I have recently is when I see these harnesses and people are talking about agents just getting better and better in these models. I keep thinking about, for lack of a better word, but drills. I want to make sure I get a model. That system of the drill system to me matters. I don't need you just to make a better drill that makes a bit that drills a hole better, but now drills can paddle mix, they can clamp things together, they can cut wood. If they get too specialized, that's too much of a problem. So I really like that systemization point of view to it, so.
Andi Gutmans:
Yeah.
Peter O'Connor:
When you think about data and you think about the skills and the context, you're talking about the least amount of context viable to make the job done, which is great. How do you measure that? How do you approach a problem space like that where it's like, yeah, here's the scope of knowledge you need. Why? Because this is all I've ever known, but really all you need is this. What's your thinking behind scoping that problem a little bit better?
Andi Gutmans:
So it's actually a really hard problem. So that's why I get a bit suspicious when you have players that say, "Well, I've got this catalog and I'm enriching all this data. It's going to make models better." You actually have to hill climb model and knowledge together and actually have the right level of evals to really understand what part of the context is actually moving the needle. Because you're optimizing for agents, you're not optimizing for humans. And I think what I see is a lot of companies are approaching this in the old-fashioned way, which is like, oh, I'm going to let humans build the curated ontology and that's going to make my agents better. But the reality is that's theory, it's not practice. So we're going after this in, I would say, much more of an AI first thinking way, which is our customer is the agent.
How do we make sure the agent gets the right outcome? And it's just a ton of work on the eval side, really understanding the trajectories, working with model builders like DeepMinds to make sure that if anything has to change in the model, it changes in the model. Otherwise, it changes in our context. And we're bringing all of Google's differentiation to the table there, both on enrichment, on search. Search is a big deal, which is how do you know which context actually matters? Most vendors don't talk about search. I put this as probably at least in the top three problems of solving. So it's a big difficult problem. Any vendor that will say they've solved it fully is not being accurate. We haven't solved it fully, but I do think we are on the right path to making sure we can really optimize these dimensions.
Peter O'Connor:
Really warms my heart to know that you haven't solved it because internally we have our own B2B SaaS product we're going out with, Stack Internal. And our position is like, hey, the context problem is difficult. What we want to try to do is have agents solve a lot of your issues and inject a human in the loop as necessary, right? Because we know they can be the authority and that'll help you achieve your business outcomes today, so.
Andi Gutmans:
Exactly.
Peter O'Connor:
I'm so glad to hear that.
Andi Gutmans:
Yeah. And by the way, we have seen that human intuition on what is helpful does not always work, which is why you need a more systematic way to tease that out.
Peter O'Connor:
Yep, for sure. Humans tend to believe things that aren't really true sometimes, but we're fallible.
Andi Gutmans:
Exactly.
Peter O'Connor:
Oh, gosh.
Andi Gutmans:
We also hallucinate sometimes.
Peter O'Connor:
Yeah, we do. We do some real amazing thing. If tokens are becoming expensive and we want to make these context windows just right and we have to pull in all this data and we have to do agent ontology creation and mapping, how do we generate a system of understanding, like return on investment? How much do I need to invest in a data platform versus making sure my harnesses are right, making sure I made the skills right? Is that in your brain space at all? Are you thinking about that?
Andi Gutmans:
Yeah. I mean, not to sound too self-serving here, but it probably will sound a bit self-serving. I think that's a bit of the benefit of we have the whole stack. We've got the GPUs, we've got the models, we've got the data platform. And so our goal is to actually make sure this is all integrated and optimized so customers actually have a pretty easy time getting to the outcomes they need. We're also open, so we'll also work with third party models and so on. But the reality is for most enterprises, solving this on their own is going to be too hard. They do need to work with folks. And I think we're pretty good at it as Google. It's working with folks who can bring that all stack together and then also make sure that that stack is super cost-effective. So for us, it's very clear that it's not only getting to the best outcomes, but we also have to be the most cost-effective in the industry, right? And so we're also consistently working on driving down the cost of being successful with AI. And I think our customers are feeling that.
Peter O'Connor:
Yeah. Yeah. You've mentioned cost a lot here. Is cost, is that up and down the market segments? Have you heard? Is it from Epic enterprise down to the little people or where do you see that most?
Andi Gutmans:
I mean, I think costs matter to everyone right now because when you think about human scale, humans can only click buttons that many times a day. So you can expect more or less or forecast what the load's going to be and what the cost is going to be. But the reality is we get to agent scale, you're going to have 50 agents working on your behalf. You may have some employees who have 100 of their own agents working on their behalf. We may have millions of agents within an enterprise environment. And agents scale, these agents are working 24/7. They don't need to sleep. They can click on buttons way faster than us, right? And so you do have a risk of getting into a cost spiral unless you're making sure you're really governing the use of AI, the use of agents, you know exactly what the ROI is from those efforts.
And then of course you're also working with vendors who bend the price performance curve for you, right? So are consistently making sure that not only the model is getting more cost-effective, but as I said before, the system as a whole, right? Because it's the model, it's the harness, it's the tools they're calling, very often the data platform, right, tools that they're calling. How do you make sure that that can actually scale in a non-linear way?
Peter O'Connor:
Yeah. No, my chief financial officer would completely agree with you. Forecasting has been very difficult. And I think AI is burdened with another problem that we haven't talked too much about, which is no one minds spending money if it has value, but AI's sort of behind the eight-ball of it seems to cost a lot more than other tools. But some people I've talked to, I ask very interesting questions that I say things like, "Hey, do you have JIRA?" And they're like, "Yeah, we have JIRA. We have Confluence." And I'm like, "Okay, how do you know the money you're spending there is actually an ROI? You're getting ROI?" And a lot of them can't explain that, right? And I don't blame them. Hey, I'm standing up a building. I need to build a building. It doesn't matter what it's like, so. But AI's gotten already a moniker of a negativity to it, so.
Andi Gutmans:
To be honest, I would say our customers are super smart and they're not going to invest a huge amount of money in something that isn't going to drive an ROI. So they're absolutely looking at the ROI, whether it's in customer support, really thinking about how many tickets they can handle, can they drive to better outcomes? How can they think about staffing in that context? We have Deutsche Telekom with autonomous network operations, being able to more proactively manage the network and optimize it, proactive maintenance. So I do think that when I look at our customer base, they're very intentional in how to adopt AI and where to invest. And at least what I'm seeing is customers are getting a lot of benefits from AI, meaning it's not just an additional cost.
They're able to accomplish more, they're able to drive the top line more. In some cases, they're able to be much more cost-effective. We use it a lot in SRE as an example, and we're driving better customer outcomes or better customer satisfaction. So I think we're already at a point today where customers are holding themselves accountable to make sure that the investment is driving returns, which is why this whole notion of token maxing, right, doesn't really work because it's not about how many tokens. It's about, am I getting to differentiated outcomes versus where I got to before? And do I have an ROI on the investment to get to those outcomes?
Peter O'Connor:
Yeah. Yeah. Oh man, pain. I remember my learning journey. I remember starting to have 10 to 12 sub-agents review specifications and tasks and then do it again after implementation. I was like, I get the same ROI if I did one or two. So why am I doing that?
Andi Gutmans:
I would just give an example. I had an idea that I though maybe my team could take a look at. I haven't actually written production code for quite a few years, as you can imagine. And so in the past, this will probably be a long conversation. Maybe I'd carve out one engineer that builds a prototype over three, four months. And I just built a prototype over the weekend, right, to just show the team like, "Hey, here's the art of the possible. It can actually be done." And right? And then it also reduces the risk because you can actually tease out some of the design risks very early upfront.
Now what I built wasn't production code. We have much better people than myself to actually make sure that it's good production code. But I think even when you think about the experimentation, being able to look at things very quickly, that's a huge saver for customers. Think about the amount of bad judgment calls, right, that you can shortcut in this matter versus having to wait for six months and you find out far too late that you've gone down the wrong path. So I think even on the path seeking side, there's just a huge benefit in AI in trying to get to these insights much, much earlier.
Peter O'Connor:
Yeah. Yeah, totally agree. Cost of experimentation's so cheap now. Let's just experiment more. I feel remiss if I didn't bring my platform people a good question or not. So I've been thinking a lot about how platform can enable AI adoption better and help achieve these outcomes. I wonder if your point of view, I thought of platforming very much as, okay, the internal devs are our product. The internal... Honestly, I would look at marketing as a problem. They're all people. What tools do you think need to be built internally for these companies to make sure that those team leads, those future team leads of AI agents are successful? So what's new in the platform world?
Andi Gutmans:
So first of all, I think it's a great question. I think about the agent as a persona we have to serve. The same way we had to serve business users, and data scientists, and data engineers. There's this new persona called agent. Of course, that persona is working on someone's behalf, but I think that is very important. I think there's multiple dimensions. Even if you think about what we had to do to enable humans, as an example, YouTube ran on MySQL, right? And then at some point that didn't scale. And then YouTube built a test and open source then, that was kind of a way to discard MySQL. But it still didn't remove the fact that it was too easy for a human to make an error on how to use the platform. And that's when basically Google said, "You know what? Let' s build a data system that takes the ownership onto the data system as opposed to on the developer." And then that's when we built Spanner, right?
So to a certain degree, that same intuition actually holds for agents because now we can throw Spanner at an agent and if it has to scale, it just works because we kind of built this for infinite scalability also for engineers. So I think about the same kind of things here, is how can we build the infrastructure, the best practices for this agentic era? So both human operators or orchestrators and agents, right, have the platform that they need. So a lot of the focus there is I mentioned trust, right? We need to build the right skills to have the right best practices, right? We need to make sure we have the right security and governance model. We need to make sure operators also understand the cost side and make sure they following that.
So there's a whole observability piece around agents and production. So I think you take the traditional enablement tools we build for human development, maybe doesn't translate one to one, but I think the buckets do, generally speaking, like security governance is a big bucket. Cost is a big bucket. Availability and scale is a big bucket, right? And some of the investments we've made in the past translate to this new world. And in some areas we have to invent new, but that's really the path we're on, is making sure that the agent as a persona is just as well served as we would think about the developer or the data science that we also need to serve today.
Peter O'Connor:
Yeah. And I'd also, I think we're serving more people too in general in a business world. You're not serving the developer anymore. You're serving... Everybody can participate now, which I think is great. There's no more gatekeeping or less gatekeeping.
Andi Gutmans:
Yeah. And by the way, the lines blurred the same way I mentioned with PHP with web development where suddenly non-developers were building web apps. I think the lines are blurring here because as we're building out data science, data engineering experiences that are agentic, we're like, "The business user could also use this. They don't actually have to know all the details of how we got the data, and how we trained the model, and how we deployed it." Right. So the reality is the lines are also blurry now across the human personas because you're getting to a phase where it's all about intent-driven outcomes. And as long as you understand the intent and the outcome you want to get to, and you have the right controls and management to make sure you're making progress in the right path, we can enable way more personas and individuals in the enterprise than we could in the past.
Peter O'Connor:
And the goal is to have the right ideas, right? We want right ideas to go out for business outcomes. So wherever they come from shouldn't matter.
Andi Gutmans:
Yeah.
Peter O'Connor:
So, and when we need an engineer because they orchestrate better, great.
Andi Gutmans:
Yeah.
Peter O'Connor:
Like you said, you hand it off and we'll get it done. So I think that makes sense. Last thing maybe on the platform thing, I was wondering, maybe you give me advice. I've thought about, we have a team internally that's specifically about our developer platform. They make the tools. I've never had a data engineer on that team or a data scientist. I'm wondering, would you advise me to actually start thinking of having that as part of a developer platform layer? What are your thoughts there?
Andi Gutmans:
It's a great question because we're also on the path of identifying that whole experience. But I do think there's a ton of value to make sure you have data scientists and data engineers. You probably don't need a lot because I think as we're building out these agentic experiences, we can really empower them. But data engineering is critical, right? Really making sure you're getting the right data into the right place. It's high quality, it's got its governance, it's ready to be activated both for analytics and AI. And then I think on the data science side, although we talk a lot about AI, right, a lot of business problems still hugely benefit and will benefit from machine learning, right, fraud detection, forecasting, and so on and so forth. So I do think it helps. Again, the tooling we're building, theoretically you could say the business user could actually be successful without the data scientist and data engineer, but it's really an amplification question, right?
Do you have that judgment, right? Are they able to understand how to get to those right outcomes in the same way I found my agent was going down the wrong path? Making sure you still have that judgment and folks who can help the organization get to the intended outcomes. So I think it's a good idea. Yeah, I would hire.
Peter O'Connor:
All right. All right. I've got to make a good decision. I appreciate it, so. Data science, you put a data science or a machine learning engineer on projects that are successful, it's just like gasoline, right? It just keeps it going and I love it, so. But it comes with intention. You have to be able to commit to it, so. I guess maybe a final thought I'd like us to maybe end on is thinking about if I was to advise somebody who's junior today, someone who's just in college, think computer science is where they want to go, what advice do you have for them? Where should they be aiming their learning and where should they be aiming their experience so that they're ready for the next world that's coming?
Andi Gutmans:
Yeah. Yeah. Well, I think there's probably two pieces. One is I think the traditional computer science fundamentals continue to matter a lot. And actually you've kind of seen the pendulum swing back to hardware now, right, after many years of everything being software. So I'd say whether it's the math side, or the system architecture, or you name it, I do think that is actually continuing to be very important. So I think it's very important to realize that computer science isn't just about creating code, it's really understanding how systems are built with technology. And so I think it continues to be very important to have that capability. The second one is, as I mentioned before, is really getting deep into how to do agentic coding, how to drive agentic outcomes, right, and really making sure that you're flexing that superpower because employers are going to be looking for people who are really able to drive that 10X impact.
And so I think it's a combination of fundamentals matter a lot, right, because otherwise you're going to use that as having your judgment on any projects and outcomes. But the second piece is making sure that you are really knowledgeable in how to use AI. And then probably the third piece is really making sure you understand the business problems. I think we're much more going into a direction now where we can actually give a high level business problem and have agents to do work on our behalf. And so it's almost like the MBA that was that second degree that a lot of folks in computer science went to study. You could almost think like, "Hey, MBAs actually matter even more now, right?" Or at least the notion of being very interested in the business problem because you have an ability to impact the business much, much more today than in the past as an individual because you can have all these agents doing work on your behalf.
Peter O'Connor:
You made me feel good again in the same call. Thank you. I advocate very heavily. For me, I think broad education is important. I won't say you have to have liberal arts education. I think there's a value to breadth of education. And to hear you say that computer science degrees still matter. I only know how the AI world works really well because I have those principles, right? I didn't have to think, "Oh, how does Java work in this? Or how does the Ada language do this?" No, it was just, "Hey, it's graph transversal. We're looking at just nodes." It's good stuff.
Andi Gutmans:
Yeah. And look, we're always going to be pushing the frontier in technology, right? That's not going to stop. Pushing the frontier does come from inherently understanding how things work.
Peter O'Connor:
For sure. Sure. Well, thanks. I really appreciate you talking with me today. Thank you so much. Andi Gutmans:
Yeah, no, thanks so much for having me. This was super fun.
Eira May:
Thank you for such a good conversation. You've been listening to Leaders of Code. Today we chatted with Stack Overflow Director of Platform Engineering, Peter O'Connor, and Andi Gutmans, who is VP of Agentic Data Cloud at Google. Where can folks find you, Peter? And then Andi, if they want to get in touch.
Peter O'Connor:
Yeah, I'm pretty easy to find on LinkedIn. There's not too many Peter O'Connor. So you could just look at me, search for Peter O'Connor LinkedIn, and you will find me. I'm the top hit.
Andi Gutmans:
Yeah, same here. LinkedIn is the best place to find me. And it's Andi with an I, so there are not that many out there. So you can find me there.
Eira May:
Great. And my name is Eira May. That's another hard name to find on LinkedIn. E-I-R-A M-A-Y. If you have any questions or feedback for me about this episode, feel free to reach out there. If you have suggestions for topics you want us to cover, guests you'd like to hear from, you can email us at podcast@stackoverflow.com. Thanks for listening and we will see you on the next episode of Leaders of Code.