Fin (formerly Intercom) is known for its customer service, chatbots and live chat products and has rebranded to be AI-first and acquired by Salesforce in 2026. Brian Scanlan, Senior Principal Systems Engineer at Fin, has been a part of the company for more than 12 years, and he knows all of the important workflows inside and out.
In the AI era, Fin managed to double its engineering capabilities through AI agents, which raised the question of how the company should measure success and growth. There needs to be a way to “count” the efficiency of AI agents, and Brian says the secret (at least for Fin) is in merged pull requests:
So we look at pull requests on our main applications and just count them. It’s very simple, it’s a little crude, it’s extremely gameable and it’s not by no means perfect, but I think if you’re in the right environment, you’ve got a high trust and people aren’t trying to game things, but you’re given guidance as well that, there’s no compromise around quality, you’re shipping real features and meaningful work, then it’s not a bad measurement.
In some areas, agents are doing all the work #
Just betting on people not abusing pull requests is one thing, but working with many AI agents makes for a rather unique environment. Brian says that in some areas, much of the work is already delegated to agents.
Brian explained that some fields, such as testing and fixing unreliable tests, are good areas for AI agents. This is because these tasks are predictable, and there is a relatively small amount of patterns that makes them unreliable. There’s also not a lot of risk:
And also, if things go wrong, you know, the worst case that’s gonna happen is you’re gonna block tests for a while or block deployments for a while, and that’s not too bad. So this is, like, a relatively low risk area and the patterns are well known. And so at the moment that’s fully delegated. Our agents just work in the background and fix all the tests.
For major features or incidents, Brian explained that agents are still working alongside humans in a supporting role. He pointed out that agents are being used throughout the process, but the main thinking and “figuring out what’s going on” still includes a lot of human brain power rather than RAM. Still, agent usage depends a lot on the project itself and its maturity, so some projects will be more agent-focused and others will have more human involvement. In general, the company’s direction is to go towards more delegation and more independent agents.
Scaling AI agents made Fin’s CI costs grow 10 times faster than the business #
Things are bound to break at this level and scale, so we asked Brian what was the first thing that broke during their efforts to scale AI usage from individuals to a company-wide effort. It turns out that their continuous integration (CI) system melted down in a few ways:
It started getting more unreliable because we were shipping more, we had more flaky tests, but also the costs went through the roof. We were auto scaling our CI environments and the cost was growing, you know, 10 times faster than the business was growing, and that’s not very comfortable. So we did the work to optimize our CI environment, to make sure our tests were actually valuable tests worth keeping and stuff like that.
The second thing was that when agents became popular and viral in the company, many people started pushing all sorts of things to GitHub, including to public GitHub repos. Brian said that there were “definite security problems” during that time, but many of these growing pains are now fixed and he is “worrying less about it now”.
React, learn, improve and write down #
When asked how to stop agents from just compounding bad decisions across the code base and projects, Brian admitted that those things are very hard to do, and that these issues remain largely unsolved in their environment:
Early on, we put a lot of work into improving our linting. We have a pretty good testing culture, I think, as well. We rely absolutely on our unit tests. We push to production hundreds of times a day. And so I think doubling down on the fundamentals, make sure that our linting is in good shape, our tests are in good shape, that people are still writing these things, that’s important. And then, you know, we still have humans in the loop on most of our code reviews.
Fin still has many safeguards in place and Brian says that everyone pays attention and makes sure none of the unreadable code and other problems get accepted. Still, there’s still “stuff that gets through”. In these situations, Brian says that the best advice is to react, learn, improve and then then put that guidance into the context so that the agent can’t make the same mistake again.
Organizations might be too conservative with AI #
Brian was also honest about how people adapt to using AI agents, saying that at the start, it wasn’t obvious who was going to get benefit out of using AI agents. For him, who’s been at the company for more than a decade, AI unlocked many benefits since he already had a lot of context and knew a lot about the systems in place. On the other side, more junior engineers also benefited from using AI since their onboarding was easier and faster, so they got more done in a shorter time.
When it comes to using AI, Brian said that many organizations are too conservative and cautious about being nice to their engineers:
I think change is hard and organizational change means real kind of thought and care when rolling it out. But that doesn’t mean you shouldn’t be ambitious. It doesn’t mean that you need to not upset anybody. I think to get real growth and real change and real adoption of AI, you have to make it very clear to have expectations that this is the future.
And honestly, I think the most important thing for people’s careers at the moment, where we are right now, is to be adopting these technologies, learning them and figuring out what the future of your own career kind of looks like.
Brian feels like the “careful approach”, which might’ve worked for others in the past, might not be the good strategy for solving the AI adoption issues.