Introducing Espresso AI, the First Neural Compute Optimizer
Espresso AI is the first compute optimization system built from the ground-up to leverage LLMs.
Read
"Sometimes someone confesses a sin in order to take credit for it." - John von Neumann
In all the talk about vibecoding CRMs and accelerating new software development, one thing that hasn't been mentioned is how much easier it now is to self-host open source software.
Espresso's systems are critical for our customers, so one of the first things we did as a company was set up OpenTelemetry. We pointed the results to Grafana Cloud so that we can get alerted on any blip that happens.
Given we are modifying and maintaining infra for a bunch of customers, this ended up requiring a lot of high cardinality metrics so that we could see issues on any single warehouse any of our customers had, and we also wanted to maintain enough logging to debug any issue that came up in prod from our algorithms.
As our customer count increased, so did our monitoring costs - this summer we were paying close to $10k a month. After pruning unnecessary telemetry, we got this down to about $7.5k/month without losing anything meaningful. But $7,500 is still real money, and Grafana's infra components happen to be open source, so one of the obvious things to look at was: could we just self-host this for much cheaper?
The answer will not shock you: yes, we can. Relying heavily on Codex, we migrated logs/metrics/traces to self-hosted High Availability Loki, Prometheus & Tempo (+ Kafka) and saved $6.5k/month, all with about 2 days of engineering effort.
We kept using the cloud-hosted dashboards, alerts, synthetics, which were still cheap for us, but we managed to move all the expensive chunks off.
A side benefit of this was that we can do more with metrics now that they're 85% cheaper; we used to avoid adding labels to things because the cardinality would blow up, but when things are cheap, you can monitor more.
To get on my soap box a bit, this is obviously a bit of a problem for open source software businesses.
We didn't use k8s for the first several years of this company's existence due to the perceived operational complexity, preferring to use ECS and focus on growing our top line revenue. But in the last 6 months, as coding agents have gotten better, we've started comfortably using EKS and running plenty of software on EKS. It's not perfect, and the coding agents don't get things right the first time, but with a staging environment, the blast radius can be limited pretty easily, even for infrastructure changes. And monitoring in particular is pretty exposed; you obviously need it to be reliable
So, how did I do the actual migration?
As a bonus, while I was working on this I noticed a bunch of problems to be fixed with our monitoring; we'd set most of it up by hand before LLMs were good, but since nobody was really an expert at PromQL we'd made mistakes and Codex did a great job of flagging them. A decent chunk of the time on this migration was spent just paying attention to edge cases in our monitoring that we'd like to fix and not on the self-hosting itself.
So, what does this mean for open source software? Open source companies have provided a huge benefit to the software industry, and personally, I feel bad for them. Building a business on top of open source wasn't easy in the best of times; today, charging on top of compute just for hosting seems unsustainable. When your code is free, the value of a managed platform is exactly the cost of recreating it; and now that LLMs are driving that cost to zero that value is evaporating.
It's honestly hard to see a path forward for companies built on top of OSS.
Subscribe to our newsletter. Get exclusive insights delivered straight to your inbox.