# My Little AI Factory > Source: > Published: 2026-09-08 13:59:18+00:00 In this post, I’m documenting my journey of building [BER](https://ber.run), my AI factory. After using agents to migrate more than 1,000 domains and studying over 10,000 AI-coding sessions, my biggest lesson is simple: generating output is easy; producing trustworthy outcomes and proving their business value is hard. ## What is an AI factory? In the near future, companies will run AI factories to automate work; each department will run its own factories, where colleagues collaborate with AI. I predict that **much of today’s office work will shift toward defining high-level specifications and redesigning business processes in natural language**. Becoming truly AI-first will require thinking clearly, redesigning existing processes, and reevaluating vendors based on whether they can integrate with these factories. That shift sounds easy, but making it work will require defining the process in the simplest terms with clear tests, success criteria, and ways to handle edge cases. If I can give you one takeaway after spending thousands of hours thinking about and building these systems, it is this: AI loves overcomplicating things. Our job will be to radically simplify everything and have a laser focus on solving the task at hand. ## My journey ### Component 1: Superglue: Automating workflows with AI agents My journey started with the idea of building a cloud infrastructure autopilot. I started the project in the spring of 2024 and managed to build a working prototype in a few months, with the promise of using native GitHub workflows (issues, PRs, code) to manage your infrastructure using natural language. It took me a few months to realize that the buyers I’d identified, despite having the budget, did not feel the pain I’d imagined. I also received a lot of pushback because my prospects saw enormous risk in rolling out a system like this, which I was unable to derisk for them at the time. In fall 2024, I started rebuilding the foundations of the orchestrator. The goal was clear from day one: build a flexible system that could automate any business-critical workflow using autonomous agents. Back then, the technology and my knowledge were limited, but I managed to prove the system worked for real use cases. The system successfully delivered an enterprise domain migration project for a large manufacturing company. The project in numbers: - Interacted with 66 international business entities - Lifted and shifted DNS configurations for **1,008 domains** - Integrated with more than 35 source DNS providers - Discovered approximately **4,500 company-owned domains** whose owners we could not identify within the organization - Achieved an **overall error rate below 1%** - Ran the migration process for six weeks - Spent four months on research and preparation I built this system before MCP was released; agents had no native tool execution, and the term “context engineering” had not yet entered common use. I ended up inventing some really stupid workarounds for all these missing capabilities. **The project succeeded only because I reviewed every change and fixed the model’s mistakes by hand.** It was extremely tiring, but what I learned fueled my research and work over the next 18 months. In the big picture, **I think AI should expose repetitive tasks and build code to automate them. The next step should be spending tokens once and enjoying the solution multiple times.** ### Component 2: rubBERduck: Operating legacy systems with AI During the delivery of the domain project, the customer’s SOX change-management process required me to fill out a change request for each domain before the DNS cutover. This system had no API I could use to integrate with Superglue, only a prehistoric web interface working in Internet Explorer. (This is a bit of an exaggeration, but you get the point.) It was clear that if I wanted to automate work in a modern business, I would need a component that could interact with legacy systems only available through a website or desktop app. As I said at the beginning of the post, companies will reevaluate their vendors. I’m certain a lot of legacy systems and vendors, especially in government, can’t be replaced, leaving computer use as the only viable way to automate interactions with them. rubBERduck’s code was 100% written by AI. This opened a new door. ### Component 3: ARX: Monitoring AI agents After delivering the domain project, I started building a lot of vibecoded prototypes for various business problems, taking detailed notes on the issues I faced. In November 2025, someone shared the now-famous [METR study](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) with me. The researchers asked 16 experienced open-source developers to fix issues in their own repositories, randomly allowing or disallowing AI tools for each task. **The developers expected AI to make them 24% faster; instead, the study found that they were 19% slower.** Even after completing the study, they still believed AI had made them 20% faster, while their estimates for working without AI were fairly accurate. Those figures were a snapshot of early-2025 tools in one particular setting; I don’t treat them as a verdict on today’s tools. What stayed with me was **the gap between perceived and measured productivity**. That gap became the starting point for [ARX](https://arx.ber.run): I wanted to measure it in my own coding sessions. The new project had three goals: - Understand my real productivity using coding agents - Identify ways to improve in order to produce better code more cheaply - Measure the real ROI: **task -> tokens -> code -> rewrites -> business value** For the past nine months, we’ve used this system to record and store metrics from over 10,000 coding-agent sessions across 10+ engineers. A Dutch neobank was asking similar questions, so they deployed ARX to a handful of engineers. The result of the collaboration was clear: although the monitoring and data extraction worked, the analysis did not yet provide value at an individual level. To understand AI-assisted development better, we needed to broaden the scope of the data we analyzed. ### Component 4: BERBench: Benchmarking AI agents At the beginning of 2026, I started talking to people more deliberately. I conducted **over 70 hours of user interviews and live AI-coding sessions**. My main observation was that everyone uses these tools differently, and the options are endless: - **Harnesses:** Claude Code, Copilot, Codex, OpenCode, Pi, etc. - **Models:** Sonnet/Opus/Fable, GPT, Kimi, Grok, Gemini - **Efforts:** medium, high, extra-high - **Skills:** Superpowers, Caveman, Grillme, … 10k more - **Plugins/MCPs:** RTK, Context Mode, Ralph, claude-mem, Context7, PAL MCP, Sequential Thinking, and thousands more - **Orchestrators:** HumanLayer, Herder, and hundreds more - **Customization:** AGENTS.md files, custom skills, and custom workflows (research → spec → plan → build) On top of all that, everyone has their own codebase and custom development environment. I’ve also learned that many people don’t use English to interact with agents. They say they can write more expressive instructions in their native language. **Any one of these can significantly affect the quality and cost of an agent’s output.** Depending on the interviewee’s sales skills, I tried their setup. I ran the same set of tests on each configuration I liked, then used ARX to extract and compare session telemetry. One thing remained constant: no matter which magic setup I used, **I was unable to produce code I could trust without exhaustive human review**. And oh boy, I’ve produced a lot of code. It quickly became obvious that humans would not be able to review all agent-generated code at scale. It was a mess. This led to [BERBench](https://berbench.com). I needed a deterministic way to compare these setups’ success rates and the agents’ trajectories: the steps they took, how they reasoned, and the conclusions they reached. The project’s goals were: - provide a simple way to define any setup - run tests on my own code - compare success rates, token consumption, and generated code In the big picture, **I see continuous benchmarking as the foundation of self-evolving agents**. ### Component 5: MUX: Orchestrating AI agents MUX is a byproduct of BERBench. While running benchmarks, I encountered three problems that cried out for a solution: 1. I needed a way to run multiple agent tasks securely and in parallel on a remote machine. This execution layer will underpin the entire factory: not just sandboxing, but orchestration and scheduling. Kubernetes seems like the obvious foundation. 2. For coding agents to operate effectively, they need to have the full dev environment, with all the dependencies working inside these remote sandboxes. 3. Running benchmarks at scale burns a lot of tokens. I needed a way to use multiple seat-based subscriptions from each provider. Running coding agents on every engineer’s machine will soon hit its limits. Organizations will require a centralized agent execution system where their employees can run their AI work. This will help them standardize workflows and models, govern agent access, and manage costs better. This will require the next component. ### Component 6: LEX: Governing AI agents LEX started as a simple rule engine attached to coding agents’ hook endpoints. It could detect and block: - Exposure of PII - Attempts to read secrets - Unauthorized network access - Unauthorized changes to production environments I soon realized that this approach would not scale for two reasons: 1. Agents are rewarded for solving the task at hand. If deleting a file helps them complete it, they will find a way, and there are too many ways to encode that action for a simple rule to catch them all. 2. A block-everything approach won’t work in a corporate setting. Some employees will need elevated access to production systems, just as they do today. I see LEX’s future as **a self-learning tool gate with flexible role-based access control (RBAC)**. ### What’s next You’re probably asking why I’m not focusing on a single narrow problem or pain point. In reality, these components were all attempts to identify a wedge and understand whether there was a real need for these capabilities. While evaluating them, I discovered new problems. Once we launch BER as a company, we will need all of these components internally in order to build a true AI-native company and, ideally, automate our own processes from day one. It is also quite obvious that releasing all components together as a unified platform is not viable today. No company will adopt this, so I’m continuing to throw wet toilet paper at the wall, gathering feedback and trying to improve my offering. **We are still at the very beginning.** I compare the state of this space to AWS around 2010: the core building blocks were available, but automating provisioning still meant writing Python, and tools such as Terraform and modern state management did not yet exist. If you think about this whole problem, the biggest question still remains: **How does spending $1,000 on tokens affect our business? How much does it increase revenue or reduce expenses? Was that $1,000 worth spending?** Today, no one can reliably answer this question. Those who can will win their markets. **BER’s ambition is simple: help companies turn every $1 spent on AI tokens into $100 in additional revenue or cost savings and prove where that value came from.**