{"slug": "what-we-optimise-for-when-we-reach-for-an-llm", "title": "What we optimise for when we reach for an LLM", "summary": "Bellroy CTO Andrew Bell argues that LLMs should be evaluated by long-term ROI, not just code generation speed, and that using them to draft boilerplate is wasteful while using them to implement features from well-defined types is valuable. Bellroy, a profitable physical-product company with an internal engineering team, treats technology as a profit centre, prioritizing maintenance costs and business collaboration over shipping code quickly.", "body_md": "**Disclaimer: I wrote this blog post entirely by hand, but used Claude to review and provide feedback\non it before seeking human review.**\n\nMost of the blog posts I read about LLMs are either gushing about a new era of infinite productivity, or bemoaning the enshittification of the entire software industry. I don’t often come across posts that take the even-handed view of evaluating these tools like any other tool, so I thought it was time to make a contribution to the discussion.\n\nIf I discovered one of our engineers repeatedly typing out boilerplate for JSON decoders, I’d be disappointed. If I discovered them spending an afternoon hand-crafting types that encode our business invariants into our systems and then letting LLMs draft the feature based on those types, I’d be delighted. The aim of this post is to communicate why I’d react differently in each circumstance.\n\n## Our context (and why yours may differ)\n\nThe lens I’m bringing to this has been shaped by the kind of company Bellroy is and the kind of CTO I strive to be. Bellroy is not a software company; it’s a physical-product company that happens to have technological proficiency as one of its core competencies. Since the very early days, Bellroy has employed an internal engineering team - initially to focus on automating the order fulfilment pipeline, but today to build and maintain an entire bespoke e-commerce platform. Bellroy has been continuously profitable since early in its history and isn’t sprinting towards the end of a funding runway. This substantially influences the way decisions are made day-to-day. One of Bellroy’s goals is sustainable, ethical profitability. To meet that goal - among other things - we must be evaluating choices based on long-term Return On Investment (ROI). That evaluation does a lot of heavy lifting: it means that the benefit of shipping more code more quickly must be offset against the cost of maintenance. This philosophy neatly fits my approach to running the Technology team at Bellroy: we must operate as a profit centre, not a cost centre.\n\n## Engineering as a profit centre\n\nSome concrete examples of “profit centre” thinking in action:\n\n[choosing statically typed functional programming languages](/our-technology-stack-and-how-we-got-here.html)to eliminate entire classes of production issues- prioritising regular dependency updates of every single code repository we manage to avoid “drop everything” zero-day vulnerability scenarios and dependency rot\n- employing\n[a “fixed time, flexible scope” project methodology](/using-shape-up-what-works-what-weve-changed-whats-next.html)to eliminate project “overhang” while still delivering tangible, measurable outcomes - using internally-built probabilistic\n[Net Present Value (NPV)](https://www.investopedia.com/terms/n/npv.asp)calculator tools that use Monte Carlo simulations to evaluate project benefits (another blog post, perhaps), and using those valuations to drive prioritisation decisions and push back on scope\n\nThis last point is important - we rely on our engineers not only to ship features and maintain our\ncodebases, but also to help project stakeholders to identify what they actually *need* as opposed to\nwhat they asked for. Technology does not operate as a silo in Bellroy; we collaborate with\nother functional teams to deeply understand their processes - and how changes to their processes\nimpact the rest of the business - before we write a single line of code.\n\nThis is where we find the current generation of LLMs still wanting. The “everyone’s a builder now” mentality sounds great if you believe that generating code is all benefit and no cost, or if it lets someone adept at business analysis - who has already validated the ROI of a particular change - bypass the engineering department and ship the thing themselves. That perspective ignores the opportunity cost of the next-best-thing that person could have been doing with their time, and the maintenance cost of the thing that the Technology team now maintains but had no part in designing.\n\n“But Mike”, I hear you say, “surely the LLMs can do the maintenance and bug fixes for all those\nvibe-coded features as well?” Well, yes, you’re (partially) right. We *do* use LLMs for maintenance\nand bug fixes. But what we don’t do is rely on LLMs to resolve those rare-but-critical production\nissues at 3am on a Sunday morning. With [so many horror stories](https://www.theguardian.com/technology/2026/apr/29/claude-ai-deletes-firm-database)\nfloating around the internet of what unmonitored LLM tools can do with access to production\ninfrastructure, we are unwilling to give them (write) access to ours. With this responsibility still\nin the hands of the humans, the humans need to have a good mental map of our systems architecture\nand our infrastructure. We need to be tracking how big the on-ramp is for new staff. And that’s\nexactly what we’d give up by ceding all engineering activities to the LLMs.\n\n## What maintainability actually means to us (and Claude!)\n\nWe use [Claude](https://claude.ai/) at Bellroy, across the entire company. And it’s great for so\nmany things.\n\nOne thing we’ve noticed about Claude-generated code is that Claude tends to want to provide verbose comments on most changes it makes. Despite explicit instructions not to do so, it uses these comments to document what the code does, or adds context from the prompts, or refers to previous states of the code that become irrelevant the second a change ships.\n\nFor us, this is bad practice. Code comments should only provide context that is current and that is\n*not* derivable from just reading the code - adding a comment that just describes what the code does\nincreases the risk of the documentation diverging from the implementation, which just adds potential\nconfusion for no real benefit. Our function, type and variable names should do enough of the work to\nallow an engineer to get oriented.\n\nWe use comments only to provide the “why” of a particular implementation choice so the *intent* -\nthe “should” of the change - is preserved, because it is that information that is hardest to\nreverse-engineer from the code one year from now. We take a similar approach to pull request\ndescriptions, using a template that asks for a problem statement (what problem exists prior to this\nchange), a solution description (how this particular change solves the problem) and an account of\nwhat the engineer has done to verify that the problem is actually resolved. Without engineers having\ntaken part in the scoping process and understanding the size and shape of the business problem, they\ncan’t provide that context.\n\nAnother deliberate decision we’ve made to assist with maintainability is to work in a [monorepo](https://monorepo.tools/).\nThis means that in our commit history, related changes are often shipped together. Coupled with the\npractices described above, this - at least in theory - should mean that an engineer working in a\nparticular area should be able to answer 3 questions:\n\n- What does this particular area of the code do?\n- Why - from a business perspective - does it do it? And why in this particular way?\n- What else had to change to make this solution workable, and <repeat steps 1-3 for those other related areas>\n\n## Where LLMs earn their keep\n\nIronically by having our humans care about and document this stuff we can provide better context to LLMs. This makes them more efficient at feature building, bug fixing and triage.\n\nLLMs are great, and you should use them! But - in my opinion - they are *best* employed producing\nwhat can be machine-verified. There are still many things better left to human judgement.\n\nAn example: writing a mechanical JSON encoder and decoder to match a wire format, repeatedly and by\nhand, is not something any human engineer should be doing in 2026. Declining to automate that is\npaying a tax for no good reason. In Haskell, we have a standard practice for testing such things - a\n[golden test](https://hackage.haskell.org/package/tasty-golden)\nwith real examples to anchor you to reality, and [roundtrip property tests](https://hackage-content.haskell.org/package/hedgehog-1.7/docs/Hedgehog.html#v:tripping)\nusing hundreds of randomly generated values each time to catch edge cases. Provide the LLM with the\nreal examples and let it do the work of writing the encoders, decoders and tests (and for the\nrecord, there are better tools than LLMs for deriving correct marshalling and unmarshalling\nstrategies for JSON wire formats from [OpenAPI](https://www.openapis.org/) specifications - I merely\nuse this as an concrete example of boilerplate code).\n\nA rule like “all currency amounts on a sales order are in the same currency, across its lines and\nassociated payments” is something you can - and we do - encode in our type system. This means a\nprogram that mixes currencies in one order is *unrepresentable in valid code*; it simply won’t\ncompile.\n\n(An aside for the Haskellers - here’s a sample of Haskell code showing how that’s achieved using a\nphantom type index on our internally-defined `Discrete`\n\ntype, a [singleton type](https://hackage.haskell.org/package/singletons)\nto reflect it at runtime and an [existential](https://wiki.haskell.org/Existential_type) unwrapped\nat serialisation boundaries - a simplified version of a mechanism from [ safe-money](https://hackage.haskell.org/package/safe-money)\nand probably worthy of its own blog post):\n\n```\nnewtype Discrete (currency :: Currency) = Discrete Integer\n\ndata SomeDiscrete where\n  SomeDiscrete :: forall currency. Sing currency -> Discrete currency -> SomeDiscrete\n\n-- the `currency` type argument enforces a common currency across attributes\ndata OrderItem (currency :: Currency) = OrderItem\n  { code :: AccountingSystemIdentifier,\n    amount :: Discrete currency,\n    description :: NonEmptyText,\n    quantity :: Natural1,\n    taxAmount :: Maybe (Discrete currency),\n    taxCode :: TaxCode\n  }\n\nfromBaseUnit :: forall (currency :: Currency). Integer -> Discrete currency\nfromBaseUnit = Discrete\n\ninstance FromJSON (OrderItem currency) where\n  parseJSON = Aeson.withObject \"OrderItem\" $ \\obj -> do\n    -- ...\n    amount <- fromBaseUnit <$> obj .: \"amount\"\n    -- ...\n```\n\nBut it’s not obvious to a machine - and frankly most humans, unless they’ve done the research -\n*why* you’d want to do such a thing. For some businesses, like ours, it’s a fundamental truth\nbecause of the way we’ve set up our platform. For others, they may be far more flexible about how\nthey take payments. And it may change in the future, which should influence exactly how you\nimplement it. And all of those factors I just described are not the sort of thing an LLM will intuit\nby default (yet), unless the person operating the LLM goes to great lengths to include that in the\ncontext. By having that rule encoded - with care and deliberate design - in the type system, you’re\ncreating an environment where you can delegate smaller coding tasks to the LLMs with confidence.\nWhere the fundamentals of the business are unbreakable contracts the LLM has to respect.\n\nThis combination of determinism and non-determinism is where you can have your cake and eat it too.\nConsider the problem of wanting to have an LLM triage an email inbox and auto-respond. Emails can\ncome from anywhere and go to anywhere, and I would lose sleep at night just plugging Claude into\n[Gmail](https://workspace.google.com/intl/en_au/products/gmail/) with a set of instructions and\nsaying, “have at it”. What I’m much more comfortable with is having a workflow automation tool like\n[n8n](https://n8n.io/) pull each email requiring triage using the Gmail API, feed the email headers\nand body into an LLM whose only task is to choose the action to take from a finite list, and then\nhaving n8n execute that action via the Gmail API. The humans retain complete control over the input\nand possible outcomes, while the LLM is doing the previously-impossible task of dealing with every\npermutation of email content you can throw at it and turning that into something sensible.\n\n## A tool, not a strategy\n\nClaude has been adopted across Bellroy and most Bellrovians use it every working day. They’re using\nChat, Cowork and Code, or plugging in other tools into Claude’s API, such as n8n. I spend a good\nchunk of my working hours each week trying to work out ways we can more effectively leverage this\ntechnology right across the company - not just in the Technology team - and at the time of writing\nwe’re actively hiring an [AI Enablement Lead](https://bellroy.com/careers) (among other roles -\nwe’re growing!). These tools are game-changing. I intend for Bellroy to remain an AI-forward\norganisation and we currently have more great ideas than we have people to execute on them.\n\nWhat these tools have done is change the distribution of costs, and that’s going to mean different\nthings to different companies depending on their strategy. The cost of building things -\nideas, processes, software tools - has clearly gone down, and to some extent the cost of maintenance\nhas as well. But I would argue the cost of maintenance has not reduced to the same extent that the\ncost of building has, and in order to effectively manage that maintenance burden you need humans to\nunderstand the why and how of the things we build. For the time being - to avoid accumulating\n[cognitive debt](https://www.media.mit.edu/publications/your-brain-on-chatgpt/) - Bellroy’s human\nengineers will still be writing code by hand… some of the time.", "url": "https://wpnews.pro/news/what-we-optimise-for-when-we-reach-for-an-llm", "canonical_source": "https://exploring-better-ways.bellroy.com/what-we-optimise-for-when-we-reach-for-an-llm.html", "published_at": "2026-08-29 10:54:09+00:00", "updated_at": "2026-08-29 11:18:31.459042+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products"], "entities": ["Bellroy", "Andrew Bell"], "alternates": {"html": "https://wpnews.pro/news/what-we-optimise-for-when-we-reach-for-an-llm", "markdown": "https://wpnews.pro/news/what-we-optimise-for-when-we-reach-for-an-llm.md", "text": "https://wpnews.pro/news/what-we-optimise-for-when-we-reach-for-an-llm.txt", "jsonld": "https://wpnews.pro/news/what-we-optimise-for-when-we-reach-for-an-llm.jsonld"}}