{"slug": "ask-hn-how-do-we-measure-software-in-llm-era", "title": "Ask HN: How do we measure software in LLM era?", "summary": "A Hacker News user laments the difficulty of measuring software quality in the LLM era, citing high variability in metrics like accuracy, cost, and latency even with the same model and provider. The post questions how developers can triage issues or assure users when probabilistic AI components introduce unpredictable behavior.", "body_md": "A bit of a rant. Sorry!\n\nWith the probablistic pluggable 'brain' existing in parts of the solution how are you measuring anything is better or worse?\n\nI am at a loss to quantify whether anything is improving or worsening anything. It probably is also because of the various metrics that keeps popping up\n\n* Accuracy\n\n* Cost of running\n\n* Context\n\n* Size\n\n* Time\n\n* Turns\n\nall these vary in a large band even with the same 'brain' on the same 'provider'. It is not so different than a database running strained under load - drawing from a simpler times. But here, which elastiuc band is getting pulled in which direction is worse than playing 3D Tetris.\n\nThen there is the harness side variability of tool choices. Which seems to be the only knob the developer these days seems to have some control over. Other than the deterministic parts of the system.\n\nHow are we even going to triage a ticket with so many variabilities. In a runtime. That apparently is still called a software.\n\nDo we just tell the users that you are on your own and whatever you need to solve is between you and your brain of choice?\n\nWhat are you doing?\n\nComments URL: [https://news.ycombinator.com/item?id=48696916](https://news.ycombinator.com/item?id=48696916)\n\nPoints: 1\n\n# Comments: 0", "url": "https://wpnews.pro/news/ask-hn-how-do-we-measure-software-in-llm-era", "canonical_source": "https://news.ycombinator.com/item?id=48696916", "published_at": "2026-06-27 10:18:44+00:00", "updated_at": "2026-06-27 10:35:30.000508+00:00", "lang": "en", "topics": ["large-language-models", "ai-products", "ai-tools"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/ask-hn-how-do-we-measure-software-in-llm-era", "markdown": "https://wpnews.pro/news/ask-hn-how-do-we-measure-software-in-llm-era.md", "text": "https://wpnews.pro/news/ask-hn-how-do-we-measure-software-in-llm-era.txt", "jsonld": "https://wpnews.pro/news/ask-hn-how-do-we-measure-software-in-llm-era.jsonld"}}