{"slug": "ask-hn-why-don-t-llm-apis-have-a-first-class-test-mode", "title": "Ask HN: Why don't LLM APIs have a first-class test mode?", "summary": "A Hacker News user argues that OpenAI, Anthropic, and other LLM providers should offer official API-compatible test endpoints that don't invoke models or consume billable tokens, similar to Stripe's test mode. The user notes that companies must build and maintain custom mocking services to stress-test their own infrastructure without incurring token costs, and suggests features like deterministic responses, configurable latency, and error simulation. The post asks how other teams handle this challenge, citing homegrown mocks, generic HTTP mocking, or record/replay as current approaches.", "body_md": "Context: At work, we’re getting ready to stress-test a chatbot for scalability.\n\nOne fairly obvious issue came up: if our load tests exercise the real OpenAI/Claude APIs, a scalability test can quickly turn into a token-spending test.\n\nFair enough. We shouldn’t burn real inference just to test whether our own gateways, queues, WebSockets, streaming paths, retries, etc. can handle load.\n\nThe proposed solution was to mock all communication between our backend and the LLM provider.\n\nAlso reasonable.\n\nWhat surprised me was the next step: we have to build and maintain that mocking service ourselves.\n\nWe can certainly do that. But should every company integrating with LLM APIs have to reinvent this?\n\nStripe solved a similar developer-experience problem years ago. They provide test mode, test data, test helpers, and even stripe-mock. It isn’t intended to perfectly reproduce Stripe’s backend behavior, but that’s okay. For many tests, you just need something API-compatible and predictable.\n\nI’d love to see OpenAI, Anthropic, and other LLM providers offer something similar: an official API-compatible test endpoint that doesn’t invoke a model or consume billable tokens.\n\nIdeally it could support things like:\n\n* deterministic canned responses * streaming responses * configurable latency / time-to-first-token * configurable token counts * tool-call responses * 429s, 5xx errors and timeouts * malformed/interrupted streams * rate-limit simulation\n\nThe goal wouldn’t be to benchmark the LLM provider. You’d still need the real API for that. The goal would be to stress-test everything around the model without paying for thousands or millions of unnecessary inference calls.\n\nWhat’s slightly ironic is that both OpenAI and Anthropic appear to use OpenAPI-based mock servers in their own SDK test suites. But, as far as I can tell, neither exposes that concept as a first-class public service for customers.\n\nAm I missing something?\n\nFor teams running LLM applications at scale, how are you handling this today — homegrown mock server, generic HTTP mocking, record/replay, or just putting a budget cap on real API load tests?\n\nComments URL: [https://news.ycombinator.com/item?id=49556909](https://news.ycombinator.com/item?id=49556909)\n\nPoints: 1\n\n# Comments: 2", "url": "https://wpnews.pro/news/ask-hn-why-don-t-llm-apis-have-a-first-class-test-mode", "canonical_source": "https://news.ycombinator.com/item?id=49556909", "published_at": "2026-09-03 20:59:59+00:00", "updated_at": "2026-09-03 21:23:33.061747+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["OpenAI", "Anthropic", "Stripe", "Hacker News"], "alternates": {"html": "https://wpnews.pro/news/ask-hn-why-don-t-llm-apis-have-a-first-class-test-mode", "markdown": "https://wpnews.pro/news/ask-hn-why-don-t-llm-apis-have-a-first-class-test-mode.md", "text": "https://wpnews.pro/news/ask-hn-why-don-t-llm-apis-have-a-first-class-test-mode.txt", "jsonld": "https://wpnews.pro/news/ask-hn-why-don-t-llm-apis-have-a-first-class-test-mode.jsonld"}}