{"slug": "building-production-ai-systems-part-3", "title": "Building Production AI Systems (Part 3)", "summary": "An engineer building production AI systems with OpenRouter shares lessons on handling provider failures, observability, and streaming resilience. The developer emphasizes that users blame the application, not the provider, when AI features break, and recommends standardized error handling, request IDs, and timeout limits.", "body_md": "Running OpenRouter in Production: What Breaks, What Works, and What I’d Do Differently\n\nGetting an LLM to respond to your first prompt is easy.\n\nKeeping it reliable in production?\n\nThat’s where things become interesting.\n\nOne thing I’ve learned building software is that users don’t care whose fault it is when something breaks.\n\nIf your AI feature fails because OpenAI is down…\n\nOr Claude is rate-limited…\n\nOr your network connection decides to disappear for five seconds…\n\nYour users won’t blame the provider.\n\nThey’ll blame your application.\n\nThat’s why integrating an AI model isn’t enough.\n\nYou need to engineer for failure, lets talk about how.\n\nOne of the biggest advantages of OpenRouter is that it standardizes responses from different providers.\n\nWithout it, you often end up writing provider-specific error handling.\n\ntypescript\nif (provider === “openai”) {\n…\n}\n\nif (provider === “anthropic”) {\n…\n}\n\nif (provider === “google”) {\n…\n}\nThat gets messy very quickly.\n\nWith OpenRouter, your application talks to one API contract.\n\nWhich means your error handling becomes predictable.\n\ntypescript\ntry {\nconst response = await client.chat.completions.create({…});\n} catch (error) {\nlogger.error(error);\n}\nSimple.\n\nYour code shouldn’t care which provider failed.\n\nIt should care how your application responds when one does.\n\nOne mistake I see quite often is developers only logging the error message.\n\nThat’s rarely enough.\n\nWhen something fails in production, you want context.\n\nLog things like:\n\nA single log entry should tell you the complete story without having to reproduce the issue.\n\nFuture you will appreciate it.\n\nImagine a customer reports:\n\n“The AI stopped working around 2 PM.”\n\nWithout observability, you’re guessing.\n\nWas it your backend?\n\nOpenRouter?\n\nClaude?\n\nGemini?\n\nA timeout?\n\nA deployment?\n\nGood observability removes the guesswork.\n\nOpenRouter provides request tracking that allows you to match requests from your application with what happened inside their dashboard.\n\nInstead of spending hours wondering where things went wrong, you can trace a single request from your backend logs all the way through the gateway.\n\nThat’s incredibly valuable once real users are involved.\n\nRequest IDs are underrated\n\nEvery request should have an identity.\n\nWhether you’re using OpenRouter’s request identifiers or generating your own correlation IDs, always include them in your logs.\n\nImagine one user reports:\n\nMy response never arrived.\n\nWithout a request ID, you’re searching through thousands of log entries.\n\nWith one, you can immediately trace:\n\nProduction debugging becomes dramatically easier.\n\nStreaming isn’t always as straightforward as it looks\n\nStreaming responses make AI applications feel much faster.\n\nInstead of waiting ten seconds for a complete answer, users start seeing words appear almost immediately.\n\nGreat user experience.\n\nBut there’s a catch.\n\nNot every provider streams responses in exactly the same way.\n\nAlthough OpenRouter normalizes much of this behavior, you’ll still want to build frontend parsers that are resilient.\n\nDon’t assume every chunk arrives perfectly.\n\nDon’t assume every event contains text.\n\nDon’t assume the connection won’t terminate unexpectedly.\n\nYour frontend should gracefully handle incomplete streams instead of crashing halfway through a response.\n\nTimeouts are not optional\n\nOne of the easiest ways to create a poor user experience is to let requests hang forever.\n\nSometimes providers are slow.\n\nSometimes networks are unreliable.\n\nSometimes things simply break.\n\nYour application should know when to stop waiting.\n\nSet reasonable timeout limits.\n\nIf a request takes too long, cancel it.\n\nYour users would rather receive a helpful message after fifteen seconds than stare at an infinite loading spinner.\n\nAn application that knows when to give up usually feels faster than one that waits forever.\n\nThis is one of my favourite engineering principles.\n\nJust because one provider fails doesn’t mean your feature should disappear completely.\n\nImagine your primary reasoning model becomes unavailable.\n\nInstead of returning:\n\nSomething went wrong.\n\nWhy not automatically switch to a faster fallback model?\n\nMaybe the answer isn’t quite as detailed.\n\nMaybe it isn’t as creative.\n\nBut your user still gets a response.\n\nThat’s infinitely better than an error screen.\n\nGraceful degradation isn’t about perfection.\n\nIt’s about maintaining functionality when perfection isn’t possible.\n\nBuild retries carefully\n\nRetries sound simple.\n\nUntil they aren’t.\n\nIf a request fails because of a temporary network issue, retrying makes sense.\n\nIf it fails because the provider is overloaded, retrying after a short delay might work.\n\nBut if the request is invalid?\n\nRetrying it five times won’t magically fix bad input.\n\nA good retry strategy should:\n\nRetries should improve reliability.\n\nNot create more traffic.\n\nFinal thoughts\n\nOne thing AI development has taught me is that choosing the right model is only a small part of building reliable applications.\n\nThe real challenge is designing systems that continue working when models slow down, providers become unavailable, or networks become unreliable.\n\nBecause eventually…\n\nSomething will fail.\n\nThe question isn’t whether it happens.\n\nThe question is whether your users notice.\n\nIf they don’t, you’ve probably built your system well.\n\n*Enjoying the series? Let me know in the comments.*\n\n*Ooh and do not forget to connect with me so you do not miss out on the next parts of the series.*", "url": "https://wpnews.pro/news/building-production-ai-systems-part-3", "canonical_source": "https://dev.to/franklyn_nmesoma/building-production-ai-systems-part-3-bnf", "published_at": "2026-07-24 05:45:00+00:00", "updated_at": "2026-07-24 06:01:48.846776+00:00", "lang": "en", "topics": ["ai-infrastructure", "developer-tools", "ai-products"], "entities": ["OpenRouter", "OpenAI", "Anthropic", "Google", "Claude", "Gemini"], "alternates": {"html": "https://wpnews.pro/news/building-production-ai-systems-part-3", "markdown": "https://wpnews.pro/news/building-production-ai-systems-part-3.md", "text": "https://wpnews.pro/news/building-production-ai-systems-part-3.txt", "jsonld": "https://wpnews.pro/news/building-production-ai-systems-part-3.jsonld"}}