{"slug": "i-built-a-multi-provider-llm-router-for-my-ai-worker-here-s-what-i-learned", "title": "I Built a Multi-Provider LLM Router for My AI Worker - Here's What I Learned", "summary": "A developer built a multi-provider LLM router for their distributed data cleaning system AuraFlow AI, which uses LangGraph agents and BullMQ. The router, called LLMRouter, maintains a configurable fallback chain across providers like Gemini, OpenAI, Claude, and Groq, automatically skipping providers without API keys and retrying on failures. The developer learned that hardcoding a single LLM provider creates a single point of failure, as experienced when Gemini's rate limit halted the system.", "body_md": "When I started integrating LLMs into my side project AuraFlow AI, I made the same mistake most backend engineers make: I hardcoded a single provider.\n\nOne week in, Gemini free tier hit its rate limit at 11 PM while I was testing. Everything stopped. I had to manually swap the API key, restart the worker, and lose 20 minutes of debugging momentum.\n\nThat was the last time I let a single LLM provider be a single point of failure.\n\nAuraFlow AI is a distributed data cleaning system I built for my portfolio. The architecture is straightforward:\n\n```\nPOST /jobs (NestJS/Fastify/Bun)\n    → BullMQ job pushed to Redis\n        → Python LangGraph worker picks up job\n            → Parser Agent: clean raw/malformed data using LLM\n            → Validator Agent: verify output, loop back if invalid\n        → HTTP callback with retry + idempotency\n    → Result persisted to PostgreSQL\n```\n\nThe LLM is at the core of both agents. If the LLM provider goes down, the entire system stops. That's the problem I needed to solve.\n\nMy first implementation looked like this:\n\n``` python\nfrom langchain_google_genai import ChatGoogleGenerativeAI\n\nllm = ChatGoogleGenerativeAI(\n    model=\"gemini-3.5-flash\",\n    google_api_key=os.getenv(\"GEMINI_API_KEY\")\n)\n\ndef parse_node(state):\n    result = llm.invoke(prompt)\n    return {\"cleaned_data\": result.content}\n```\n\nSimple. Works. Completely fragile.\n\nThe problems were obvious once I started thinking about production:\n\nI needed a proper abstraction layer.\n\nI wanted three things:\n\nThe result is `LLMRouter`\n\n: a registry-based provider abstraction with a configurable fallback chain.\n\n``` python\nclass LLMRouter:\n    def __init__(self):\n        self._chain: list[tuple[str, BaseChatModel]] = []\n        self._build_chain()\n\n    def _build_chain(self):\n        # Priority order from env — change without touching code\n        raw = os.getenv(\"LLM_PROVIDER_ORDER\", \"gemini,groq,openai,claude,custom\")\n        order = [p.strip() for p in raw.split(\",\")]\n\n        for provider_name in order:\n            loader, default_model = PROVIDER_REGISTRY[provider_name]\n            model = os.getenv(f\"{provider_name.upper()}_MODEL\", default_model)\n            instance = loader(model)\n\n            if instance:\n                self._chain.append((provider_name, instance))\n                logger.info(\"Provider loaded: %s (model: %s)\", provider_name, model)\n            else:\n                logger.info(\"Provider skipped (no API key): %s\", provider_name)\n\n        if not self._chain:\n            raise RuntimeError(\"No LLM provider available.\")\n\n        logger.info(\"Fallback chain: %s\",\n            \" -> \".join(name for name, _ in self._chain))\n\n    def invoke(self, prompt: str) -> str:\n        last_error = None\n        for provider_name, llm in self._chain:\n            try:\n                result = llm.invoke(prompt)\n                return result.content.strip()\n            except Exception as e:\n                logger.warning(\"Provider %s failed: %s\", provider_name, e)\n                last_error = e\n                continue\n        raise RuntimeError(f\"All providers failed. Last error: {last_error}\")\n```\n\nEach provider has its own loader function that returns `None`\n\nif the API key is not set:\n\n``` php\ndef _load_gemini(model: str) -> Optional[BaseChatModel]:\n    api_key = os.getenv(\"GEMINI_API_KEY\")\n    if not api_key:\n        return None  # Skip silently — not an error\n    try:\n        from langchain_google_genai import ChatGoogleGenerativeAI\n        return ChatGoogleGenerativeAI(model=model, google_api_key=api_key)\n    except Exception as e:\n        logger.warning(\"Gemini load failed: %s\", e)\n        return None\n```\n\nThe registry maps provider names to loader functions and default models:\n\n```\nPROVIDER_REGISTRY = {\n    \"gemini\":       (_load_gemini,       \"gemini-3.6-flash\"),\n    \"openai\":       (_load_openai,       \"gpt-4o-mini\"),\n    \"claude\":       (_load_claude,       \"claude-haiku-4-5-20251001\"),\n    \"groq\":         (_load_groq,         \"openai/gpt-oss-20b\"),\n    \"azure_openai\": (_load_azure_openai, \"gpt-4o-mini\"),\n    \"custom\":       (_load_custom,       \"llama3.2\"),\n}\n```\n\nThe most useful feature ended up being one I almost didn't build: the custom OpenAI-compatible endpoint.\n\n``` php\ndef _load_custom(model: str) -> Optional[BaseChatModel]:\n    base_url = os.getenv(\"CUSTOM_LLM_BASE_URL\")\n    if not base_url:\n        return None\n    from langchain_openai import ChatOpenAI\n    return ChatOpenAI(\n        model=model,\n        base_url=base_url,\n        api_key=os.getenv(\"CUSTOM_LLM_API_KEY\", \"custom\"),\n    )\n```\n\nThis works with anything that implements the OpenAI chat completions spec: Ollama, OpenRouter, vLLM, or a third-party API gateway. In my case, I'm using a custom API gateway as primary provider with Gemini as fallback:\n\n```\n# .env\nLLM_PROVIDER_ORDER=custom,gemini\nCUSTOM_LLM_BASE_URL=https://my-api-gateway.example.com/v1\nCUSTOM_LLM_API_KEY=my_key\nCUSTOM_LLM_MODEL=claude-opus-4-8\nGEMINI_MODEL=gemini-3.6-flash\n```\n\nStartup log confirms the chain:\n\n```\nProvider loaded: custom (model: claude-opus-4-8)\nProvider loaded: gemini (model: gemini-3.6-flash)\nProvider skipped (no API key): groq\nProvider skipped (no API key): openai\nFallback chain: custom -> gemini\n```\n\nI didn't plan this, but it happened during testing and it validated the entire design.\n\nI submitted a job while Docker Compose was running. The gateway crashed mid-flight (`exited with code 137`\n\n— OOM killed). Here's the exact log sequence from the worker:\n\n```\njob_received job_id=cmsi1m2dn000001pmnnhwojpf\nparse_node attempt=1\nInvoking provider: custom\nparse_node output={\"name\": \"Test Retry1\", ...}\nvalidate_node attempt=1\nvalidate_node is_valid=False reason=name must be two words with each word\n    capitalized and contain only letters; 'Retry1' contains a digit\ngraph_decision result=retry attempt=1\n\nparse_node attempt=2\nparse_node output={\"name\": \"Test Retry\", ...}\nvalidate_node is_valid=True reason=OK\ngraph_decision result=valid attempts=2\n\njob_finished job_id=cmsi1m2dn000001pmnnhwojpf is_valid=True attempts=2\n\ncallback_connection_error ... attempt=1/5\ncallback_retry_scheduled delay=1.0s next_attempt=2\ncallback_connection_error ... attempt=2/5\ncallback_retry_scheduled delay=2.0s next_attempt=3\ncallback_connection_error ... attempt=3/5\ncallback_retry_scheduled delay=4.0s next_attempt=4\ncallback_sent status_code=200 attempt=4\n```\n\nThree things happened simultaneously that I didn't orchestrate:\n\n**1. LangGraph retry loop worked.** The validator correctly rejected `Test Retry1`\n\nbecause `Retry1`\n\ncontains a digit. The parser received the rejection reason and fixed it on attempt 2. The feedback loop between validator and parser is exactly what makes this more than a simple LLM call.\n\n**2. Callback retry with exponential backoff worked.** The gateway was down. The worker kept retrying with 1s → 2s → 4s delays. When the gateway came back up, attempt 4 succeeded. No data was lost.\n\n**3. The worker kept running.** Because LangGraph and the callback retry are separate from the gateway's lifecycle, the worker finished its job correctly even though the service it was trying to reach was down.\n\nThis is what I was aiming for. To switch from custom + Gemini to OpenAI + Groq:\n\n```\n# Before\nLLM_PROVIDER_ORDER=custom,gemini\n\n# After — no code change, just env update\nLLM_PROVIDER_ORDER=openai,groq\nOPENAI_API_KEY=sk-...\nGROQ_API_KEY=gsk_...\n```\n\nTo override the model for a specific provider:\n\n```\nGEMINI_MODEL=gemini-3.6-flash\nOPENAI_MODEL=gpt-4o\nGROQ_MODEL=openai/gpt-oss-20b\n```\n\nThe router reads these at startup and builds the chain. Five providers configured, but only the ones with valid credentials get loaded.\n\n**Health check before adding to chain.** Right now, `_load_custom`\n\nsucceeds if `CUSTOM_LLM_BASE_URL`\n\nis set — even if the endpoint is unreachable. A lightweight ping at startup would catch misconfigured endpoints before the first real job comes in.\n\n**Provider-level metrics.** I want to know which provider is being invoked most often and which ones are failing. Right now this lives in logs. It should be structured data I can query.\n\n**Circuit breaker per provider.** If Gemini returns 5xx three times in a row, stop trying Gemini for the next 60 seconds before retrying. Right now every invoke attempt goes through the full retry before moving to the next provider in the chain.\n\nSingle provider = single point of failure. This is obvious in hindsight, but it took an 11 PM rate limit hit to make me actually fix it.\n\nThe pattern is simple: registry of loaders, environment-configured priority, iterate and fallback on exception. About 100 lines of Python. The complexity is low; the resilience gain is significant.\n\nIf you're building any backend system that calls an LLM, this abstraction is worth the hour it takes to implement.\n\n**AuraFlow AI source code:** [github.com/awaluddin-dev/auraflow-ai](https://github.com/awaluddin-dev/auraflow-ai)\n\n*I'm Awaluddin — Backend Engineer & AI Integrator based in Jakarta, currently consulting at an enterprise automotive company. Building toward a fully remote role. You can find my work at awaluddin-dev.vercel.app or connect on LinkedIn.*", "url": "https://wpnews.pro/news/i-built-a-multi-provider-llm-router-for-my-ai-worker-here-s-what-i-learned", "canonical_source": "https://dev.to/awaluddin/i-built-a-multi-provider-llm-router-for-my-ai-worker-heres-what-i-learned-l8d", "published_at": "2026-08-24 11:59:37+00:00", "updated_at": "2026-08-24 12:13:35.254212+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "developer-tools", "ai-agents", "mlops"], "entities": ["AuraFlow AI", "Gemini", "OpenAI", "Claude", "Groq", "LangGraph", "BullMQ", "Redis"], "alternates": {"html": "https://wpnews.pro/news/i-built-a-multi-provider-llm-router-for-my-ai-worker-here-s-what-i-learned", "markdown": "https://wpnews.pro/news/i-built-a-multi-provider-llm-router-for-my-ai-worker-here-s-what-i-learned.md", "text": "https://wpnews.pro/news/i-built-a-multi-provider-llm-router-for-my-ai-worker-here-s-what-i-learned.txt", "jsonld": "https://wpnews.pro/news/i-built-a-multi-provider-llm-router-for-my-ai-worker-here-s-what-i-learned.jsonld"}}