Claude Codesessions, OpenAI function calls, and a patchwork of open-weight models through brittle scripts. Nobody was watching because the real work happens in private repos and Slack threads, not conference keynotes.
Last month I spent two weeks untangling a client's AI pipeline that had grown organically across three different model ecosystems. What I found wasn't pretty: prompt chains duplicated across services, auth tokens scattered in six places, and a CI/CD setup that broke every time OpenAI changed their API schema. This isn't unique—it's the default state of most teams shipping LLM features today.
The Three-Horse Mess #
Most organizations end up here by accident. They start with ChatGPT for quick prototypes, move to Claude for document reasoning, then discover they need local models for compliance or latency reasons. The result is a Frankenstack that looks nothing like the clean architectures in blog posts.
: excels at long-context reasoning, terrible at structured output, zero built-in agent toolingClaudeCodeOpenAI Agents: slick SDK, good function calling, expensive at scale, rate limits kill batch jobs** Open Models (Llama 3.1, Qwen 2.5)**: full control, no rate limits, require 20x the engineering effort to match SaaS quality
I've watched teams waste months trying to build "one interface to rule them all" abstractions that collapse under real-world prompt complexity. The abstraction always leaks.
My Current Stack After 50+ Customer Deployments #
I stopped pretending there's a unified solution. Instead, I route based on the actual task:
-
Document analysis + rewriting → Claude 3.5 Sonnet (it just works, even with 150K tokens)
-
Structured data extraction → GPT-4o-mini with function calling (cheaper, faster, reliable JSON)
-
Compliance-sensitive local processing → Qwen 2.5 72B on-prem (no egress, no vendor lock-in)
-
High-throughput classification → Llama 3.1 8B quantized (runs on a $300 GPU, handles 10K RPM)
The glue? A simple routing layer that picks based on context length, output schema complexity, and cost constraints. No fancy orchestration framework—just YAML configs and retry logic.
Why Nobody Talks About This #
The vendors want you buying into their walled gardens. Open-source folks want you contributing to their repos. Meanwhile, real teams just want their prompts to stop failing at 3 AM. That's why the most successful implementations I've seen look embarrassingly primitive: a few shell scripts, some environment variables, and lots of manual oversight.
The gap between demo-day polish and production reality is wider than ever. Until someone builds tooling that survives contact with actual business requirements—nested prompts, shifting legal constraints, mixed data sources—the messy middle ground is where most of us live.
I'm curious what other teams' routing heuristics look like. Are you still trying to unify everything under one agent framework, or have you given up and gone full poly-repo?
OpenAI Just Dragged Apple Into the Dirt Over Those Trade Secret 1h ago
[OpenAI Settles $3. 2h ago](/en/news/5038/)
[Google's $200B Bet on Anthropic 8h ago](/en/news/4991/)
Anthropic Secures $10B Compute Deal with Volta for AI Scaling 8h ago
Apple's latest legal filing reveals a concerning pattern 11h ago
OpenAI Fires Back at Apple's Trade Secret Lawsuit with Public 12h ago
Next OpenAI Just Dragged Apple Into the Dirt Over Those Trade Secret →