# Superlinked Inference Engine > Source: > Published: 2026-09-07 13:01:03+00:00 Everyone talks about the LLM in an AI agent. Almost nobody talks about everything else the agent calls before it ever reaches that LLM. Walk through a single turn of a realistic retrieval-augmented agent: it embeds the incoming query, reranks the candidates that search turned up, pulls structured fields out of a PDF nobody formatted for machines, and maybe runs a small guard model to check the input is safe to act on. Only *then* does it call a language model to write the answer. None of those initial four steps needs a Big LLM model. But fundamentally, each of them has needed its own server — a process for embedding creation, a separate container for the reranker, another for OCR, another for extraction. Each with its own dependencies, its own GPU allocation, its own health check, its own scaling rule to babysit. Superlinked has a name it as **Model sprawl**. And it’s the specific problem their new open-source project, the **Superlinked Inference Engine (SIE)**, is built to solve. Most inference tooling was built around one assumption: you have one model, and your job is to serve it as fast and cheaply as possible. That’s the world vLLM, TGI, and SGLang live in — KV cache management, continuous batching, parallelism, kernels tuned for a single model sitting on its own dedicated GPU. Agent workloads break that assumption. They’re not dominated by one giant model — they’re dominated by a *long tail* of small ones, lightweight models to justify its own GPU, but each still needing to run somewhere. That’s not a throughput problem. It’s a **Density** issue, and it’s a different engineering question entirely. Strip away the pitch, and SIE is a single self-hosted server that exposes the same four operations no matter which of its 100+ models is handling the request. Swapping models is a one-line change to a model-name string — no redeploy, no new container. It also speaks fluent OpenAI: SIE answers on /v1/embeddings, /v1/chat/completions, /v1/completions, and /v1/responses, so a lot of existing OpenAI-client code can just point its base URL at SIE and keep working. Getting it running is genuinely two commands: ``` docker run -p 8080:8080 \ -v sie-hf-cache:/app/.cache/huggingface \ ghcr.io/superlinked/sie-server:latest-cpu-default pip install sie-sdk # Pythonpnpm add @superlinked/sie-sdk # TypeScript ``` This is the part worth sitting with: SIE isn’t trying to out-serve vLLM at serving one big model. It’s answering a different question — given dozens of models that are each individually too small to justify a dedicated GPU, how do you serve all of them from infrastructure that doesn’t scale linearly with how many exist? Superlinked’s own framing of this is blunt: for a mixed agent stack, the answer is often both tools, or sometimes neither. A production stack plausibly runs vLLM (or an equivalent) for the one heavyweight generative model, and SIE for the long tail of small models around it — the reranker, the extractor, the OCR pass, the guard model. **Idle GPUs are the default, not the exception.** A reranker or extractor gets called intermittently, not continuously. Give it a whole GPU and most of that GPU sits idle between calls. Sharing GPUs across many on-demand models turns that idle capacity into headroom instead of a second line item. **Every extra server is a second job.** Five one-model-per-box deployments means five dependency sets, five health checks, five things that can page someone at 3am for reasons that have nothing to do with the agent’s actual logic. **Per-token pricing scales with volume whether you want it to or not.** Routing embeddings, reranking, and extraction through managed APIs is the fastest way to ship — and the cheapest way to end up with a bill that grows linearly with traffic. Self-hosting caps that cost at the price of the infrastructure. **Some documents genuinely can’t leave the building.** OCR and extraction are often the steps touching the most sensitive input — contracts, IDs, internal PDFs. Running them inside your own network boundary is a different security posture than shipping raw documents to a third party. A few design choices make this practical rather than theoretical: The generative-model tier already got a decade of serving research aimed squarely at it — vLLM, TGI, and everything downstream of them. SIE is a bet that the small-model tier underneath every agent — the embedder, the reranker, the parser, the extractor — deserves the same dedicated, production-grade serving story, instead of a folder of half-maintained scripts holding it together. If your agent stack currently looks like four separate model servers wearing a trench coat, it’s worth a look. *SIE is Apache 2.0 and open source, self-hostable via Docker.* [Superlinked Inference Engine](https://pub.towardsai.net/superlinked-inference-engine-f7556613a808) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.