Getting a self-hosted model to answer your first prompt feels like the hard part.
The weights load, the server starts, and a response appears in your terminal. It is easy to think you are nearly done.
Then several users arrive at once.
One sends a long document. An agent keeps repeating its tool definitions. The embedding model sits on a mostly idle GPU while the chat model has requests waiting.
Those are different problems, and they need different tools. An inference runtime decides how efficiently a model runs.
A multi-model server manages the smaller models around it. Monitoring and scaling tools help you see when capacity is running short and respond.
Start by looking at where requests wait and where your hardware sits idle.
THE QUICK READ
For a busy generative endpoint, start by comparing vLLM and SGLang*. Consider* TensorRT-LLM for an NVIDIA-focused deployment, LMDeploy for supported quantized models, and llama.cpp when CPU or local hardware matters. TEI suits a dedicated embedding or reranking endpoint; SIE addresses a fleet of specialist models sharing capacity.
Ten thousand requests a day sounds like a useful sizing target. Spread evenly, it is roughly seven requests a minute.
Arriving in bursts, with long prompts and long answers, those same requests can keep a server busy for very different reasons.
Write down peak concurrency, typical input and output lengths, and how long a user can wait.
Include the requests you would rather ignore: the unusually long document, the agent that calls a model several times, and the retry after a timeout.
That gives you a way to judge the tools below. A runtime with excellent throughput may still miss your latency target.
A shared model server may save capacity but introduce delays. An overnight processing job can tolerate a wait that would frustrate someone using a chat interface.
These tools sit close to the model. They handle work such as batching requests, managing attention memory, and producing tokens. Use the same requests when you compare them, including the long ones.
vLLM uses PagedAttention to manage attention memory and continuous batching to keep processing requests as others finish. It also supports prefix caching and an OpenAI-compatible server interface.
It is a sensible first candidate for a dedicated chat or generation endpoint. Use it as a baseline, then check whether another runtime improves the measurements you care about on your hardware.
For Kubernetes deployments, the separate vLLM Production Stack adds Helm-based deployment, request routing, and Prometheus and Grafana monitoring. The runtime and the deployment stack solve different parts of the job. An agent might send the same system prompt and tool definitions on every turn.
SGLang’s RadixAttention and prefix caching let it reuse computation for shared prompt prefixes.
That makes repeated agent context a useful workload to test. The benefit depends on how much of the prefix actually matches and whether requests reach a worker holding the relevant cache.
vLLM has prefix caching too, so compare both with real request sequences before choosing.
TensorRT-LLM is NVIDIA’s inference runtime. Its trtllm-serve command exposes OpenAI-compatible endpoints, including chat completions and Responses.
Put it on the shortlist when you already run NVIDIA GPUs and have time to tune the deployment.
Check the supported model, GPU, precision, and feature combination you intend to use. A good result on a different configuration tells you only so much.
LMDeploy combines its TurboMind and PyTorch engines with model compression and serving tools.
It supports weight quantization and key-value cache quantization, which can help reduce memory requirements for supported models.
It is worth evaluating when the model you want is a tight fit on the GPUs you have.
Check features against the chosen engine, then test output quality after quantization. A model that fits but misses fields in your extraction task has not solved the problem.
llama.cpp’s server runs GGUF models behind an OpenAI-compatible interface. The project supports CPU inference and hardware backends for Apple silicon and several GPU families.
That makes it useful for local applications, edge devices, and on-premises machines with limited hardware choices.
Measure concurrency on the actual machine you plan to deploy. A responsive single-user session does not tell you how a shared endpoint will behave.
IF YOU REMEMBER TGI
Hugging Face’s Text Generation Inference repository was archived on March 21, 2026. Existing installations need a maintenance plan.
For a new deployment, begin with a maintained runtime. A retrieval application can embed a query, rerank search results, and extract fields before the main LLM writes anything. Each step may use a different model.
Give every lightly used model a dedicated GPU and the idle capacity starts to hurt.
With a few busy models, separate endpoints can work well. With a dozen models that receive occasional requests, sharing capacity becomes more appealing.
Hugging Face’s Text Embeddings Inference (TEI) serves supported embedding and reranking models. It includes batching and Prometheus metrics.
For one or two models with steady demand, a dedicated instance can be easy to reason about. You know which model is loaded and can size its capacity independently. As the catalog grows, you also take on more separate deployments.
Superlinked Inference Engine (SIE) exposes four operations: encode, score, extract, and generate. Its documentation lists more than 100 supported models, with on-demand and least-recently-used eviction so models can share GPU memory.
That addresses a different operational problem: several specialist models receiving uneven traffic. Keeping the whole catalog available no longer requires keeping every model loaded.
There is a cost to switching. A model that has been evicted must load again, so test the sequence of requests as well as each model in isolation.
SIE’s deployment architecture also uses dependency bundles and worker pools; sharing capacity still takes planning.
You can keep a busy generative model on vLLM or SGLang and use SIE for the specialist models around it. The choice does not have to apply to every model in the application.
A server can expose a healthy endpoint while requests spend too long waiting. You need measurements that describe what users experience.
Prometheus collects metrics that you can use for dashboards and alerts. Track queue depth, errors, and time to first token.
For generated responses, also watch the delay between output tokens. Include a percentile such as p95 so an acceptable average does not hide the slowest sessions. Grafana dashboards help you inspect those signals together.
The vLLM Production Stack includes a monitoring setup, and SIE documents its own Prometheus integration and Grafana dashboards. Make the alerts useful before traffic arrives.
In a Kubernetes deployment, KEDA can scale workloads from signals such as queue length or Prometheus metrics.
The serving runtime exposes the data; the scaling configuration decides when to add workers. Provisioning more GPU nodes may require separate cluster autoscaling.
SIE documents KEDA-based autoscaling and scale-to-zero. Before enabling it, measure how long the first request waits for capacity and model .
Also check the infrastructure bill: scaling worker pods to zero does not necessarily release the GPU nodes beneath them.
A starting point for evaluation. Test performance with your own models and traffic.
An OpenAI-compatible API can reduce client changes, but endpoint compatibility does not guarantee identical behavior.
Check streaming, tool calls, and structured output with your exact model and chat template. Valid JSON still needs to satisfy your application’s rules.
If you serve multiple LoRA fine-tunes, verify adapter support and switching behavior for the runtime and engine you selected. A feature listed for a project may have restrictions in a particular backend.
For a single busy LLM, begin with a dedicated runtime and a load test.
For a growing set of lightly used specialist models, compare separate TEI instances with SIE’s shared approach. Add capacity when the measurements show where it is needed.
Keep a record of queue time, model execution time, and cost during that test.
You will need those numbers when the next model or traffic spike arrives.
The Tools Powering Open-Source AI in Production was originally published in Stackademic on Medium, where people are continuing the conversation by highlighting and responding to this story.