Nowadays in enterprise software engineering there are two primary models of using LLMs and provision them to a fleet of users:
- Commercial Public APIs: Using closed-source models, such asClaude Opus , either directly through vendors like Anthropic or via cloud providers offering managed access to these models, such as Amazon Bedrock and Microsoft Azure AI Foundry.One of the downsides is privacy as companies are sending private information and proprietary code. Other problem the developers are facing is deprecation . The vendors tends to deprecate old models. Such models are used by applications in production that were carefully instrumented and designed to enhance the integration with these models. Forcing developers to go through a complete evaluation, benchmark and Blue/Green deployment process before the model is complete deprecated.There are more concerns. For instance, today it remains unclear whether the judgments done by LLMs are implicitly shaped by geopolitical cues.
- On-Premise Clusters: Deploying open-weights models, such as DeepSeek, Qwen,AMALIA (A Multimodal focused in Portuguese languague and the Portuguese culture ),APERTUS (Fully Open Foundation Model for Sovereign AI developed in Switzerland) on internal hardware, such as NVIDIA. These deployments often involved aggressive quantization techniques (e.g., NVFP4) to optimize the underlaying hardware achieving higher throughput and lower latency compared with the base model while keeping all data interaction within the corporate network.There are of course challenges when running self-hosted LLMs. Few of them are:
- Computing. You have to own your own GPUs.
- Knowledge. You have to know how to run an inference-distributed cluster and maintain it.
- Operability. As everything you own, you have to maintain, monitor and fix when problems appears
I strongly believe we should reduce our dependency on external LLM vendors. They could change their terms, pricing, or policies at any time, potentially putting our workloads and data at risk. We need to achieve AI and LLM sovereignty by maintaining control over our infrastructure, workloads, and data. The internal data is one of the company’s most valuable assets, and we must ensure it remains private and under our control.
In this post, I’ll show you how easy it is to run APERTUS on your own server using the vLLM framework.
Preparation #
Before running anything or make any hardware decisions, we need to understand the task at hand. This starts with getting familiar with the characteristics of the LLM we want to run.
Model
We are going to run the 8B (8-billion-parameter) version of APERTUS. Given that the model uses the BF16 (bfloat16) data type, we will need at least 32 GB of GPU VRAM to run it comfortably.
This is a multilingual model optimized for smaller GPUs. You can learn more about it here.
In order to enable thinking mode, we would need to set --reasoning-parser and --default-chat-template-kwargs.enable_thinking .
As today (October 2026) the Apertus 1.5 release has not been completely merged in the Transformers and vLLM libraries hence we will need to use at least --tool-call-parser as extra Argument when running vLLM.
Hardware
The accelerator we are going to use is 1 x Nvidia RTX A6000 with 48GB VRAM. Which is enough to run the model we describe above for our own purposes.
LLM Engine
The framework we are going to use to run inference is the very well stablished vLLM.
vLLM comes with a lot of built-in ready optimizations that allow run faster inference servers. Some of its features are:
- Efficient management of attention key and value memory with PagedAttention .
- Continuous batching of incoming requests, chunked prefill, prefix caching
- And more…
Deployment #
We are running on an Ubuntu Server 24.04 LTS. Let’s install few pre-requisites packages. For the sake of this post we are going to use the vLLM Production-stack which is a reference stack for production deployments.
apt-get update
git clone https://github.com/vllm-project/production-stack.git
cd production-stack/utils
bash install-kubectl.sh
kubectl version --client
Client Version: v1.37.1
Kustomize Version: v5.8.1
### Install helm and verify it is installed
bash install-helm.sh
helm version
version.BuildInfo{Version:"v3.22.1", GitCommit:"d6c1a0e4027caa6abc7892e7ea9f0cda7447baa3", GitTreeState:"clean", GoVersion:"go1.26.9"}
### Instal helm diff plugin
helm plugin install https://github.com/databus23/helm-diff
Installed plugin: diff
### Installing Minikube with GPU support
sudo apt remove minikube
bash install-minikube-cluster.sh
#### Verifying GPU Configuration
minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured
kubectl describe nodes | grep -i gpu
nvidia.com/gpu: 1
kubectl run gpu-test --image=nvidia/cuda:12.2.0-runtime-ubuntu22.04 --restart=Never -- nvidia-smi
kubectl logs gpu-test
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.195.03 Driver Version: 570.195.03 CUDA Version: 12.8 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA RTX A6000 On | 00000000:00:06.0 Off | Off |
| 30% 34C P8 35W / 300W | 1MiB / 49140MiB | 0% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+
+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
+-----------------------------------------------------------------------------------------+
Easy. At this point we have a Kubernetes cluster using minikube with the Nvidia plugin installed in the cluster and ready to run any GPU workload. Let’s now deploy APERTUS with vLLM.
### Deploy vLLM Instance (model: vllm_apertus_1.5_release)
helm repo add vllm https://vllm-project.github.io/production-stack
servingEngineSpec:
runtimeClassName: ""
modelSpec:
- name: "apertus-8b"
repository: "ghcr.io/swiss-ai/vllm_apertus_1.5_release"
tag: "latest-amd64"
modelURL: "swiss-ai/Apertus-v1.5-8B"
replicaCount: 1
requestCPU: 10
requestMemory: "16Gi"
requestGPU: 1
pvcStorage: "50Gi"
pvcAccessMode:
- ReadWriteOnce
vllmConfig:
enableChunkedPrefill: true
enablePrefixCaching: true
maxModelLen: 16384
extraArgs: ["--gpu-memory-utilization", "0.9", "--enable-auto-tool-choice", "--tool-call-parser", "apertus"]
hf_token: hf_1234..
helm diff upgrade --install apertus vllm/vllm-stack -f apertus.yaml
helm upgrade --install apertus vllm/vllm-stack -f apertus.yaml
helm list
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
apertus default 1 2026-10-19 19:34:35.173857134 +0000 UTC deployed vllm-stack-0.1.13
kubectl get pods
NAME READY STATUS RESTARTS AGE
apertus-apertus-8b-deployment-vllm-5c6f6bf484-7twjx 0/1 ContainerCreating 0 26s
apertus-deployment-router-76c84b85fb-c49sx 0/1 ContainerCreating 0 26s
spec.containers{vllm}: Pulling image "ghcr.io/swiss-ai/vllm_apertus_1.5_release:latest-amd64"
We need to wait until the model is pulled from HuggingFace.
Then the apertus deployment pod is going to change phase to running. I took few screenshots to give you an idea of the startup process. This will take few minutes to the engine to load all model weights into GPU and start the engine server.
Inference server is ready when you see:
Starting vLLM server on http://0.0.0.0:8000
Available routes are:
Route: /openapi.json, Methods: GET, HEAD
Route: /docs, Methods: GET, HEAD
Route: /docs/oauth2-redirect, Methods: GET, HEAD
...
Initial Tests
Let’s confirm our inference server is working properly and ready to take prompts. An easy way is to expose the kubernetes service to the localhost of the Ubuntu server machine and then in another terminal run a the curl command below
kubectl port-forward svc/apertus-router-service 30080:80
curl -o- http://localhost:30080/v1/models
You should get a response similar to this:
You can see model swiss-ai/Apertus-v1.5-8B is available! 🎉
Let’s now instal a tiny LLM CLI tool which comes handy to use for sending prompts and streaming the output back to the user in the terminal.
This is not the final interface you would like to give to users. This is only for educational purposes.
As a final interface to users you can use any of these open source solutions:
As a second test let’s open 6 terminals and send the following prompt to see our LLM working!.
Prompt: “Describe kubernetes in 200000 words“
We can confirm our vLLM deployment is working and ready to serve to users. Let’s now use few benchmarking tools to figure out the limits of our deployment (Model-Hardware-Workload)
Benchmarking #
For educational purposes we are going to start with vLLM bench. This is an “old“ way provided by vLLm team to benchmark inference systems. it is written in Rust and is basically a client sending different and multiple prompts of different input-len at different rates to your inference server.
In order to take advantage that vLLM libraries are already installed in the deployment Kubernetes pod, we can run vllm-bench inside the pod.
kubectl get pods
kubectl exec -it apertus-apertus-8b-deployment-vllm-6df55cbc8-gpcgl -- /bin/bash
vllm bench serve \
--backend openai \
--base-url http://localhost:8000 \
--endpoint /v1/completions \
--model swiss-ai/Apertus-v1.5-8B \
--dataset-name random \
--num-prompts 500 \
--random-input-len 512 \
--random-output-len 1024 \
--request-rate 5 \
--max-concurrency 32 \
--temperature=0
At the end of the run you will get a report in the terminal. It could as well generate a JSON report for generating graphs afterwards. The screenshot below shows you while running the benchmark.
Then the results:
These metrics are very important, specially TTFT, ITL, TPOT and Request throughput.
Let’s run it again changing input-len from 512 to 1024. We expect the ITL, TPOT won’t change but the TTFT should increase as we are sending a longest prompt. We can confirm this in the results:
Now let’s change the request rate from 5 to 15 and max concurrency from 32 to 64. Now we expect ITL and TPOT be affected as we will be more close to compute-bound.
Here I use a basic configuration for benchmarking the LLM but you could use different datasets, even a custom one and way more advanced configurations.
GuideLLM
Let’s now use a more modern benchmarking tool and which is the one recommended as today by the vLLM maintainers.
GuideLLM allows you to evaluate how LLMs perform under real workloads and configurations. It simulates end-to-end interactions with OpenAI-compatible and vLLM-native servers, generates workload patterns that reflect production usage, and produces detailed reports that help teams understand system behavior, resource needs, and operational limits
Again, this is a basic command that illustrate how this tool could help us to determinate the required GPU footprint and configuration to our enterprise LLM inference server.
It is very important to mention that there is not a silver bullet that works for everyone. The model you choose, the hardware and configuration will depends on your workload and use-cases. For instance, KV caching optimizations will be very beneficial for agentic systems where wend same system prompt multiple times and we run multiple turns during the conversation with the agent.
The TTFT metric could be impacted by the input-len be too large in RAG systems where we send a prompt and a bunch of documents. This then needs to be optimized.
I took few screenshots of the guideLLM running. The command I ran was:
guidellm run \
--backend kind=openai_http,target=http://localhost:30080 \
--profile kind=concurrent,streams=16,warmup=0.1,cooldown=0.1 \
--constraint kind=max_errors,count=5 \
--constraint kind=over_saturation \
--data kind=synthetic_text,prompt_tokens=256,output_tokens=128 \
--output kind=html,path=benchmarks.html
Then the results:
It also generates an HTML with the results. I took few screenshots as well, see them below.
Having More Fun 🖼️ #
As a Multimodel LLM that supports images (and audio!) as an input we can ask questions about the image. Few examples below.
Question 1
Source: https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco_sample.png
Response:
In the image, there are two tabby cats lying on a pink couch. The cat on the left is stretched out on its side, appearing relaxed, with its front paws extended and eyes closed, as if in a state of sleep or rest. The cat on the right is curled up in a more compact position, also sleeping with its eyes closed; this cat has its head resting comfortably on the couch.\n\nBeside the cats, there are two remote controls. The one on the left side of the couch is near the head of the left cat and is a standard TV remote control with multiple buttons. The other remote control is placed on the right side of the couch, near the right cat’s midsection. The couch’s surface is smooth and hints at a soft texture, providing a comfortable resting place for the cats.\n\nThe overall scene suggests a peaceful moment with indoor pets at home, enjoying a quiet nap. The image conveys a sense of warmth and tranquility.
Question 2
source: https://asterixthegaul.com/wp-content/uploads/2025/12/asterix-obelix-idefix-wbg-689x1024.jpg
Response:
Asterix and Obelix
Conclusions #
It was really easy and fun to self-host APERTUS and explore its capabilities. I didn’t include all the experiments I ran, such as setting up monitoring tools like Grafana to inspect metrics during benchmarks. Maybe I’ll write another post about LLM observability in the future.
Anyway, I hope this post has shown you how easy it is to own and run your own LLM using open-weight models. The open-source community around LLMs is growing very fast, with inference frameworks such as SGLang and many others. It’s up to you to experiment and choose the framework, hardware and model that best fits your needs.