Running Apertus: A Swiss-Made Multimodal LLM

A hands-on guide demonstrates running the Swiss-made Apertus 1.5 8B multimodal model on a single Nvidia RTX A6000 with 48GB of VRAM using the vLLM inference framework, requiring at least 32GB of GPU VRAM in BF16. The author argues enterprises should reduce dependency on commercial API vendors such as Anthropic's Claude Opus, citing deprecation risk and data privacy, and notes that as of October 2026 the Apertus 1.5 release is not fully merged into the Transformers and vLLM libraries, so a --tool-call-parser argument is needed alongside --reasoning-parser and --default-chat-template-kwargs.enable_thinking to enable thinking mode.

Nowadays in enterprise software engineering there are two primary models of using LLMs and provision them to a fleet of users: 1. Commercial Public APIs: Using closed-source models, such as Claude Opus https://www.anthropic.com/claude/opus , either directly through vendors like Anthropic or via cloud providers offering managed access to these models, such as Amazon Bedrock and Microsoft Azure AI Foundry.One of the downsides is privacy as companies are sending private information and proprietary code. Other problem the developers are facing is deprecation . The vendors tends to deprecate old models. Such models are used by applications in production that were carefully instrumented and designed to enhance the integration with these models. Forcing developers to go through a complete evaluation, benchmark and Blue/Green deployment process before the model is complete deprecated. There are more concerns. For instance, today it remains unclear whether the judgments done by LLMs are implicitly shaped by geopolitical cues. 2. On-Premise Clusters: Deploying open-weights models, such as DeepSeek, Qwen, AMALIA https://amaliallm.pt/ A Multimodal focused in Portuguese languague and the Portuguese culture , APERTUS https://apertus-ai.org/articles/2026-07-apertus-1-5/ Fully Open Foundation Model for Sovereign AI developed in Switzerland on internal hardware, such as NVIDIA. These deployments often involved aggressive quantization techniques e.g., NVFP4 to optimize the underlaying hardware achieving higher throughput and lower latency compared with the base model while keeping all data interaction within the corporate network.There are of course challenges when running self-hosted LLMs. Few of them are: 1. Computing. You have to own your own GPUs. 2. Knowledge. You have to know how to run an inference-distributed cluster and maintain it. 3. Operability. As everything you own, you have to maintain, monitor and fix when problems appears I strongly believe we should reduce our dependency on external LLM vendors. They could change their terms, pricing, or policies at any time, potentially putting our workloads and data at risk. We need to achieve AI and LLM sovereignty by maintaining control over our infrastructure, workloads, and data. The internal data is one of the company’s most valuable assets, and we must ensure it remains private and under our control. In this post, I’ll show you how easy it is to run APERTUS https://www.apertus-ai.org/ on your own server using the vLLM framework. Preparation Before running anything or make any hardware decisions, we need to understand the task at hand. This starts with getting familiar with the characteristics of the LLM we want to run. Model We are going to run the 8B 8-billion-parameter version of APERTUS https://huggingface.co/swiss-ai/Apertus-v1.5-8B . Given that the model uses the BF16 bfloat16 data type, we will need at least 32 GB of GPU VRAM to run it comfortably. This is a multilingual model optimized for smaller GPUs. You can learn more about it here https://huggingface.co/swiss-ai/Apertus-v1.5-8B . In order to enable thinking mode , we would need to set --reasoning-parser and --default-chat-template-kwargs.enable thinking . As today October 2026 the Apertus 1.5 release has not been completely merged in the Transformers and vLLM libraries hence we will need to use at least --tool-call-parser as extra Argument when running vLLM. Hardware The accelerator we are going to use is 1 x Nvidia RTX A6000 https://www.nvidia.com/en-us/products/workstations/rtx-a6000/ with 48GB VRAM. Which is enough to run the model we describe above for our own purposes. LLM Engine The framework we are going to use to run inference is the very well stablished vLLM https://vllm.ai/ . vLLM comes with a lot of built-in ready optimizations that allow run faster inference servers. Some of its features are: - Efficient management of attention key and value memory with PagedAttention https://vllm.ai/blog/2023-06-20-vllm . - Continuous batching of incoming requests, chunked prefill, prefix caching - And more… Deployment We are running on an Ubuntu Server 24.04 LTS. Let’s install few pre-requisites packages. For the sake of this post we are going to use the vLLM Production-stack https://github.com/vllm-project/production-stack which is a reference stack for production deployments. Initial OS packages updates apt-get update Clone production-stack repository git clone https://github.com/vllm-project/production-stack.git cd production-stack/utils install kubectl and verify it is installed bash install-kubectl.sh kubectl version --client Expected similar output: Client Version: v1.37.1 Kustomize Version: v5.8.1 Install helm and verify it is installed bash install-helm.sh helm version Expected similar output: version.BuildInfo{Version:"v3.22.1", GitCommit:"d6c1a0e4027caa6abc7892e7ea9f0cda7447baa3", GitTreeState:"clean", GoVersion:"go1.26.9"} Instal helm diff plugin Source: https://github.com/databus23/helm-diff helm plugin install https://github.com/databus23/helm-diff Expected similar output: Installed plugin: diff Installing Minikube with GPU support sudo apt remove minikube bash install-minikube-cluster.sh Verifying GPU Configuration minikube status Expected similar output: minikube type: Control Plane host: Running kubelet: Running apiserver: Running kubeconfig: Configured kubectl describe nodes | grep -i gpu Expected similar output: nvidia.com/gpu: 1 Run a test pod in kubernetes kubectl run gpu-test --image=nvidia/cuda:12.2.0-runtime-ubuntu22.04 --restart=Never -- nvidia-smi Wait for the pod to be in phase: completed, then: kubectl logs gpu-test Expected similar output: +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 570.195.03 Driver Version: 570.195.03 CUDA Version: 12.8 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA RTX A6000 On | 00000000:00:06.0 Off | Off | | 30% 34C P8 35W / 300W | 1MiB / 49140MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | No running processes found | +-----------------------------------------------------------------------------------------+ Easy. At this point we have a Kubernetes cluster using minikube with the Nvidia plugin installed in the cluster and ready to run any GPU workload. Let’s now deploy APERTUS with vLLM. Deploy vLLM Instance model: vllm apertus 1.5 release helm repo add vllm https://vllm-project.github.io/production-stack Create a file apertus.yaml with the following or similar content DO NOT FORGET to set your Hugging Face token in 'hf token' servingEngineSpec: runtimeClassName: "" modelSpec: - name: "apertus-8b" repository: "ghcr.io/swiss-ai/vllm apertus 1.5 release" tag: "latest-amd64" modelURL: "swiss-ai/Apertus-v1.5-8B" replicaCount: 1 requestCPU: 10 requestMemory: "16Gi" requestGPU: 1 pvcStorage: "50Gi" pvcAccessMode: - ReadWriteOnce vllmConfig: enableChunkedPrefill: true enablePrefixCaching: true maxModelLen: 16384 extraArgs: "--gpu-memory-utilization", "0.9", "--enable-auto-tool-choice", "--tool-call-parser", "apertus" hf token: hf 1234.. See first what is going to be installed helm diff upgrade --install apertus vllm/vllm-stack -f apertus.yaml Then deploy helm upgrade --install apertus vllm/vllm-stack -f apertus.yaml Confirm the helm chart was installed helm list NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION apertus default 1 2026-10-19 19:34:35.173857134 +0000 UTC deployed vllm-stack-0.1.13 List the pods an you should see two pods being created kubectl get pods NAME READY STATUS RESTARTS AGE apertus-apertus-8b-deployment-vllm-5c6f6bf484-7twjx 0/1 ContainerCreating 0 26s apertus-deployment-router-76c84b85fb-c49sx 0/1 ContainerCreating 0 26s You could inspect the pod: apertus-8b-deployment-vllm and wills ee it is fetching the docker image which contains the model weights. it will take few minutes. spec.containers{vllm}: Pulling image "ghcr.io/swiss-ai/vllm apertus 1.5 release:latest-amd64" We need to wait until the model is pulled from HuggingFace. Then the apertus deployment pod is going to change phase to running . I took few screenshots to give you an idea of the startup process. This will take few minutes to the engine to load all model weights into GPU and start the engine server. Inference server is ready when you see: Starting vLLM server on http://0.0.0.0:8000 Available routes are: Route: /openapi.json, Methods: GET, HEAD Route: /docs, Methods: GET, HEAD Route: /docs/oauth2-redirect, Methods: GET, HEAD ... Initial Tests Let’s confirm our inference server is working properly and ready to take prompts. An easy way is to expose the kubernetes service to the localhost of the Ubuntu server machine and then in another terminal run a the curl command below Expose vLLM server to port 30080 on localhost kubectl port-forward svc/apertus-router-service 30080:80 Then get the models available on our vLLM server curl -o- http://localhost:30080/v1/models You should get a response similar to this: You can see model swiss-ai/Apertus-v1.5-8B is available 🎉 Let’s now instal a tiny LLM CLI tool https://github.com/simonw/LLM which comes handy to use for sending prompts and streaming the output back to the user in the terminal. This is not the final interface you would like to give to users. This is only for educational purposes. As a final interface to users you can use any of these open source solutions: As a second test let’s open 6 terminals and send the following prompt to see our LLM working . Prompt: “Describe kubernetes in 200000 words“ We can confirm our vLLM deployment is working and ready to serve to users. Let’s now use few benchmarking tools to figure out the limits of our deployment Model-Hardware-Workload Benchmarking For educational purposes we are going to start with vLLM bench https://docs.vllm.ai/en/latest/cli/bench/ . This is an “old“ way provided by vLLm team to benchmark inference systems. it is written in Rust and is basically a client sending different and multiple prompts of different input-len at different rates to your inference server. In order to take advantage that vLLM libraries are already installed in the deployment Kubernetes pod, we can run vllm-bench inside the pod. List the pods kubectl get pods Exec into the apertus deployment pod kubectl exec -it apertus-apertus-8b-deployment-vllm-6df55cbc8-gpcgl -- /bin/bash Run a benchmark with the following configuration vllm bench serve \ --backend openai \ --base-url http://localhost:8000 \ --endpoint /v1/completions \ --model swiss-ai/Apertus-v1.5-8B \ --dataset-name random \ --num-prompts 500 \ --random-input-len 512 \ --random-output-len 1024 \ --request-rate 5 \ --max-concurrency 32 \ --temperature=0 At the end of the run you will get a report in the terminal. It could as well generate a JSON report for generating graphs afterwards. The screenshot below shows you while running the benchmark. Then the results: These metrics are very important, specially TTFT, ITL, TPOT and Request throughput. Let’s run it again changing input-len from 512 to 1024. We expect the ITL, TPOT won’t change but the TTFT should increase as we are sending a longest prompt. We can confirm this in the results: Now let’s change the request rate from 5 to 15 and max concurrency from 32 to 64. Now we expect ITL and TPOT be affected as we will be more close to compute-bound. Here I use a basic configuration for benchmarking the LLM but you could use different datasets, even a custom one and way more advanced configurations. GuideLLM Let’s now use a more modern benchmarking tool and which is the one recommended as today by the vLLM maintainers. GuideLLM https://github.com/vllm-project/guidellm allows you to evaluate how LLMs perform under real workloads and configurations. It simulates end-to-end interactions with OpenAI-compatible and vLLM-native servers, generates workload patterns that reflect production usage, and produces detailed reports that help teams understand system behavior, resource needs, and operational limits Again, this is a basic command that illustrate how this tool could help us to determinate the required GPU footprint and configuration to our enterprise LLM inference server. It is very important to mention that there is not a silver bullet that works for everyone. The model you choose, the hardware and configuration will depends on your workload and use-cases. For instance, KV caching optimizations will be very beneficial for agentic systems where wend same system prompt multiple times and we run multiple turns during the conversation with the agent. The TTFT metric could be impacted by the input-len be too large in RAG systems where we send a prompt and a bunch of documents. This then needs to be optimized. I took few screenshots of the guideLLM running. The command I ran was: guidellm run \ --backend kind=openai http,target=http://localhost:30080 \ --profile kind=concurrent,streams=16,warmup=0.1,cooldown=0.1 \ --constraint kind=max errors,count=5 \ --constraint kind=over saturation \ --data kind=synthetic text,prompt tokens=256,output tokens=128 \ --output kind=html,path=benchmarks.html Then the results: It also generates an HTML with the results. I took few screenshots as well, see them below. Having More Fun 🖼️ As a Multimodel LLM that supports images and audio as an input we can ask questions about the image. Few examples below. Question 1 Source: https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/coco sample.png Response: In the image, there are two tabby cats lying on a pink couch. The cat on the left is stretched out on its side, appearing relaxed, with its front paws extended and eyes closed, as if in a state of sleep or rest. The cat on the right is curled up in a more compact position, also sleeping with its eyes closed; this cat has its head resting comfortably on the couch.\n\nBeside the cats, there are two remote controls. The one on the left side of the couch is near the head of the left cat and is a standard TV remote control with multiple buttons. The other remote control is placed on the right side of the couch, near the right cat’s midsection. The couch’s surface is smooth and hints at a soft texture, providing a comfortable resting place for the cats.\n\nThe overall scene suggests a peaceful moment with indoor pets at home, enjoying a quiet nap. The image conveys a sense of warmth and tranquility. Question 2 source: https://asterixthegaul.com/wp-content/uploads/2025/12/asterix-obelix-idefix-wbg-689x1024.jpg Response: Asterix and Obelix Conclusions It was really easy and fun to self-host APERTUS and explore its capabilities. I didn’t include all the experiments I ran, such as setting up monitoring tools like Grafana to inspect metrics during benchmarks. Maybe I’ll write another post about LLM observability in the future. Anyway, I hope this post has shown you how easy it is to own and run your own LLM using open-weight models. The open-source community around LLMs is growing very fast, with inference frameworks such as SGLang https://github.com/sgl-project/sglang and many others. It’s up to you to experiment and choose the framework, hardware and model that best fits your needs.