{"slug": "hetzner-inference-first-look", "title": "Hetzner Inference: First Look", "summary": "Hetzner has launched an experimental LLM inference API called Hetzner Inference, offering an OpenAI-compatible endpoint running on its own infrastructure. Currently, only the Qwen/Qwen3.6-35B-A3B-FP8 model is available, and the company is testing demand and scalability with no billing or SLA. The experiment suggests Hetzner may be exploring a low-cost inference service leveraging its efficient hardware and data center operations.", "body_md": "Hetzner is experimenting with LLM inference.\n\nThat is not a sentence I expected to write, but I think it is pretty interesting :)\n\nBefore anyone moves their production AI workloads to Hetzner: this is very much an **experiment**. There is no billing, no SLA, no production guarantee, and currently only one model. Hetzner says it wants to learn whether people actually want this, how the system scales, which features matter, and what kind of load it can handle.\n\nSo this is not a finished product launch. It is Hetzner putting something early in front of users and seeing what happens. I really like that approach.\n\n[Hetzner Inference](https://experiments.hetzner.com) is an OpenAI-compatible API running on Hetzner's own infrastructure. You create an API token in the Experiments dashboard, point an OpenAI client at Hetzner's base URL, and use it like most other inference APIs.\n\nRight now, the only available model is [ Qwen/Qwen3.6-35B-A3B-FP8](https://huggingface.co/Qwen/Qwen3.6-35B-A3B-FP8). It is a 35-billion-parameter Mixture-of-Experts model with 3 billion active parameters. It accepts text and images, has a 262K context window, and uses FP8-quantized weights.\n\nThat is a perfectly reasonable model for an experiment. It is small enough to serve without a ridiculous GPU cluster, but still useful enough to test the API with real workloads.\n\nHetzner also published a [short tutorial for connecting OpenCode to the API](https://community.hetzner.com/tutorials/opencode-with-hetzner-inference-api-systemd-sandbox/), if you want to try it without writing any code.\n\nBecause the API is OpenAI-compatible, there is almost nothing special about the integration:\n\n```\npip install openai\npython\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=\"https://inference.hetzner.com/api/v1\",\n    api_key=\"YOUR_TOKEN\",\n)\n\nresponse = client.chat.completions.create(\n    model=\"Qwen/Qwen3.6-35B-A3B-FP8\",\n    messages=[\n        {\"role\": \"user\", \"content\": \"Explain why the sky is blue in one sentence.\"}\n    ],\n    extra_body={\n        \"chat_template_kwargs\": {\n            \"enable_thinking\": False,\n        }\n    },\n)\n\nprint(response.choices[0].message.content)\n```\n\nThe `enable_thinking`\n\noption is worth mentioning. Without it, the model can spend a surprising amount of the completion budget reasoning before it returns a visible answer. The option worked in my tests, but it is not documented by Hetzner, so I would not build anything important around that exact request shape yet.\n\nI ran a few small tests on July 23, 2026. I do not want to turn this post into a giant benchmark report, because the product is experimental and a benchmark against it will probably age badly. But the rough numbers were:\n\nThat is fast! It is also just one test from one client at one point in time. It is not an SLA, and it says almost nothing about what happens when many people use the service at once.\n\nThe model itself was roughly what I expected. It followed most formatting and retrieval instructions, handled an image correctly, and failed two very simple arithmetic questions. So: a small, slightly shitty LLM :D\n\nThe Qwen endpoint is fun, but I do not think the current model is the interesting part.\n\nThe interesting part is why Hetzner is testing inference in the first place.\n\nEverything from here on is purely my speculation. I have no insider information, and nobody at Hetzner told me what they are planning. I am just looking at the product and trying to connect a few dots.\n\nOpen-weight inference is a commodity market. Everyone can download the same weights, run more or less the same serving software, and expose an OpenAI-compatible API. Switching providers is also easy, especially with products like OpenRouter or LiteLLM for those you self-host.\n\nThat makes it hard to build huge margins unless you have some kind of advantage. Usually that means:\n\nHetzner is very good at buying hardware, putting it into its own data centers, and operating it with a brutally efficient cost structure. That is basically the whole company. If anyone can turn inference into another low-margin infrastructure product, Hetzner is at least a believable candidate.\n\nThere is also a nice utilization story here. A rented bare-metal GPU belongs to one customer, whether that customer uses it or not. An inference API can share GPU capacity across many users and keep the hardware busy. If Hetzner has spare GPU capacity — or plans to build a much larger GPU fleet — an inference product could help turn that capacity into revenue.\n\nAgain, I have no idea whether this is actually what they are doing. It would just make economic sense to me.\n\nThis is where I am not yet convinced.\n\nHetzner's current [public dedicated GPU server lineup](https://docs.hetzner.com/robot/dedicated-server/server-lines/gpu-server/) uses two GPU types:\n\nThose are capable GPUs, and the 96 GB RTX PRO 6000 is a pretty nice inference machine for small and medium-sized models. The FP8 files for Hetzner's current Qwen model are around 38 GB, with actual VRAM use landing somewhere above that depending on context length, cache size, and serving setup.\n\nBut these are workstation GPUs, not the dense multi-GPU systems you need for the really large open models.\n\nTake [GLM-5](https://huggingface.co/zai-org/GLM-5) as an extreme example. It has 754 billion parameters, and the official serving recipe splits it across eight GPUs. Even with aggressive quantization, you are talking about hundreds of gigabytes of VRAM. Realistically, that is B200/B300-class hardware, or something similar, with very fast links between multiple GPUs.\n\nHetzner does not currently offer that kind of hardware in its public bare-metal lineup.\n\nOf course, that does not tell us what sits behind the experimental API. Hetzner may use completely different internal hardware, and a public inference product does not have to mirror its dedicated-server catalogue.\n\nStill, this is the part I am watching.\n\nIf Hetzner keeps serving one or two smaller models, I do not really see it becoming an important inference provider. That would be a cool experiment, but not much more.\n\nIf this experiment is the first step toward larger GPU clusters, a proper model catalogue, and B200/B300-class hardware, then it gets much more interesting. Hetzner already has the data centers, network, hardware experience, European positioning, and reputation for aggressive pricing. That combination could make it a serious competitor.\n\nFor now, the API is fast, free, and fun to try. The next hardware announcement will tell us much more than another small model would.\n\nCheers,\n\nJonas", "url": "https://wpnews.pro/news/hetzner-inference-first-look", "canonical_source": "https://dev.to/code42cate/hetzner-inference-first-look-587", "published_at": "2026-07-24 09:19:14+00:00", "updated_at": "2026-07-24 09:34:55.716172+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-products"], "entities": ["Hetzner", "Qwen", "OpenAI", "OpenRouter", "LiteLLM"], "alternates": {"html": "https://wpnews.pro/news/hetzner-inference-first-look", "markdown": "https://wpnews.pro/news/hetzner-inference-first-look.md", "text": "https://wpnews.pro/news/hetzner-inference-first-look.txt", "jsonld": "https://wpnews.pro/news/hetzner-inference-first-look.jsonld"}}