{"slug": "hetzner-inference-a-practical-deep-dive", "title": "Hetzner Inference: A Practical Deep Dive", "summary": "Hetzner has launched an OpenAI-compatible inference API offering the Qwen/Qwen3.6-35B-A3B-FP8 model, a Mixture-of-Experts model with a 262K context window. In low-load tests, the service achieved a median time to first token of 153 ms and an output speed of 224 tokens per second, though the model struggled with basic arithmetic. The move signals Hetzner's potential shift from selling VMs to competing in the commoditized open-weight inference market.", "body_md": "# Hetzner Inference: A Practical Deep Dive\n\nThe setup is an OpenAI-compatible API. You grab a token from their experiments dashboard, swap the base URL in your client, and you're in. Currently, the only model available is `Qwen/Qwen3.6-35B-A3B-FP8`\n\n. It's a Mixture-of-Experts model with a 262K context window. It's a strategic choice: small enough to run without a massive GPU farm, but capable enough to handle actual prompts.\n\nIntegration is trivial since it follows the standard OpenAI spec. I put together this quick deployment snippet for anyone wanting to test it:\n\n``` python\nfrom openai import OpenAI\n\nclient = OpenAI(\n base_url=\"https://inference.hetzner.com/api/v1\",\n api_key=\"YOUR_TOKEN\",\n)\n\nresponse = client.chat.completions.create(\n model=\"Qwen/Qwen3.6-35B-A3B-FP8\",\n messages=[\n {\"role\": \"user\", \"content\": \"Explain why the sky is blue in one sentence.\"}\n ],\n extra_body={\n \"chat_template_kwargs\": {\n \"enable_thinking\": False,\n }\n },\n)\n\nprint(response.choices[0].message.content)\n```\n\nOne detail: the `enable_thinking`\n\nflag in `extra_body`\n\n. If you leave it enabled, the model might burn through your token budget reasoning internally before giving you a final answer. It's not officially documented, so don't bet your production stability on it.\n\nPerformance-wise, the numbers looked suspiciously good in my brief tests:\n\n**Median time to first token:** 153 ms**Output speed:** 224 tokens per second\n\nThat's fast, but remember this is a low-load environment. As for the model quality? It's a mixed bag. It handles images and formatting well, but it tripped over basic arithmetic. It's a small, slightly flawed model, but that's expected for an FP8-quantized MoE.\n\nThe real question isn't whether the model is great—it's why Hetzner is doing this. Open-weight inference is becoming a commodity. Everyone is running the same weights on the same serving stacks. If Hetzner figures out how to scale this efficiently, they're playing a very different game than just selling VMs.\n\n[Next MoE Model Layout: A Deep Dive into Weight Reordering →](/en/threads/2606/)", "url": "https://wpnews.pro/news/hetzner-inference-a-practical-deep-dive", "canonical_source": "https://promptcube3.com/en/threads/2657/", "published_at": "2026-07-24 01:47:09+00:00", "updated_at": "2026-07-24 10:10:07.414917+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-products"], "entities": ["Hetzner", "Qwen/Qwen3.6-35B-A3B-FP8"], "alternates": {"html": "https://wpnews.pro/news/hetzner-inference-a-practical-deep-dive", "markdown": "https://wpnews.pro/news/hetzner-inference-a-practical-deep-dive.md", "text": "https://wpnews.pro/news/hetzner-inference-a-practical-deep-dive.txt", "jsonld": "https://wpnews.pro/news/hetzner-inference-a-practical-deep-dive.jsonld"}}