The setup is an OpenAI-compatible API. You grab a token from their experiments dashboard, swap the base URL in your client, and you're in. Currently, the only model available is Qwen/Qwen3.6-35B-A3B-FP8
. It's a Mixture-of-Experts model with a 262K context window. It's a strategic choice: small enough to run without a massive GPU farm, but capable enough to handle actual prompts.
Integration is trivial since it follows the standard OpenAI spec. I put together this quick deployment snippet for anyone wanting to test it:
from openai import OpenAI
client = OpenAI(
base_url="https://inference.hetzner.com/api/v1",
api_key="YOUR_TOKEN",
)
response = client.chat.completions.create(
model="Qwen/Qwen3.6-35B-A3B-FP8",
messages=[
{"role": "user", "content": "Explain why the sky is blue in one sentence."}
],
extra_body={
"chat_template_kwargs": {
"enable_thinking": False,
}
},
)
print(response.choices[0].message.content)
One detail: the enable_thinking
flag in extra_body
. If you leave it enabled, the model might burn through your token budget reasoning internally before giving you a final answer. It's not officially documented, so don't bet your production stability on it.
Performance-wise, the numbers looked suspiciously good in my brief tests:
Median time to first token: 153 msOutput speed: 224 tokens per second
That's fast, but remember this is a low-load environment. As for the model quality? It's a mixed bag. It handles images and formatting well, but it tripped over basic arithmetic. It's a small, slightly flawed model, but that's expected for an FP8-quantized MoE.
The real question isn't whether the model is great—it's why Hetzner is doing this. Open-weight inference is becoming a commodity. Everyone is running the same weights on the same serving stacks. If Hetzner figures out how to scale this efficiently, they're playing a very different game than just selling VMs.
Next MoE Model Layout: A Deep Dive into Weight Reordering →