# Hetzner Inference: A Practical Deep Dive

> Source: <https://promptcube3.com/en/threads/2657/>
> Published: 2026-07-24 01:47:09+00:00

# Hetzner Inference: A Practical Deep Dive

The setup is an OpenAI-compatible API. You grab a token from their experiments dashboard, swap the base URL in your client, and you're in. Currently, the only model available is `Qwen/Qwen3.6-35B-A3B-FP8`

. It's a Mixture-of-Experts model with a 262K context window. It's a strategic choice: small enough to run without a massive GPU farm, but capable enough to handle actual prompts.

Integration is trivial since it follows the standard OpenAI spec. I put together this quick deployment snippet for anyone wanting to test it:

``` python
from openai import OpenAI

client = OpenAI(
 base_url="https://inference.hetzner.com/api/v1",
 api_key="YOUR_TOKEN",
)

response = client.chat.completions.create(
 model="Qwen/Qwen3.6-35B-A3B-FP8",
 messages=[
 {"role": "user", "content": "Explain why the sky is blue in one sentence."}
 ],
 extra_body={
 "chat_template_kwargs": {
 "enable_thinking": False,
 }
 },
)

print(response.choices[0].message.content)
```

One detail: the `enable_thinking`

flag in `extra_body`

. If you leave it enabled, the model might burn through your token budget reasoning internally before giving you a final answer. It's not officially documented, so don't bet your production stability on it.

Performance-wise, the numbers looked suspiciously good in my brief tests:

**Median time to first token:** 153 ms**Output speed:** 224 tokens per second

That's fast, but remember this is a low-load environment. As for the model quality? It's a mixed bag. It handles images and formatting well, but it tripped over basic arithmetic. It's a small, slightly flawed model, but that's expected for an FP8-quantized MoE.

The real question isn't whether the model is great—it's why Hetzner is doing this. Open-weight inference is becoming a commodity. Everyone is running the same weights on the same serving stacks. If Hetzner figures out how to scale this efficiently, they're playing a very different game than just selling VMs.

[Next MoE Model Layout: A Deep Dive into Weight Reordering →](/en/threads/2606/)