{"slug": "hetzner-inference-experiment-open-weights-slms-free-of-charge", "title": "Hetzner Inference Experiment: Open-weights SLMs free of charge", "summary": "Hetzner Online GmbH launched its Inference Experiment, a free inference API for open-weights small language models hosted on its infrastructure, one week ago. The company reported that small models like Qwen 3.6 proved sufficient for most OpenClaw-based use cases, and that its GEX131 server with a single RTX PRO 6000 Blackwell GPU was adequate for small LLMs but not larger ones. Hetzner plans to offer larger models using new 8-GPU servers with RTX PRO 6000 Blackwell GPUs.", "body_md": "#### We recently launched our Inference Experiment: an inference API that allows you to use Large Language Models (LLMs) which we host on our infrastructure. You can connect any agent, harness, or coding tool of your choice to it, and use our LLM free of charge. This way, you can try out LLMs running on powerful hardware, while we gain experience running such models.\n\nNow that the Inference Experiment is one week old, we would like to give you some insights into how it works, some of the things we learned, and what the future for it might look like. To do this, we’d like to show you a timeline of the Hetzner Experiment Platform and all experiments on it so far.\n\n## T - 4 months: introducing Hetzner Experiments Platform with OpenClaw\n\nEver since we’ve been offering GPU servers to our customers, we have tested ways of serving LLMs internally. With many years of Kubernetes experience, we were able to build an internal inference platform, which we used to test different server configurations and LLM-serving software.\n\nWe eventually settled on a stack consisting of a few GEX131 servers, vLLM and Open WebUI. This combination allowed us to run a few smaller LLMs, which we could use for some internal production-grade applications. It also allowed us to trial and benchmark larger models internally.\n\nAt the start of 2026, [OpenClaw](https://openclaw.ai/) became a much-hyped topic in the AI space. Many enthusiasts were using Hetzner Cloud VMs to try out OpenClaw, but struggled with safely configuring the server for it. To help our customers with this challenge, we decided to offer OpenClaw as a preconfigured instance.\n\nWe quickly built a new platform, which we could use to offer not only OpenClaw, but also other services and products we might want to experiment with in the future. It would allow rapid iteration and experimentation, while coming with fewer guarantees than our more mature products. One core feature would be the pricing: we would initially offer all Experiments completely for free.\n\nOpenClaw was also an opportunity to gain experience with self-hosted inference. Our OpenClaw came preconfigured with our own inference API. The API was not publicly available but was limited to OpenClaw, so we could control the request volume to a degree. By offering inference only via OpenClaw, we could try out some smaller open-weight models, in various configurations. And our customers would get an AI product that was completely EU-based.\n\nTo show how unique the Hetzner Experiments Platform is, we built a separate interface panel for it at [experiments.hetzner.com](https://experiments.hetzner.com/).\n\nThis limited OpenClaw Experiment gave us lots of insights:\n\n- Small models like Qwen 3.6 proved sufficient for most OpenClaw-based use cases.\n\n- GEX131 with its single RTX PRO™ 6000 Blackwell GPU was sufficient for small LLMs, but couldn’t run large models in a meaningful way.\n\n- Lots of users praised the fact that this was entirely EU-based.\n\n- OpenClaw’s code quality was a constant problem: Each new OpenClaw release introduced new challenges, as the volume of upstream bugs was quite high.\n\n## T - 1 month: getting specialized hardware\n\nAt Hetzner, our hardware teams are constantly working on new server models. One such model was a new custom server, which houses eight RTX PRO™ 6000 Blackwell GPUs in a single server, instead of the single GPU of a GEX131. This would be much better suited for running larger models.\n\nWith some of these 8-GPU servers, we could offer larger models to our customers. The OpenClaw Experiment had already shown that our customers are interested in AI topics and inference. However, the software quality of OpenClaw remained a challenge.\n\nTo solve this, we decided to offer inference directly as an OpenAI-spec API that everyone could use, without having to go through OpenClaw. All they would need is an auth token, which they could generate via [experiments.hetzner.com](https://experiments.hetzner.com/), in the separate “Inference” tab.\n\n## T - 2 weeks: benchmarking and picking the right models\n\nWhat followed was a lot of benchmarking and market research.\n\nWe came up with a list of LLMs we wanted to offer:\n\n• [GLM-5.2](https://huggingface.co/zai-org/GLM-5.2)\n\nQwen would continue from the OpenClaw Experiment as a smaller, “baseline” model. Kimi would offer a coding-focused alternative to the two largest models, GLM and DeepSeek. Each of those models represented the newest version of the respective LLM family.\n\nThe only exception to this was Kimi: K3 had already come out, and promised great results in public benchmarks. However, K3’s very specific license requirements did not allow us to offer it as a free experiment.\n\n## T - 1 week: final preparations\n\nAs final preparations for the Inference Experiment, some internal components were overhauled to deal with the expected high demand.\n\nRouting for inference requests was greatly improved by introducing llm-d and making changes to some routing components we had developed in-house. Features such as kv-cache offloading into the host’s RAM also promised substantial performance improvements.\n\nWe also added some more of the custom 8-GPU servers, and dedicated each one to running either DeepSeek, GLM or Kimi. And we dedicate the GEX131 single-GPU servers to serving the smaller Qwen 3.6 model.\n\n## T - 0 hours: launch\n\nDemand for the Inference Experiment was hard to judge up front. To be on the safe side, we staggered the launch over multiple days:\n\nOn August 7th, we enabled the new models and the Inference Experiment on the Hetzner Experiments Platform. This gave early users the weekend to discover any issues we might have missed. After the weekend, on August 10th, we started actively communicating the experiment to our customers. We expected that word of mouth would eventually increase demand as well.\n\n## T + 6 hours: hitting capacity limits\n\nWord was spreading fast: within a few hours, we noticed a sharp increase in requests to the inference API. The demand exceeded our most optimistic expectations, and continued to grow at a high rate.\n\nQwen 3.6, which was the default model for OpenClaw, proved the most popular on the first day, both in requests and generated tokens. The other models also saw high demand, but trailed Qwen by quite a bit.\n\nOur setup was able to generate enough token throughput to keep up with the requests, but we already noticed an increase in latency:\n\nThe 99th percentile for the time to first token (TTFT) had seen short spikes for some models, but it generally still performed reasonably well, sitting at around 5 to 10 seconds for the majority of the time.\n\nAnother surprise was the huge increase in OpenClaw instances. While a bit more demand was expected, the number of instances skyrocketed on the first day. Our small Kubernetes cluster for running OpenClaw instances was rapidly reaching its capacity limit. So we disabled the OpenClaw tab on [experiments.hetzner.com](https://experiments.hetzner.com/) for every account that hadn’t signed in to Hetzner Experiments Platform yet.\n\n## T + 1 day: coping with the demand\n\nOn the second day of the experiment, the number of requests for all models increased drastically, yet again.\n\nUsage patterns started to become clear:\n\n- DeepSeek saw the most interest. Many users reported favorable benchmark results.\n\n- Qwen remained popular, with very many small requests.\n\n- GLM and Kimi were also in high demand, but much less so than the other two models. GLM saw the highest latency of all, and needed more fine tuning.\n\n- We do not log any prompt data, but our token metrics implied that many users were still running benchmarks at this time.\n\nThis number of requests was exceeding the capacity of our experiment infrastructure, however. Time to first token (TTFT) went through the roof, peaking at almost 10 minutes for the 99th percentile, rendering the API effectively unusable at times. Our experiment had not been designed for this scale of requests, and it showed.\n\nThe high number of small, concurrent requests was especially challenging. To better deal with this, we reduced the number of parallel requests we would accept.\n\nWe also received lots of feedback surrounding the reasoning settings of the various models, which some users found unintuitive.\n\n## T + 2 days: more adjustments, more hardware\n\nDemand for inference increased even more, pushing our inference infrastructure to the limit. While the number of requests to the API endpoint was dropping, the overall number of input tokens kept rising.\n\nIn an effort to provide more raw power, we were able to build a few more special 8-GPU servers, and added them to the pool serving GLM as that model was struggling the most at the time.\n\nUsage patterns now settled into what we’d see for the rest of the week:\n\n- DeepSeek saw more and more complex requests, with huge amounts of input and output tokens.\n\n- Qwen continued to see a very high number of smaller requests.\n\n- GLM and Kimi saw medium-sized requests, but on a lower level than the previous two.\n\n- Requests were larger on average than on the first day. This implied more use for practical purposes, with large real-world datasets.\n\n- All models saw higher throughput thanks to our tweaks.\n\n- During peak times, a high number of requests had to spend minutes waiting in queue before a GPU server could even start dealing with the inference request.\n\nWe tried many improvements over the course of the week: General inference rate limits were adjusted down to give more users a chance to try the experiment. Some usage patterns seemed malicious, so we took steps to rate limit and filter those. Actual usage was also a different pattern than our benchmarks. We learned that, for some models, a lower number of concurrent requests per instance was the most optimal with regard to throughput. With this change, the overall token throughput increased for the experiment.\n\nWe also expanded the hardware pool for OpenClaw. Early users of OpenClaw kept creating even more OpenClaw instances. The number of software issues with OpenClaw also increased, however: Multiple OpenClaw instances had actually misconfigured themselves when prompted by the users, and ended up in a broken state.\n\n## T + 7 days: taking stock and scaling down the experiment\n\nLooking back after one week, we have learned a lot:\n\nDemand is much higher than we expected, both in terms of users as well as requests and tokens per request. The 8-GPU servers have proven that they can run the large models, but the demand was simply too high for the few servers in our experiment. Metrics also showed us some bottlenecks in the software used for routing and serving requests.\n\nAddressing some of those bottlenecks will require taking hardware out of the pool for the inference API in order to test some changes. We will also look into improving our documentation by adding more details and tutorials to make it easier to use for beginners.\n\nWe will also review our selection of LLMs based on the metrics we saw. We might also look at other use cases like re-ranking or embedding, as used by RAG pipelines. In addition, we will critically review the OpenClaw Experiment, and the support effort required for it.\n\nThe Inference Experiment will continue, but in a reduced scope: We’ll continue serving small models such as Qwen 3.6 and 3.8, but will take a break from serving large models.\n\nTry it out for yourself at [experiments.hetzner.com](https://experiments.hetzner.com/). We’ll continue to launch new projects and experiment to see what else might be possible. Stay tuned to see what comes next.", "url": "https://wpnews.pro/news/hetzner-inference-experiment-open-weights-slms-free-of-charge", "canonical_source": "https://www.hetzner.com/blog/inference-experiment/", "published_at": "2026-08-19 15:54:11+00:00", "updated_at": "2026-08-19 16:15:14.801930+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-products"], "entities": ["Hetzner Online GmbH", "OpenClaw", "Qwen 3.6", "GEX131", "RTX PRO 6000 Blackwell", "vLLM", "Open WebUI"], "alternates": {"html": "https://wpnews.pro/news/hetzner-inference-experiment-open-weights-slms-free-of-charge", "markdown": "https://wpnews.pro/news/hetzner-inference-experiment-open-weights-slms-free-of-charge.md", "text": "https://wpnews.pro/news/hetzner-inference-experiment-open-weights-slms-free-of-charge.txt", "jsonld": "https://wpnews.pro/news/hetzner-inference-experiment-open-weights-slms-free-of-charge.jsonld"}}