I’m the Linux admin in a company that does print stuff for their customers. Think invoices, medical reports and so on. Privacy is extremely important, and our customers are demanding 100% on-premise only IT.
In 2023 i started playing with llama.cpp on some of our beefy compute nodes. It ended in Deepseek R1 671b oneshotting a working flappy bird clone at 0.7 t/s decode speed over night.
C-level ignored my pleas for dedicated hardware, until end of 2025, when i got a 2U Gigabyte with 4x 6000 Blackwell MaxQ. I was quick to setup OpenWebUI and a bunch of models for testing.
Our business is slow, not much happens. There is no software left to write, no sudden changes, no need for a proper AI strategy. This isn’t the heart of Meta or Google, it’s the same business as ist was 20 years ago.
However, users started to use ChatGPT and other AI services for their office work, reformatting emails for example. C-level allowed it, as long as the data is sanitized. However, that just isn’t possible, as we have seen a few times. But our local AI wasn’t ready, until Qwen 3.5 122b happened. Since then we are 100% local. We then went to 3.6 27b and now 3.8 27b. We want to use Deepseek V4 Flash 0713, but can’t get it under control yet. People prefer 2.8 27b.
Other apps we run, besides OpenWebUI, are Vexa AI, a Teams bot that uses speech to text to make transcipts, FIM models in VSCode, Hermes or OpenCode inside VMs on the developer workstations, Qdrant for RAG databases, ComfyUI, and N8N. We are thinking about getting another box for redundancy and load balancing.
It’s a pretty straight setup, but if there is interest i can share some details.