{"slug": "show-hn-open-source-verification-and-tuning-layer-for-self-hosted-llms", "title": "Show HN: Open-source verification and tuning layer for self-hosted LLMs", "summary": "Thomas, a machine learning engineer, open-sourced Coloma, a verification and tuning layer for self-hosted large language models. The tool includes a vLLM tuner to prevent out-of-memory errors and optimize for latency or throughput, plus an output verification system that runs during GPU idle time for continuous improvement.", "body_md": "Hi HN,\n\nThomas here, I'm a Machine Learning, and I've been deploying models in prod for the last 5 years and decided it was time to open-source some of my learnings!\n\nIn Coloma you'll find my vLLM tuner, I use it to ensure models don't OOM at runtime, and are optimized for the metric that matters the most for that deployment: low per-user latency vs maximum system throughput. To make it easier the dashboard ships with an estimator of the performance expected at runtime depending on the parameters picked.\n\nI've also started to add my outputs verification system that triggers when the GPUs are idling, so that at runtime the predictions can run fast, and get flagged later on for continuous improvement.\n\nReally looking forward to see if it that can help someone!\n\nComments URL: [https://news.ycombinator.com/item?id=48949625](https://news.ycombinator.com/item?id=48949625)\n\nPoints: 1\n\n# Comments: 0", "url": "https://wpnews.pro/news/show-hn-open-source-verification-and-tuning-layer-for-self-hosted-llms", "canonical_source": "https://github.com/colomalabs/coloma", "published_at": "2026-07-17 17:03:43+00:00", "updated_at": "2026-07-17 17:21:05.778849+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-infrastructure", "mlops", "developer-tools"], "entities": ["Thomas", "Coloma", "vLLM"], "alternates": {"html": "https://wpnews.pro/news/show-hn-open-source-verification-and-tuning-layer-for-self-hosted-llms", "markdown": "https://wpnews.pro/news/show-hn-open-source-verification-and-tuning-layer-for-self-hosted-llms.md", "text": "https://wpnews.pro/news/show-hn-open-source-verification-and-tuning-layer-for-self-hosted-llms.txt", "jsonld": "https://wpnews.pro/news/show-hn-open-source-verification-and-tuning-layer-for-self-hosted-llms.jsonld"}}