Show HN: Open-source verification and tuning layer for self-hosted LLMs

Thomas, a machine learning engineer, open-sourced Coloma, a verification and tuning layer for self-hosted large language models. The tool includes a vLLM tuner to prevent out-of-memory errors and optimize for latency or throughput, plus an output verification system that runs during GPU idle time for continuous improvement.

Hi HN, Thomas here, I'm a Machine Learning, and I've been deploying models in prod for the last 5 years and decided it was time to open-source some of my learnings In Coloma you'll find my vLLM tuner, I use it to ensure models don't OOM at runtime, and are optimized for the metric that matters the most for that deployment: low per-user latency vs maximum system throughput. To make it easier the dashboard ships with an estimator of the performance expected at runtime depending on the parameters picked. I've also started to add my outputs verification system that triggers when the GPUs are idling, so that at runtime the predictions can run fast, and get flagged later on for continuous improvement. Really looking forward to see if it that can help someone Comments URL: https://news.ycombinator.com/item?id=48949625 https://news.ycombinator.com/item?id=48949625 Points: 1 Comments: 0