Show HN: Modelship – an open source ray backed model cluster

Modelship, an open source Ray-backed model cluster, was released on GitHub, scoring 17/17 on the Open Responses conformance suite according to its production readiness documentation. The project spins up a Ray cluster, deploys models via loaders including vllm, llamacpp, diffusers and sherpa_onnx, and serves an OpenAI responses-ready API on top of them, supporting CPU, Metal and CUDA with Prometheus metrics, a Grafana dashboard, Docker, Kubernetes or native installs, automatic model context sizing and MCP support.

Hey all, just wanted to share how my spare time has been spent for quite some time now. Ok, this one started oddly. My aim initially was just to serve multiple models on one GPU and fire it up in my unraid machine. By the time that first bit was done, I realised I could just add one more thing... and another... and well, from beginning of this year when, imho, AI became useful, the whole thing it kind of blew up. So here's what it does: - spins a ray cluster - deploys models using various loaders vllm, llamacpp, diffusers, sherpa onnx, etc. - serves an openai responses ready api on top of them Now some the details, already listed in my production readiness https://docs.model-ship.ai/production-readiness/ production-... https://docs.model-ship.ai/production-readiness/ production-readiness plan: - scores 17/17 on the Open Responses conformance suite and it's got its own responses redis backed store - currently supports cpu, metal and cuda - it has prometheus metrics and grafana dashboard - can run under docker, k8s or simply native install - automatically sizes model context - has MCP support Feedback would be much appreciated. https://github.com/modelship-ai/modelship https://github.com/modelship-ai/modelship Comments URL: https://news.ycombinator.com/item?id=50040528 https://news.ycombinator.com/item?id=50040528 Points: 1 Comments: 0