Show HN: OSS, K8s-native AI platform for distributed multi-model inference

Axem co-founders open sourced Shaide, a Kubernetes-native AI platform for distributed multi-model inference, under the Apache 2.0 license. The platform uses vLLM for inference and llm-d for orchestration, supports independent scaling of multiple models across GPU nodes, and can operate fully air-gapped. It is designed to run on existing Kubernetes clusters, including on-prem RKE2, EKS, GKE, and AKS, and provides an OpenAI-compatible API.

Hi everyone, I’m one of the co-founders of axem. We recently open sourced Shaide, a project we’ve been working on to make running multiple LLMs on your own infrastructure less painful. It started pretty simply. Running one model wasn’t the hard part. The hard part came when we needed several models running at the same time, scaling them independently across GPU nodes, routing requests between replicas, and making the whole setup reproducible without relying on an external cloud service. Over time we ended up building most of that infrastructure into one platform, and decided it made more sense to open source it rather than keep it internal. Current setup: - vLLM for inference - llm-d for multi-instance orchestration - multiple models running and scaling independently - KV-cache-aware scheduling i- nternal OCI registry for container images + model weights OpenAI-compatible API - the entire platform is managed as infrastructure as code - interactive installer that runs from Docker against an existing Kubernetes cluster - can operate fully air-gapped with no cluster egress It currently works with on-prem RKE2 as well as EKS/GKE/AKS. The project is Apache 2.0 and still fairly early, so there are definitely things that will change as more people try it. Comments URL: https://news.ycombinator.com/item?id=49521905 https://news.ycombinator.com/item?id=49521905 Points: 1 Comments: 0