{"slug": "shepherd-model-gateway-model-routing-gateway-for-llm-deployments", "title": "Shepherd Model Gateway – Model-routing gateway for LLM deployments", "summary": "LightSeek Org released Shepherd Model Gateway (SMG), an engine-agnostic, high-performance model-routing gateway for large-scale LLM deployments that centralizes worker lifecycle management and balances traffic across HTTP, gRPC, and OpenAI-compatible backends. The gateway supports cache-aware routing for vLLM, TensorRT-LLM, TokenSpeed, and SGLang, offers multi-tenant rate limiting with OIDC and WebAssembly plugins, and provides 40+ Prometheus metrics with OpenTelemetry tracing. SMG is available via Docker, pip, and cargo, and can be run with a single command to point at inference workers.", "body_md": "Engine-agnostic, high-performance model-routing gateway for large-scale LLM deployments. Centralizes worker lifecycle management, balances traffic across HTTP/gRPC/OpenAI-compatible backends, and provides enterprise-ready control over history storage, MCP tooling, and privacy-sensitive workflows.\n\n🚀 Maximize GPU Utilization |\nCache-aware routing understands your inference engine's KV cache state—whether vLLM, TensorRT-LLM, TokenSpeed, or SGLang—to reuse prefixes and reduce redundant computation. |\n🔌 One API, Any Backend |\nRoute to self-hosted models (vLLM, TensorRT-LLM, TokenSpeed, SGLang) or cloud providers (OpenAI, Anthropic, Gemini, Bedrock, and more) through a single unified endpoint. |\n⚡ Built for Speed |\nNative Rust with gRPC pipelines, sub-millisecond routing decisions, and zero-copy tokenization. Circuit breakers and automatic failover keep things running. |\n🔒 Enterprise Control |\nMulti-tenant rate limiting with OIDC, WebAssembly plugins for custom logic, and a privacy boundary that keeps conversation history within your infrastructure. |\n📊 Full Observability |\n40+ Prometheus metrics, OpenTelemetry tracing, and structured JSON logs with request correlation—know exactly what's happening at every layer. |\n\n**API Coverage:** OpenAI Chat/Completions/Embeddings, Responses API for agents, Anthropic Messages, and MCP tool execution.\n\n**Install** — pick your preferred method:\n\n```\n# Docker\ndocker pull lightseekorg/smg:latest\n\n# Python\npip install smg\n\n# Rust\ncargo install smg\n```\n\n**Run** — point SMG at your inference workers:\n\n```\n# Single worker\nsmg launch --worker-urls http://localhost:8000\n\n# Multiple workers with cache-aware routing\nsmg launch --worker-urls http://gpu1:8000 http://gpu2:8000 --policy cache_aware\n\n# With high availability mesh\nsmg launch --worker-urls http://gpu1:8000 --enable-mesh \\\n  --mesh-advertise-host 10.0.0.1 --mesh-peer-urls 10.0.0.2:39527\n```\n\n**Use** — send requests to the gateway:\n\n```\ncurl http://localhost:30000/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\"model\": \"llama3\", \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}]}'\n```\n\nThat's it. SMG is now load-balancing requests across your workers.\n\n| Self-Hosted | Cloud Providers |\n|---|---|\n| vLLM | OpenAI |\n| TensorRT-LLM | Anthropic |\n| TokenSpeed | Google Gemini |\n| SGLang | AWS Bedrock |\n| Ollama | Azure OpenAI |\n| Any OpenAI-compatible server | Any OpenAI-compatible provider |\n\n| Feature | Description |\n|---|---|\n|\ncache_aware, round_robin, power_of_two, consistent_hashing, prefix_hash, manual, random, bucket |\n|\nNative gRPC with streaming, reasoning extraction, and tool call parsing |\n|\nConnect external tool servers via Model Context Protocol |\n|\nMesh networking with SWIM protocol for multi-node deployments |\n|\nPluggable storage: PostgreSQL, Oracle, Redis, or in-memory |\n|\nExtend with custom WebAssembly logic |\n|\nCircuit breakers, retries with backoff, rate limiting |\n\n|\n\n[Architecture](/lightseekorg/smg/blob/main/docs/concepts/architecture/overview.md)[Configuration](/lightseekorg/smg/blob/main/docs/reference/configuration.md)[API Reference](/lightseekorg/smg/blob/main/docs/reference/api/openai.md)[Kubernetes Setup](/lightseekorg/smg/blob/main/docs/getting-started/service-discovery.md)We welcome contributions! See [Contributing Guide](/lightseekorg/smg/blob/main/docs/contributing/index.md) for details.", "url": "https://wpnews.pro/news/shepherd-model-gateway-model-routing-gateway-for-llm-deployments", "canonical_source": "https://github.com/lightseekorg/smg", "published_at": "2026-07-26 22:36:58+00:00", "updated_at": "2026-07-26 22:52:23.649850+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "developer-tools"], "entities": ["LightSeek Org", "Shepherd Model Gateway", "vLLM", "TensorRT-LLM", "TokenSpeed", "SGLang", "OpenAI", "Anthropic"], "alternates": {"html": "https://wpnews.pro/news/shepherd-model-gateway-model-routing-gateway-for-llm-deployments", "markdown": "https://wpnews.pro/news/shepherd-model-gateway-model-routing-gateway-for-llm-deployments.md", "text": "https://wpnews.pro/news/shepherd-model-gateway-model-routing-gateway-for-llm-deployments.txt", "jsonld": "https://wpnews.pro/news/shepherd-model-gateway-model-routing-gateway-for-llm-deployments.jsonld"}}