# Shepherd Model Gateway – Model-routing gateway for LLM deployments

> Source: <https://github.com/lightseekorg/smg>
> Published: 2026-07-26 22:36:58+00:00

Engine-agnostic, high-performance model-routing gateway for large-scale LLM deployments. Centralizes worker lifecycle management, balances traffic across HTTP/gRPC/OpenAI-compatible backends, and provides enterprise-ready control over history storage, MCP tooling, and privacy-sensitive workflows.

🚀 Maximize GPU Utilization |
Cache-aware routing understands your inference engine's KV cache state—whether vLLM, TensorRT-LLM, TokenSpeed, or SGLang—to reuse prefixes and reduce redundant computation. |
🔌 One API, Any Backend |
Route to self-hosted models (vLLM, TensorRT-LLM, TokenSpeed, SGLang) or cloud providers (OpenAI, Anthropic, Gemini, Bedrock, and more) through a single unified endpoint. |
⚡ Built for Speed |
Native Rust with gRPC pipelines, sub-millisecond routing decisions, and zero-copy tokenization. Circuit breakers and automatic failover keep things running. |
🔒 Enterprise Control |
Multi-tenant rate limiting with OIDC, WebAssembly plugins for custom logic, and a privacy boundary that keeps conversation history within your infrastructure. |
📊 Full Observability |
40+ Prometheus metrics, OpenTelemetry tracing, and structured JSON logs with request correlation—know exactly what's happening at every layer. |

**API Coverage:** OpenAI Chat/Completions/Embeddings, Responses API for agents, Anthropic Messages, and MCP tool execution.

**Install** — pick your preferred method:

```
# Docker
docker pull lightseekorg/smg:latest

# Python
pip install smg

# Rust
cargo install smg
```

**Run** — point SMG at your inference workers:

```
# Single worker
smg launch --worker-urls http://localhost:8000

# Multiple workers with cache-aware routing
smg launch --worker-urls http://gpu1:8000 http://gpu2:8000 --policy cache_aware

# With high availability mesh
smg launch --worker-urls http://gpu1:8000 --enable-mesh \
  --mesh-advertise-host 10.0.0.1 --mesh-peer-urls 10.0.0.2:39527
```

**Use** — send requests to the gateway:

```
curl http://localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "llama3", "messages": [{"role": "user", "content": "Hello!"}]}'
```

That's it. SMG is now load-balancing requests across your workers.

| Self-Hosted | Cloud Providers |
|---|---|
| vLLM | OpenAI |
| TensorRT-LLM | Anthropic |
| TokenSpeed | Google Gemini |
| SGLang | AWS Bedrock |
| Ollama | Azure OpenAI |
| Any OpenAI-compatible server | Any OpenAI-compatible provider |

| Feature | Description |
|---|---|
|
cache_aware, round_robin, power_of_two, consistent_hashing, prefix_hash, manual, random, bucket |
|
Native gRPC with streaming, reasoning extraction, and tool call parsing |
|
Connect external tool servers via Model Context Protocol |
|
Mesh networking with SWIM protocol for multi-node deployments |
|
Pluggable storage: PostgreSQL, Oracle, Redis, or in-memory |
|
Extend with custom WebAssembly logic |
|
Circuit breakers, retries with backoff, rate limiting |

|

[Architecture](/lightseekorg/smg/blob/main/docs/concepts/architecture/overview.md)[Configuration](/lightseekorg/smg/blob/main/docs/reference/configuration.md)[API Reference](/lightseekorg/smg/blob/main/docs/reference/api/openai.md)[Kubernetes Setup](/lightseekorg/smg/blob/main/docs/getting-started/service-discovery.md)We welcome contributions! See [Contributing Guide](/lightseekorg/smg/blob/main/docs/contributing/index.md) for details.