Show HN: Mantis, A self-hosted LLM gateway

Mantis, an open-source self-hosted LLM gateway, launched to provide teams with a unified API for multiple model targets, centralizing routing, failover, caching, guardrails, observability, and AWS-native deployment. The project targets small teams seeking control over infrastructure and data while simplifying multi-LLM application development.

Mantis is an open-source, self-hosted LLM gateway for teams building applications across multiple model targets. It gives client applications one stable chat-completions API while centralizing routing policy, failover behavior, response caching, guardrails, observability, and AWS deployment configuration. The project is designed for small teams that want the benefits of an LLM gateway without giving up control of their infrastructure or data. One API for LLM calls: send chat-completion requests through a single gateway endpoint instead of integrating directly with each provider. Configurable routing: route by metadata, model aliases, weighted targets, fallback chains, retries, timeouts, and cooldowns. Response caching: reduce repeated LLM calls with exact prompt caching and optional semantic caching. Guardrails: use AWS Bedrock guardrails to mask sensitive data and block policy-violating prompts or responses. Observability: capture request IDs, latency, token usage, cache behavior, errors, and request outcomes through CloudWatch. AWS-native deployment: provision and run Mantis with Terraform, ECS Fargate, ALB, ElastiCache, Parameter Store, S3, IAM, and CloudWatch. llm-gateway https://github.com/mantis-llm-gateway/llm-gateway : the FastAPI gateway service, React configuration dashboard, Terraform infrastructure, and deployment scripts. mantis-sdk https://github.com/mantis-llm-gateway/mantis-sdk : a Python SDK for calling the Mantis /v1/chat/completions endpoint from application code. mantis-llm-gateway.github.io https://github.com/mantis-llm-gateway/mantis-llm-gateway.github.io : the public documentation site and case study. - Read the documentation https://mantis-llm-gateway.github.io/ for the project overview, guides, API reference, and architecture case study. - Follow the quick start https://mantis-llm-gateway.github.io/guides/quick-start/ to run or deploy the gateway. - Review the routing configuration guide https://mantis-llm-gateway.github.io/guides/routing-config/ to understand how model selection, fallback, caching, and cooldown behavior are controlled. Mantis exists to make multi-LLM application development more reliable, observable, and operationally manageable. Instead of spreading provider-specific logic across application code, teams can put model routing, cache policy, failover behavior, guardrails, and deployment concerns behind one gateway layer. The result is a system where application code stays simple, model choices remain configurable, and teams keep control over how requests move through their own AWS environment.