{"slug": "orbit-an-open-source-toolkit-for-retrieval-based-inference", "title": "Orbit, an Open-Source Toolkit for Retrieval-Based Inference", "summary": "Orbit, an open-source AI gateway for retrieval-based inference, has been released, enabling self-hosted private RAG, natural-language data access, and tool-calling agents across 37+ model providers. The toolkit supports querying SQL, NoSQL, vector stores, APIs, and files in plain English, with admin controls for keys, quotas, prompts, metrics, and audit logs.", "body_md": "**A self-hosted, OpenAI-compatible AI gateway for private RAG, natural-language data access, and tool-calling agents — run it in your own environment across 37+ model providers.**\n\n[ Quick Start](#-quick-start)\n•\n\n[•](#-demos)\n\n**Demos**[•](#-features)\n\n**Features**[•](/schmitech/orbit/blob/main/docs/tutorial.md)\n\n**Tutorial**\n\n**Docs** ORBIT is a self-hosted AI gateway that lets you chat with private data through one OpenAI-compatible endpoint. Put it in front of local or cloud models, connect files, SQL, NoSQL, vector stores, APIs, and tools, then operate everything from an admin panel with keys, quotas, prompts, metrics, and audit logs.\n\n**Reach for ORBIT when you need to:**\n\n- 💬 Query SQL, NoSQL, vector stores, REST/GraphQL APIs, Elasticsearch, and files in plain English\n- 🔀 Route one API contract across local (Ollama, llama.cpp, vLLM) and cloud models\n- 🛠️ Give models scoped access to MCP tools and multi-step agent workflows\n- 🔒 Ship private RAG with auth, quotas, moderation, metrics, and admin controls\n- 📨 Drive inference asynchronously over a message queue for decoupled, batch-style workloads\n- 🧪 Prototype locally, then keep the same architecture in production\n\nComparisons:\n\n[ORBIT vs. Open WebUI]·[ORBIT vs. LiteLLM]\n\n## multimodal.mp4\n\n*Upload private files, ask questions in chat, and keep the whole workflow behind your own gateway.*\n\nInstall the latest stable release, then start ORBIT locally.\n\n```\ncurl -LO https://github.com/schmitech/orbit/releases/download/v2.9.1/orbit-2.9.1.tar.gz\ntar -xzf orbit-2.9.1.tar.gz && cd orbit-2.9.1\n./install/setup.sh        # add --wizard for interactive setup\n./bin/orbit.sh start\n```\n\nOpen the admin panel:\n\n| URL | Login |\n|---|---|\n|\n\n`admin`\n\n/ `admin123`\n\nVerify the gateway:\n\n```\ncurl -X POST http://localhost:3000/v1/chat \\\n  -H 'Content-Type: application/json' \\\n  -H 'X-API-Key: default-key' \\\n  -H 'X-Session-ID: local-test' \\\n  -d '{\"messages\": [{\"role\": \"user\", \"content\": \"Summarize ORBIT in one sentence.\"}], \"stream\": false}'\n```\n\nOpen a browser chat client:\n\n```\nORBIT_ADAPTER_KEYS='{\"simple-chat\":\"default-key\"}' npx orbitchat --open\n```\n\nWhat you should see: the server responds through the OpenAI-compatible `/v1/chat`\n\nendpoint, the admin panel shows live health and adapters, and OrbitChat opens a local chat UI.\n\nFor a Docker-based local demo with Ollama, use the development compose stack:\n\n```\ngit clone https://github.com/schmitech/orbit.git\ncd orbit/docker\ndocker compose up -d\n```\n\nThe first Docker run can take a few minutes while the local model is downloaded.\n\nWarning\n\nDocker from `main`\n\nis for local evaluation and development. For production installs, use the latest stable release tarball from [GitHub Releases](https://github.com/schmitech/orbit/releases).\n\nMore setup paths: [Docker Guide](/schmitech/orbit/blob/main/docker/README.md) · [Tutorial](/schmitech/orbit/blob/main/docs/tutorial.md) · [Windows native install](/schmitech/orbit/blob/main/install/windows.md)\n\nIf ORBIT saves you setup time, a GitHub star helps other developers find it.\n\nPick the path closest to what you want to build.\n\n| Demo | What it shows | Run it |\n|---|---|---|\nChat with private files |\nUpload PDFs, spreadsheets, and images, then query them in one thread. |\n|\n\n**Ask SQL questions in English**[Tutorial](/schmitech/orbit/blob/main/docs/tutorial/sql-database-sqlite.md)** Operate the gateway**`/admin`\n\n.[First chat](/schmitech/orbit/blob/main/docs/tutorial/first-chat.md)**Chat with private files**\n\n## multimodal.mp4\n\n*Analysts stop hunting across files: upload PDFs, spreadsheets, and images, then query them together in one thread with context cached across the conversation.*\n\n**Ask SQL questions in English**\n\n## db-query.mp4\n\n*No SQL and no ticket queue: ask in plain English, and ORBIT generates the query, runs it against your database, and charts the result in chat.*\n\n**Operate the gateway from the admin panel**\n\n## orbit-admin.mp4\n\n*Monitor health, latency, tokens, sessions, adapters, and logs from one dashboard behind API keys, quotas, and rate limits.*\n\n| Capability | What you get |\n|---|---|\nOpenAI-compatible gateway |\nOne `/v1/chat` interface across local, self-hosted, and cloud providers. |\nModel routing (37+ providers) |\nLocal: Ollama, llama.cpp, vLLM, TensorRT-LLM, Transformers, LM Studio, BitNet. Cloud: OpenAI, Anthropic, Gemini, Bedrock, Vertex, Azure, Groq, Mistral, DeepSeek, xAI, and\n|\n\n**Natural-language data access****Vector RAG****File & multimodal RAG****Pluggable file storage****File encryption at rest****Cloud secrets management**`.env`\n\n— selected with one config key. [Guide](/schmitech/orbit/blob/main/docs/security/secrets-management-setup.md).**Web search****MCP tool agents****A2A peer protocol**[Google Agent-to-Agent](https://google.github.io/A2A/)support — discovery via`/.well-known/agent.json`\n\nand task delegation over JSON-RPC. [Guide](/schmitech/orbit/blob/main/docs/a2a-protocol.md).**Message-queue (async) surface**[Guide](/schmitech/orbit/blob/main/docs/server.md#message-queue-async-protocol).** Media generation****Voice (STT/TTS)****Production controls****Config-first**📚 Deep dive: [Docs index](/schmitech/orbit/blob/main/docs/README.md) · [Adapter guide](/schmitech/orbit/blob/main/docs/adapters/adapters.md) · [Configuration](/schmitech/orbit/blob/main/docs/configuration.md)\n\nSame gateway, different jobs. ORBIT is useful anywhere a team needs private data access, controlled model routing, and operational visibility behind one API.\n\n| Business outcome | Who feels the pain | What it replaces |\n|---|---|---|\nAnswers from scattered documents |\nInsurance, legal, finance, research | Hours spent manually cross-referencing PDFs, spreadsheets, and scans |\nSelf-service data access |\nRetail, operations, finance, SaaS | BI ticket backlogs and waiting on the data team to write SQL |\nFaster incident response |\nSRE, DevOps, security ops | Bottlenecks where only a few specialists can write query DSL |\nAutomated internal workflows |\nIT ops, support, back-office | Manual glue work across databases, Slack, files, and tickets |\nConversational tool use, zero orchestration code |\nProduct, support, SaaS builders | Bolt-on agent frameworks (LangChain, AutoGen) just to get tool calling |\nAI on regulated data |\nHealthcare, government, legal, banking | Compliance blocks on sending sensitive data to cloud LLMs |\nGoverned AI rollout |\nAny regulated or enterprise org | Shadow AI with no audit trail, cost control, or visibility |\nIn-flow content generation |\nMarketing, e-learning, communications | Slow, fragmented media-production tooling |\n\n**Cut incident response time by searching logs in plain English**\n\n## es-logs.mp4\n\n*Investigations no longer stall on DSL expertise: ask operational questions naturally, and ORBIT compiles the Elasticsearch Query DSL and returns the answer.*\n\n**Automate multi-step workflows across your internal tools**\n\n## mcp-tool-demo.mp4\n\n*Replace manual glue work: give models scoped, server-side access to MCP tools — filesystem, Slack, Postgres, GitHub, Jira — with bounded agent loops.*\n\n**Let the model call tools on its own, like ChatGPT or Claude**\n\nBeyond explicit, client-requested tool skills, any conversational adapter can\nopt into **opportunistic MCP tool calling**: the model decides,\nturn by turn, whether an external tool is needed and calls it inline — no\n`skill`\n\nfield, no adapter swap, same thread throughout. Multi-step\nchains, self-correction from tool errors, and mixing several tools in one turn\nall work out of the box, powered by providers' native function calling with\n**no LangChain, AutoGen, or CrewAI dependency** — just a small,\nbounded, fully-owned server-side loop. See the\n[opportunistic mode guide](/schmitech/orbit/blob/main/docs/adapters/mcp-agent.md#opportunistic-mode-mcp_tools-capability).\n\n**Keep regulated data in-house by running AI fully offline**\n\n## sensitive-data.mp4\n\n*Meet data-residency and compliance rules: run local llama.cpp/Ollama models so sensitive PII never leaves your environment.*\n\n**Generate on-brand media inside the same chat flow**\n\n## puppy.mp4\n\n*Produce content without leaving the conversation: generate images and video as cross-adapter skills that carry conversation context.*\n\n**More capabilities**\n\n## svg-rendering.mp4\n\n*Render dynamic LLM-generated SVGs inline.*\n\n## second-opinion.mp4\n\n*Switch inference models mid-conversation without breaking chat history.*\n\n## business-analytics-demo.mp4\n\n*Sub-conversation threading and document caching for faster retrieval.*\n\n| Client | Description |\n|---|---|\nORBIT Chat |\n\n`ORBIT_ADAPTER_KEYS='{\"simple-chat\":\"default-key\"}' npx orbitchat`\n\n**Node.js SDK****Python client**| Topic | Start here |\n|---|---|\nGetting started |\n|\n\n**Configuration**[Configuration guide](/schmitech/orbit/blob/main/docs/configuration.md)·[Adapter config](/schmitech/orbit/blob/main/docs/adapters/adapter-configuration.md)**Adapters & RAG**[Adapters overview](/schmitech/orbit/blob/main/docs/adapters/adapters.md)·[File adapter](/schmitech/orbit/blob/main/docs/adapters/file-adapter-guide.md)**NL data access**[SQL retriever architecture](/schmitech/orbit/blob/main/docs/sql-retriever-architecture.md)·[Intent SQL RAG](/schmitech/orbit/blob/main/docs/intent-sql-rag-system.md)**MCP agents**[MCP agent guide](/schmitech/orbit/blob/main/docs/adapters/mcp-agent.md)** Production ops**[Rate limiting](/schmitech/orbit/blob/main/docs/rate-limiting-architecture.md)·[Fault tolerance](/schmitech/orbit/blob/main/docs/fault-tolerance/fault-tolerance-architecture.md)**Auth & SSO**[Authentication guide](/schmitech/orbit/blob/main/docs/authentication.md)— built-in users, Entra ID & Auth0Contributions are welcome — new retrievers, adapters, and provider integrations; better examples and deployment guides; tests, bug fixes, and docs. Start with [CONTRIBUTING.md](/schmitech/orbit/blob/main/CONTRIBUTING.md), open an [issue](https://github.com/schmitech/orbit/issues), or send a PR. Roadmap and active work live in [GitHub Issues](https://github.com/schmitech/orbit/issues).\n\nMaintained by [ Remsy Schmilinsky](https://www.linkedin.com/in/remsy/).\n\nORBIT is licensed under the Apache 2.0 License. See [LICENSE](/schmitech/orbit/blob/main/LICENSE) for details.", "url": "https://wpnews.pro/news/orbit-an-open-source-toolkit-for-retrieval-based-inference", "canonical_source": "https://github.com/schmitech/orbit", "published_at": "2026-07-09 22:34:15+00:00", "updated_at": "2026-07-09 23:07:19.673094+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools", "ai-infrastructure", "natural-language-processing", "ai-products"], "entities": ["Orbit", "Ollama", "llama.cpp", "vLLM", "OpenAI", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/orbit-an-open-source-toolkit-for-retrieval-based-inference", "markdown": "https://wpnews.pro/news/orbit-an-open-source-toolkit-for-retrieval-based-inference.md", "text": "https://wpnews.pro/news/orbit-an-open-source-toolkit-for-retrieval-based-inference.txt", "jsonld": "https://wpnews.pro/news/orbit-an-open-source-toolkit-for-retrieval-based-inference.jsonld"}}