{"slug": "i-kept-rebuilding-llm-provider-setup-so-i-built-a-small-go-gateway", "title": "I Kept Rebuilding LLM Provider Setup - So I Built a Small Go Gateway", "summary": "A developer built Go-Ai, a small open-source HTTP gateway written in Go that gives application backends a single stable endpoint for chat-completion requests while handling provider-facing details like API keys, model aliases, and streaming. The gateway resolves local model aliases and proxies OpenAI-compatible requests upstream, letting applications change providers without rewriting each codebase. The developer notes it is not a general AI platform or agent runtime and positions it as a lighter alternative to LiteLLM, LangChain, and Vercel AI SDK for smaller projects.", "body_md": "There is a particular point in an AI feature where my energy tends to drop. The idea is working. I have a screen or a route I want to test. The prompt is good enough to learn something from a real response.\n\nAnd then I am not testing the feature anymore.\n\nI am finding the right provider key for this project, checking whether a model name still works, comparing a streaming response to a non-streaming one, and wondering whether the environment where I deployed the app can use the provider at all. None of that is dramatic work. It is just the same surrounding setup, again and again, sitting between me and the question I actually wanted to answer: does this feature help a user?\n\nThat repetition is why I built **Go-Ai**, a small HTTP gateway for my own projects. It gives application backends one stable place to send chat-completion requests while the gateway deals with provider-facing details. Its boundaries are deliberate: it is not a general AI platform, agent runtime, or replacement for the application that calls it.\n\nSource: [github.com/JakuninOleg/Go-Ai](https://github.com/JakuninOleg/Go-Ai)\n\nI also wanted a real reason to spend time with Go. I had heard good things about it for years, but did not want to learn it through comparisons or a throwaway demo. A compact service with HTTP, configuration, streaming, and clear failure cases gave me a useful problem. Go was a pragmatic choice, not a claim that it is better than every alternative.\n\nThe clearest example for me was a development app where I kept trying chat, streaming, and later voice. Every improvement to the product surface meant revisiting the same provider chores: which key belongs in which environment, whether this host can reach that upstream, and whether a model slug that worked last month still exists. The feature itself was rarely the hard part. The surrounding setup was.\n\nThe applications I was building did not need a grand AI platform. They needed to ask a model a question, sometimes stream the answer, and occasionally move away from a provider-specific model name without hunting through several codebases.\n\nDirect provider integrations make sense, especially for one small application. But with more than one app, credentials need safe homes, upstream model slugs can change, and provider availability can vary by environment or region. I did not want each new application to rediscover those constraints independently.\n\n*Before, each application carries provider setup alongside its own work. After, provider-facing concerns have one small home while applications keep product decisions.*\n\nThere are already strong projects in this area. LiteLLM, LangChain, and Vercel AI SDK solve real problems, and I would reach for them when their broader coverage or integrations are what a project needs. I am not trying to replace any of them.\n\nFor my case, a wider framework or gateway meant learning more surface than the job required. I wanted a service I could follow from request to response and deploy next to my apps.\n\n**When I would not reach for Go-Ai:**\n\nAn application backend sends an OpenAI-compatible chat request to Go-Ai, using a local alias rather than a provider model slug. The gateway resolves that alias and proxies the request upstream; the application keeps its user session, validation, rate limits, database access, and response shape.\n\nThat modest boundary gives provider credentials one server-side home and lets an upstream model change without a sweep through every application route. The detailed mechanics -- runtime alias selection, defined fallback classes, the route and SSE contract, audio proxying, and validation -- are in the companion article: [Building a Small Go LLM Gateway with Explicit Boundaries](https://jakuninoleg.hashnode.dev/building-a-small-go-llm-gateway-with-explicit-boundaries).\n\nApplication wiring constructs three provider clients: Gemini and OpenRouter for the chat router, plus Groq for the separate audio service. I like that the wiring is ordinary and visible in [`cmd/api/main.go`](https://github.com/JakuninOleg/Go-Ai/blob/main/cmd/api/main.go): configuration enters in one place, providers are constructed explicitly, and the chat and audio services have distinct responsibilities.\n\nVoice stayed on the same principle. Speech-to-text and text-to-speech are separate proxy routes, not a new kind of chat request. The application decides whether an answer should be spoken; the gateway does not turn chat into realtime voice or an automatic speaking feature.\n\nTool calling was where the boundary mattered most to me. Go-Ai can pass OpenAI-compatible tool-related payloads through, but it does not execute tools. The calling application knows who the user is, what they are allowed to do, which tenant or record they can access, and which side effects are acceptable. The gateway does not.\n\n*The gateway carries provider concerns; the application keeps the context required to make an authorization or business decision. Tool execution stays where that context exists.*\n\nIn practice, an application receives a tool call, validates the user and arguments, performs the work against its trusted systems, and sends the follow-up message. It also keeps the complete tool-call message intact between turns, including any opaque provider metadata, instead of trying to rebuild it from a smaller shape. Provider credentials and the bearer secret stay in trusted server-side configuration; a browser talks to its application backend, not directly to the gateway.\n\nThe [Go-Ai repository](https://github.com/JakuninOleg/Go-Ai) is the project itself, not a hosted product. You can run it beside an existing backend and decide whether its small boundary is useful.\n\nDocker Compose is one supported option. Copy the example env, set your own secret and at least the Gemini key, then start it. OpenRouter stays optional unless you want its configured fallback path. Do not put real values in version control.\n\n```\ncp .env.example .env\n# Edit .env: set GO_AI_SHARED_SECRET and GEMINI_API_KEY.\ndocker compose up -d --build\n```\n\nThat starts the repository's Docker setup; it does not validate credentials or promise that an upstream provider is reachable from every host. A simple request from trusted server-side code can look like this:\n\n```\nexport GO_AI_BASE_URL=http://localhost:8080\nexport GO_AI_SHARED_SECRET=replace-with-your-gateway-secret\n\ncurl \"$GO_AI_BASE_URL/v1/chat/completions\" \\\n  -H \"Authorization: Bearer $GO_AI_SHARED_SECRET\" \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"default\",\n    \"messages\": [\n      { \"role\": \"user\", \"content\": \"Say hello in one sentence.\" }\n    ]\n  }'\n```\n\nIf you want the same boundary inside an existing frontend-backed app, start with the [agent handoff](https://github.com/JakuninOleg/Go-Ai/blob/main/docs/agent-prompts/start-here.md): copy the guides into the target repo, keep `GO_AI_BASE_URL` and `GO_AI_SHARED_SECRET` server-side only, and ask the agent to propose the smallest migration before editing files. The browser must never receive the shared secret.\n\nThe omissions are as important as the code that exists:\n\nThis is scope discipline, not a claim that those things are unimportant. If a future application needs full orchestration, a rich client SDK, billing, or broad provider management, I should choose a tool built for that job instead of stretching this one past recognition.\n\nThe most useful result has been removing a repeat decision from the next project: one service boundary and one provider-side home for credentials instead of provider setup in every codebase.\n\nBuilding it also gave my Go learning a practical shape: where HTTP errors belong, how to preserve a stream, which state should be local, and where not to hide responsibility.\n\nMy next focus is not to pile features onto Go-Ai. It is to keep the boundary reliable, document integrations clearly, and learn from the cases where the small shape stops being the right one. If you build or try something similar, I would value specific feedback: open an issue in the [Go-Ai repository](https://github.com/JakuninOleg/Go-Ai), or contribute a fix, test, or documentation improvement when you see a concrete gap. The most useful contributions will help keep the boundary honest rather than merely making it larger.", "url": "https://wpnews.pro/news/i-kept-rebuilding-llm-provider-setup-so-i-built-a-small-go-gateway", "canonical_source": "https://dev.to/jakuninoleg/i-kept-rebuilding-llm-provider-setup-so-i-built-a-small-go-gateway-5e06", "published_at": "2026-09-15 19:00:37+00:00", "updated_at": "2026-09-15 19:19:22.290630+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "ai-tools", "developer-tools", "ai-products"], "entities": ["Go-Ai", "Go", "LiteLLM", "LangChain", "Vercel AI SDK", "Gemini", "OpenRouter", "Groq"], "alternates": {"html": "https://wpnews.pro/news/i-kept-rebuilding-llm-provider-setup-so-i-built-a-small-go-gateway", "markdown": "https://wpnews.pro/news/i-kept-rebuilding-llm-provider-setup-so-i-built-a-small-go-gateway.md", "text": "https://wpnews.pro/news/i-kept-rebuilding-llm-provider-setup-so-i-built-a-small-go-gateway.txt", "jsonld": "https://wpnews.pro/news/i-kept-rebuilding-llm-provider-setup-so-i-built-a-small-go-gateway.jsonld"}}