There’s a specific kind of dread that comes with giving an AI model access to your infrastructure. Even a small, self-hosted one. The moment you let a language model near a terminal, the question stops being “will it help me” and becomes “what’s the worst thing it could accidentally do.”
So when I set out to build a DevOps assistant that could answer questions about a live Kubernetes cluster, that question sat at the center of every design decision. Not “how do I make this smart,” but “how do I make this safe even when it isn’t.”
You type a plain-English question into a dashboard: “what pods are currently running?” or “why might this pod be crashing?” or “is this node healthy?” The assistant decides whether it needs live cluster data to answer, calls one of a small set of read-only tools if so, reads the real result, and answers you grounded in what it actually found. If a question falls outside what its tools can check, it says so instead of guessing. That last part mattered more to me than any feature: a DevOps tool that confidently makes things up is worse than no tool at all.
A browser dashboard talks to a FastAPI /ask endpoint. That endpoint hands the question to Ollama, running Qwen3 locally, no external API calls, no data leaving the machine. When the model decides it needs cluster data, it doesn't run a command. It requests one of six fixed Python functions: get_pods, get_logs, describe_pod, get_deployments, get_services, get_nodes. Each of those wraps exactly one hardcoded kubectl command template. The result comes back, gets handed to the model, and the model turns it into a plain-English answer.
Here’s the distinction that took me a while to fully articulate: the model never generates a command, it only expresses intent. It can ask for get_logs and supply a pod name as a parameter, but it can't add a flag, chain a command, or invoke anything outside those six functions. The application decides how the intent gets executed. The model just picks from a menu.
That’s one layer. The second layer is enforced independently, at the infrastructure level, so a bug in my Python code can’t become a cluster-wide problem. The agent runs under a Kubernetes ServiceAccount bound to a ClusterRole that only grants get, list, and watch on pods, pod logs, deployments, services, and nodes. No write verbs anywhere. No access to secrets. I didn't just assume this worked, I verified it by attempting a write operation from inside the running pod and confirming Kubernetes rejected it with a permissions error. If the reasoning layer ever got fooled, the RBAC layer would still hold.
Getting this agent to actually reach both Ollama and the Kubernetes API from inside a real pod turned out to be the hardest part of the whole build, harder than anything involving the model itself.
The first wall: localhost inside a container refers to the container, not the host machine. Ollama running on my host was completely unreachable from inside any container using localhost. The usual workaround, host.docker.internal, works fine for a plain docker run container, but it doesn't exist inside real Kubernetes pods. There's no equivalent mechanism in a standard pod spec. I spent a while assuming I'd misconfigured something before realizing the mechanism itself just isn't there.
The fix that actually worked: find the real gateway IP of the Docker bridge network the kind cluster runs on, using ip a to look for the br-xxxxxxxx interface, and point OLLAMA_URL at that IP directly. It works because pods share that same underlying Docker network as the host.
A related problem showed up when trying to reach the Kubernetes API server specifically. A host-mapped port can be bound only to 127.0.0.1, making it invisible to other containers even on the same machine. The more reliable fix was attaching to the same Docker network as the kind control-plane container and addressing it directly by container name on its internal port, 6443.
And then there was the bug that had nothing to do with any of this. docker run -p port publishing intermittently failed specifically when combined with a custom --network flag, even though both the app and the network were healthy on their own. I worked around it by testing against the container's direct IP instead of the published port. Oddly, this never showed up again once the app moved to a real Kubernetes deployment, since Ingress and Services use an entirely different mechanism than Docker's host-port publishing.
Self-hosted, smaller LLMs are noticeably less reliable at tool-calling than large hosted ones, and this project made that concrete instead of theoretical. Sometimes the model described a tool call as plain-text JSON in its response instead of using its actual structured tool-calling mechanism, which needed a fallback parser to recover. Sometimes it correctly reasoned about which tool to call but never completed the call at all.
One failure looked exactly like model flakiness but wasn’t. get_nodes was registered as callable in the backend, and it was named in the system prompt, so the model would reason about wanting to use it, but the actual schema list sent to the model never included it. The model wasn't malfunctioning. It was reasoning correctly about a tool that had never actually been exposed to it. That distinction, a genuine application bug versus a genuine model limitation, turned out to be its own skill to build, separate from writing the agent itself. Switching the underlying model from llama3.2 to llama3.1 to qwen3 produced steadily more reliable tool-calling along the way, for what that's worth.
A GitOps setup with ArgoCD to remove the manual helm upgrade step, since deployment currently isn't automated in CI at all, GitHub's runners have no path to a local kind cluster. Additional tools like kubectl top for resource usage and broader event history. Eventually moving off a local kind cluster to a real cloud-managed one so CI/CD can deploy on its own. Conversation memory across multiple questions in a session. And a "runbook" mode: describe an incident, and have the assistant lay out the diagnostic steps it would actually take.
Full code, the Helm chart, and the CI pipeline are up on GitHub if you want to dig into any of it. If you’ve hit the host.docker.internal-doesn't-exist-in-pods wall yourself, I'd like to compare notes.
I Built an AI Agent That Can Query My Kubernetes Cluster, But Never Break It was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.