Foreman 101: agentic coding as Kubernetes resources

Foreman is an agentic coding system that runs as Kubernetes resources, treating model claims of success as untrusted requests rather than results. The system uses Workloads, agents, and verifiers to ensure only verified code lands, with a Helm chart installable on any cluster.

Foreman is an agentic coder that runs as Kubernetes resources. You describe work as a Workload, it decomposes into tasks, agents running on your nodes pick them up, and a branch comes out the other end with something deterministic standing between that branch and your main. This is the walkthrough. Four objects to understand, an install, an agent, a verifier, and a real run. Every command and every output below is from a working cluster. Foreman is deliberately small. Almost everything you do is one of these. coder and verifier .The shape of a run is: you apply a Workload, the controller synthesizes AgenticTasks, the scheduler routes each to a FleetNode whose agent can serve it, the agent runs the model in a loop with tools, and the result lands as a branch plus a verdict. Worth stating plainly, because it shapes every design decision: the model is not trusted, and specifically its claim to have succeeded is not trusted. A coder agent finishes by calling a tool that says "I am done, verdict GO." Foreman treats that as a request, not a result. If the model says GO and produced no diff, the run is recorded as NO-GO. If the verifier's checks do not pass, the work does not land, no matter how confident the summary was. That is the difference between an agent that writes code and a system you can leave running. Everything else in this post is plumbing around that idea. Foreman ships as a Helm chart that depends on LLMKube core. Any cluster will do; a throwaway kind cluster is fine, and no GPU is required to follow along. kind create cluster --name foreman-101 helm repo add llmkube https://defilantech.github.io/LLMKube helm repo update helm upgrade --install llmkube llmkube/llmkube \ --namespace llmkube-system --create-namespace helm upgrade --install foreman llmkube/foreman \ --namespace foreman-system --create-namespace \ --set agent.mode=native \ --set 'agent.roles={worker,coder,verifier}' Two flags on that second install are worth understanding rather than copying. agent.mode=native runs the real agent loop. The chart's default is stub , which registers a FleetNode and does nothing, so that a chart install can be smoke-tested on a cluster with no model and no credentials. agent.roles is what the scheduler matches against. A node advertising only worker will never be given coder work. Advertise the roles this node should actually serve. You should end up with three pods and a registered node: bash $ kubectl get pods -n llmkube-system NAME READY STATUS RESTARTS AGE llmkube-controller-manager-777d9f756b-xb7xr 1/1 Running 0 56s $ kubectl get pods -n foreman-system NAME READY STATUS RESTARTS AGE foreman-agent-568b74c49f-mnkc8 1/1 Running 0 27s foreman-operator-7c687499f9-k96mb 1/1 Running 0 27s $ kubectl get fleetnodes NAME PHASE VERSION HEARTBEAT AGE foreman-agent-568b74c49f-mnkc8 Ready 0.9.12 20s 20s Foreman needs two kinds of credential, and they do different jobs. A git credential lets the agent read the issue and push the branch. A model credential lets it talk to a hosted API, and is not needed at all if you serve the model yourself. git: the API token, plus the credential used for the push kubectl create secret generic foreman-github \ --from-literal=GITHUB TOKEN="$GITHUB TOKEN" -n foreman-system kubectl create secret generic foreman-git-credentials \ --from-literal=token="$GITHUB TOKEN" -n foreman-system helm upgrade foreman llmkube/foreman -n foreman-system --reuse-values \ --set agent.githubToken.secretName=foreman-github \ --set coder.gitCredentialsSecret=foreman-git-credentials \ --set agent.gitRemoteURL=https://github.com/<you /<repo model: read from the AGENT's namespace, not the operator's kubectl create secret generic anthropic-key \ --from-literal=apiKey="$ANTHROPIC API KEY" -n default agent.gitRemoteURL is where branches are pushed. Pointing it at your fork keeps agent output off the upstream repository, which is the arrangement you want while you are still learning what it does. Note the namespaces differ, and it is not arbitrary. The git secrets are consumed by the agent process, so they live alongside it in foreman-system . The model credential is resolved per Agent, from that Agent's namespace, so it belongs wherever you create your Agents. Here that is default . An Agent needs a model. The cloud-proxy provider speaks to any OpenAI-compatible endpoint, so this path works with a hosted API and no GPU. apiVersion: foreman.llmkube.dev/v1alpha1 kind: Agent metadata: name: claude-coder namespace: default spec: role: coder provider: cloud-proxy providerConfig: /chat/completions is appended, so give the /v1 prefix. baseURL: https://api.anthropic.com/v1 model: claude-sonnet-5 apiKeySecretRef: name: anthropic-key key: apiKey tools: - read file - write file - str replace - grep - bash - submit result maxTurns: 12 contextWindowTokens: 30000 maxOutputTokens: 4096 requiredCapability: roles: coder systemPrompt: | You are a senior staff software engineer. You fix one issue per run. The repository is already cloned in your workspace, checked out on a branch created for you. You communicate only by calling tools. Every action is a tool call. Work in this order: 1. read file / grep to find the change site. Do not survey the whole repository; find the file and start. 2. str replace or write file to make the change. Make your first edit early, even if incomplete, then refine it. 3. bash to verify. Run the narrowest check that proves your change. 4. submit result to finish. submit result is terminal. Call it with verdict=GO and a commit message when the change is complete and verified, or verdict=INCOMPLETE if you cannot finish, saying plainly what is missing. Make the minimum change that fixes the issue. Do not refactor adjacent code, and do not fix unrelated problems you notice on the way. Three fields there deserve more than a glance. The budgets are a cost control. Defaults are 120 turns and a 90k context, and every turn re-sends the whole context to a metered API. Twelve turns and 30k is the difference between a run that costs pennies and one that does not. The tool list is the agent's authority. It cannot do anything not on that list. Removing bash gives you an agent that can edit but not execute; removing the write tools gives you one that can only read and report. The system prompt is the behaviour. The ordering instructions above are not decoration. Left to explore, a coding model will survey a repository for a long time before touching it, so "find the file and start" and "make your first edit early" are load bearing. So is the scope fence at the end, which is what keeps a one-line fix from arriving as a refactor. This is the object that makes Foreman different, and it is the smallest one. apiVersion: foreman.llmkube.dev/v1alpha1 kind: Agent metadata: name: gate namespace: default spec: role: verifier provider: local tools: - run gate job requiredCapability: accelerator: any roles: verifier There is no model here. No provider config, no API key, no system prompt. An Agent with an empty inferenceServiceRef is a deterministic agent : Foreman skips the model loop entirely and just runs the tool. run gate job submits a Kubernetes Job that clones the branch and runs your checks. In a Go repository that is your build, vet, linter and tests, plus a bite check that reverts the production change and re-runs the new tests to confirm they fail without it. A test that still passes with the fix reverted is not a test. The component that decides whether the AI's work is acceptable is not itself AI. That is the whole point. A Workload is what you author day to day. The only strictly required field is intent . apiVersion: foreman.llmkube.dev/v1alpha1 kind: Workload metadata: name: abtest-s1 namespace: default spec: repo: defilantech/LLMKube issues: 1311 coderAgentRef: name: claude-coder verifierAgentRef: name: gate openPullRequest: false intent: | Fix 1311: HPA-driven scale-up is not gated by GPUQuota, and status.usedGPUCount under-reports because it counts the admitted replica figure, not the autoscaling maxReplicas the service can reach. Charge autoscaling.maxReplicas not spec.replicas toward GPUQuota usage so the cap reflects the worst case, and reject admission when maxReplicas would exceed the quota. Add a regression test. The intent is a brief to a competent engineer who has not seen your codebase. Being exact about the change, charge maxReplicas and not spec.replicas , and naming the deliverable a regression test does more for the outcome than any model setting. Where there is one obvious file to start in, name it; here the fix spans two accounting paths, so the brief names both. Apply it and watch: bash $ kubectl get workload,agentictask NAME PHASE REPO AGE workload.foreman.llmkube.dev/abtest-s1 Dispatched defilantech/LLMKube 8s NAME PHASE KIND VERDICT agentictask.foreman.llmkube.dev/abtest-s1-code-1311 Running issue-fix The Workload synthesized a task named abtest-s1-code-1311 : the Workload name, then the step, then the issue. With a verifier configured you also get a abtest-s1-verify-1311 that waits on the coder finishing. A finished task carries its verdict and the evidence for it. verdict: GO summary: "Charge autoscaling.maxReplicas not spec.replicas toward GPUQuota usage in both the admission webhook and the GPUQuota status reconciler, so HPA scale-up is gated at admission and status.usedGPUCount reflects the worst-case headroom. Added regression tests." turnCount: 64 elapsedSec: 1608.38 branch: foreman/abtest-s1/issue-1311 commitSHA: 87a0d70c804a6fc4a51fba6550083e976d0d2c06 Sixty-four turns and about twenty-seven minutes: a real change across two accounting paths with a regression test, not a one-line edit, and the branch is on the fork. The full transcript is persisted as a ConfigMap, referenced from the task status, so you can read every tool call the model made rather than guessing at it. The verdicts you will actually see, and what each means: GO the coder finished and produced a diff. NO-GO it did not, or it claimed success without one. INCOMPLETE it ran out of turns or could not verify its own work. GATE-PASS the verifier's checks passed on the branch. GATE-FAIL they did not, and the branch does not land.A verifier runs whatever you tell it to. You can point one at an existing branch to see the mechanism on its own. apiVersion: foreman.llmkube.dev/v1alpha1 kind: AgenticTask metadata: name: gate-check namespace: default spec: kind: verify agentRef: name: gate gateProfile: "generic" runs the commands below instead of the Go make targets. language: generic image: golang:1.26 commands: Swap this line to see the other verdict. format: "grep -q Foreman docs/site/foreman/README.md" build: "true" lint: "true" test: "true" payload: repo: defilantech/LLMKube branch: main Run it once as written, then once with a string that is not in the file, and the Job logs show exactly what happened: === clone defilantech/LLMKube @ main === === grep -q Foreman docs/site/foreman/README.md === GATE PASS === clone defilantech/LLMKube @ main === === grep -q NO SUCH STRING IN THIS REPO docs/site/foreman/README.md === GATE FAIL Same agent, same branch, one command different, opposite verdicts. The task status carries deterministic: true , because no model was consulted about whether the work was good. A grep decided. Everything above used a hosted API because it is the version anyone can try today. The reason LLMKube exists is the other path: the model runs on hardware you own and nothing leaves the building. The Agent barely changes. No API key and no external URL, just a reference to an InferenceService that LLMKube is already serving. spec: role: coder provider: local inferenceServiceRef: name: laguna-s-2.1-strix model: laguna-s-2.1-q4km-strix tools, budgets and systemPrompt are identical to the cloud-proxy Agent Foreman resolves that to the service's in-cluster address at task time. The agent that runs the loop must be able to reach it, which means an in-cluster agent for an in-cluster service. Everything downstream, the tools, the verdicts, the gate, is identical. The model behind that reference in our own cluster is a 118B-parameter, roughly 8B-active open-weight coder on an AMD Strix Halo box with 128GB of unified memory. Four of its pull requests merged into LLMKube while this post was being written, each through the full gate. Being straight about scope, because it determines whether this is useful to you. Mechanical, well-scoped work lands: a documented bug with a known change site, a refactor with a clear contract, a missing test, a docs correction. Give it the file and a scope fence and it will do the job and stop. Architectural work does not, yet. A change that requires holding two packages in your head and deciding which of two components should own a behaviour is still yours. When a run fails on that kind of task it usually fails honestly, as INCOMPLETE with a summary, rather than landing something plausible and wrong. That is the failure mode you want, but it is still a failure. A kind cluster, the two Helm charts, one coder Agent against whatever API key you already have, one verifier, and a genuinely small issue. Twenty minutes end to end. Then add the verifier before you add anything else. An agent without one is autocomplete with a Kubernetes API; an agent with one is a system you can leave running. The docs are at llmkube.com/docs/foreman https://llmkube.com/docs/foreman and the code is on GitHub https://github.com/defilantech/LLMKube . Both are Apache 2.0.