Foreman is an agentic coder that runs as Kubernetes resources. You describe work as a Workload, it decomposes into tasks, agents running on your nodes pick them up, and a branch comes out the other end with something deterministic standing between that branch and your main.
This is the walkthrough. Four objects to understand, an install, an agent, a verifier, and a real run. Every command and every output below is from a working cluster.
Foreman is deliberately small. Almost everything you do is one of these.
coder
and verifier
.The shape of a run is: you apply a Workload, the controller synthesizes AgenticTasks, the scheduler routes each to a FleetNode whose agent can serve it, the agent runs the model in a loop with tools, and the result lands as a branch plus a verdict.
Worth stating plainly, because it shapes every design decision: the model is not trusted, and specifically its claim to have succeeded is not trusted.
A coder agent finishes by calling a tool that says "I am done, verdict GO." Foreman treats that as a request, not a result. If the model says GO and produced no diff, the run is recorded as NO-GO. If the verifier's checks do not pass, the work does not land, no matter how confident the summary was.
That is the difference between an agent that writes code and a system you can leave running. Everything else in this post is plumbing around that idea.
Foreman ships as a Helm chart that depends on LLMKube core. Any cluster will do; a throwaway kind cluster is fine, and no GPU is required to follow along.
kind create cluster --name foreman-101
helm repo add llmkube https://defilantech.github.io/LLMKube
helm repo update
helm upgrade --install llmkube llmkube/llmkube \
--namespace llmkube-system --create-namespace
helm upgrade --install foreman llmkube/foreman \
--namespace foreman-system --create-namespace \
--set agent.mode=native \
--set 'agent.roles={worker,coder,verifier}'
Two flags on that second install are worth understanding rather than copying.
agent.mode=native
runs the real agent loop. The chart's default is stub
, which registers a FleetNode and does nothing, so that a chart install can be smoke-tested on a cluster with no model and no credentials.
agent.roles
is what the scheduler matches against. A node advertising only worker
will never be given coder work. Advertise the roles this node should actually serve.
You should end up with three pods and a registered node:
$ kubectl get pods -n llmkube-system
NAME READY STATUS RESTARTS AGE
llmkube-controller-manager-777d9f756b-xb7xr 1/1 Running 0 56s
$ kubectl get pods -n foreman-system
NAME READY STATUS RESTARTS AGE
foreman-agent-568b74c49f-mnkc8 1/1 Running 0 27s
foreman-operator-7c687499f9-k96mb 1/1 Running 0 27s
$ kubectl get fleetnodes
NAME PHASE VERSION HEARTBEAT AGE
foreman-agent-568b74c49f-mnkc8 Ready 0.9.12 20s 20s
Foreman needs two kinds of credential, and they do different jobs.
A git credential lets the agent read the issue and push the branch. A model credential lets it talk to a hosted API, and is not needed at all if you serve the model yourself.
kubectl create secret generic foreman-github \
--from-literal=GITHUB_TOKEN="$GITHUB_TOKEN" -n foreman-system
kubectl create secret generic foreman-git-credentials \
--from-literal=token="$GITHUB_TOKEN" -n foreman-system
helm upgrade foreman llmkube/foreman -n foreman-system --reuse-values \
--set agent.githubToken.secretName=foreman-github \
--set coder.gitCredentialsSecret=foreman-git-credentials \
--set agent.gitRemoteURL=https://github.com/<you>/<repo>
kubectl create secret generic anthropic-key \
--from-literal=apiKey="$ANTHROPIC_API_KEY" -n default
agent.gitRemoteURL
is where branches are pushed. Pointing it at your fork keeps agent output off the upstream repository, which is the arrangement you want while you are still learning what it does.
Note the namespaces differ, and it is not arbitrary. The git secrets are consumed by the agent process, so they live alongside it in foreman-system
. The model credential is resolved per Agent, from that Agent's namespace, so it belongs wherever you create your Agents. Here that is default
.
An Agent needs a model. The cloud-proxy
provider speaks to any OpenAI-compatible endpoint, so this path works with a hosted API and no GPU.
apiVersion: foreman.llmkube.dev/v1alpha1
kind: Agent
metadata:
name: claude-coder
namespace: default
spec:
role: coder
provider: cloud-proxy
providerConfig:
baseURL: https://api.anthropic.com/v1
model: claude-sonnet-5
apiKeySecretRef:
name: anthropic-key
key: apiKey
tools:
- read_file
- write_file
- str_replace
- grep
- bash
- submit_result
maxTurns: 12
contextWindowTokens: 30000
maxOutputTokens: 4096
requiredCapability:
roles: [coder]
systemPrompt: |
You are a senior staff software engineer.
You fix one issue per run. The repository is already cloned in your
workspace, checked out on a branch created for you.
You communicate only by calling tools. Every action is a tool call.
Work in this order:
1. read_file / grep to find the change site. Do not survey the whole
repository; find the file and start.
2. str_replace or write_file to make the change. Make your first edit
early, even if incomplete, then refine it.
3. bash to verify. Run the narrowest check that proves your change.
4. submit_result to finish.
submit_result is terminal. Call it with verdict=GO and a commit_message
when the change is complete and verified, or verdict=INCOMPLETE if you
cannot finish, saying plainly what is missing.
Make the minimum change that fixes the issue. Do not refactor adjacent
code, and do not fix unrelated problems you notice on the way.
Three fields there deserve more than a glance.
The budgets are a cost control. Defaults are 120 turns and a 90k context, and every turn re-sends the whole context to a metered API. Twelve turns and 30k is the difference between a run that costs pennies and one that does not.
The tool list is the agent's authority. It cannot do anything not on that list. Removing bash
gives you an agent that can edit but not execute; removing the write tools gives you one that can only read and report.
The system prompt is the behaviour. The ordering instructions above are not decoration. Left to explore, a coding model will survey a repository for a long time before touching it, so "find the file and start" and "make your first edit early" are load bearing. So is the scope fence at the end, which is what keeps a one-line fix from arriving as a refactor.
This is the object that makes Foreman different, and it is the smallest one.
apiVersion: foreman.llmkube.dev/v1alpha1
kind: Agent
metadata:
name: gate
namespace: default
spec:
role: verifier
provider: local
tools:
- run_gate_job
requiredCapability:
accelerator: any
roles: [verifier]
There is no model here. No provider config, no API key, no system prompt. An Agent with an empty inferenceServiceRef
is a deterministic agent: Foreman skips the model loop entirely and just runs the tool.
run_gate_job
submits a Kubernetes Job that clones the branch and runs your checks. In a Go repository that is your build, vet, linter and tests, plus a bite check that reverts the production change and re-runs the new tests to confirm they fail without it. A test that still passes with the fix reverted is not a test.
The component that decides whether the AI's work is acceptable is not itself AI. That is the whole point.
A Workload is what you author day to day. The only strictly required field is intent
.
apiVersion: foreman.llmkube.dev/v1alpha1
kind: Workload
metadata:
name: abtest-s1
namespace: default
spec:
repo: defilantech/LLMKube
issues: [1311]
coderAgentRef:
name: claude-coder
verifierAgentRef:
name: gate
openPullRequest: false
intent: |
Fix #1311: HPA-driven scale-up is not gated by GPUQuota, and
status.usedGPUCount under-reports because it counts the admitted replica
figure, not the autoscaling maxReplicas the service can reach. Charge
autoscaling.maxReplicas (not spec.replicas) toward GPUQuota usage so the
cap reflects the worst case, and reject admission when maxReplicas would
exceed the quota. Add a regression test.
The intent is a brief to a competent engineer who has not seen your codebase. Being exact about the change, charge maxReplicas
and not spec.replicas
, and naming the deliverable (a regression test) does more for the outcome than any model setting. Where there is one obvious file to start in, name it; here the fix spans two accounting paths, so the brief names both.
Apply it and watch:
$ kubectl get workload,agentictask
NAME PHASE REPO AGE
workload.foreman.llmkube.dev/abtest-s1 Dispatched defilantech/LLMKube 8s
NAME PHASE KIND VERDICT
agentictask.foreman.llmkube.dev/abtest-s1-code-1311 Running issue-fix
The Workload synthesized a task named abtest-s1-code-1311
: the Workload name, then the step, then the issue. With a verifier configured you also get a abtest-s1-verify-1311
that waits on the coder finishing.
A finished task carries its verdict and the evidence for it.
verdict: GO
summary: "Charge autoscaling.maxReplicas (not spec.replicas) toward
GPUQuota usage in both the admission webhook and the GPUQuota
status reconciler, so HPA scale-up is gated at admission and
status.usedGPUCount reflects the worst-case headroom. Added
regression tests."
turnCount: 64
elapsedSec: 1608.38
branch: foreman/abtest-s1/issue-1311
commitSHA: 87a0d70c804a6fc4a51fba6550083e976d0d2c06
Sixty-four turns and about twenty-seven minutes: a real change across two accounting paths with a regression test, not a one-line edit, and the branch is on the fork. The full transcript is persisted as a ConfigMap, referenced from the task status, so you can read every tool call the model made rather than guessing at it.
The verdicts you will actually see, and what each means:
GO
the coder finished and produced a diff.NO-GO
it did not, or it claimed success without one.INCOMPLETE
it ran out of turns or could not verify its own work.GATE-PASS
the verifier's checks passed on the branch.GATE-FAIL
they did not, and the branch does not land.A verifier runs whatever you tell it to. You can point one at an existing branch to see the mechanism on its own.
apiVersion: foreman.llmkube.dev/v1alpha1
kind: AgenticTask
metadata:
name: gate-check
namespace: default
spec:
kind: verify
agentRef:
name: gate
gateProfile:
language: generic
image: golang:1.26
commands:
format: "grep -q Foreman docs/site/foreman/README.md"
build: "true"
lint: "true"
test: "true"
payload:
repo: defilantech/LLMKube
branch: main
Run it once as written, then once with a string that is not in the file, and the Job logs show exactly what happened:
=== clone defilantech/LLMKube @ main ===
=== grep -q Foreman docs/site/foreman/README.md ===
GATE PASS
=== clone defilantech/LLMKube @ main ===
=== grep -q NO_SUCH_STRING_IN_THIS_REPO docs/site/foreman/README.md ===
GATE FAIL
Same agent, same branch, one command different, opposite verdicts. The task status carries deterministic: true
, because no model was consulted about whether the work was good. A grep decided.
Everything above used a hosted API because it is the version anyone can try today. The reason LLMKube exists is the other path: the model runs on hardware you own and nothing leaves the building.
The Agent barely changes. No API key and no external URL, just a reference to an InferenceService that LLMKube is already serving.
spec:
role: coder
provider: local
inferenceServiceRef:
name: laguna-s-2.1-strix
model: laguna-s-2.1-q4km-strix
Foreman resolves that to the service's in-cluster address at task time. The agent that runs the loop must be able to reach it, which means an in-cluster agent for an in-cluster service. Everything downstream, the tools, the verdicts, the gate, is identical.
The model behind that reference in our own cluster is a 118B-parameter, roughly 8B-active open-weight coder on an AMD Strix Halo box with 128GB of unified memory. Four of its pull requests merged into LLMKube while this post was being written, each through the full gate.
Being straight about scope, because it determines whether this is useful to you.
Mechanical, well-scoped work lands: a documented bug with a known change site, a refactor with a clear contract, a missing test, a docs correction. Give it the file and a scope fence and it will do the job and stop.
Architectural work does not, yet. A change that requires holding two packages in your head and deciding which of two components should own a behaviour is still yours. When a run fails on that kind of task it usually fails honestly, as INCOMPLETE with a summary, rather than landing something plausible and wrong. That is the failure mode you want, but it is still a failure.
A kind cluster, the two Helm charts, one coder Agent against whatever API key you already have, one verifier, and a genuinely small issue. Twenty minutes end to end.
Then add the verifier before you add anything else. An agent without one is autocomplete with a Kubernetes API; an agent with one is a system you can leave running.
The docs are at llmkube.com/docs/foreman and the code is on GitHub. Both are Apache 2.0.