Over two weeks I moved six self-hosted services from Docker Compose onto Kubernetes — ArgoCD, supply-chain policy, the whole GitOps loop. Then, in a single day, I moved all six back, and took a seventh with them: a site that had been born on the cluster. Almost nobody writes this part. Not the triumphant migration, but the reversal, and the reasoning that justified both halves. Nothing failed technically. What broke was an assumption I'd been carrying about what Kubernetes is for.
The upward half is the half the internet is full of, so I'll compress it. A single 4-core, 24 GB machine runs a pile of compose stacks behind one reverse proxy. I installed K3s on that same box — not new hardware, but alongside the compose services, on the same kernel — and moved the services over in phases. ArgoCD with an app-of-apps root. A per-service migration template so every service followed the same path. Kyverno admission policy, Trivy image scanning, SealedSecrets for credentials. All the machinery that makes a cluster something you'd let near production.
One rule kept the migration honest: external domains and ports never change, and the reverse proxy stays the anchor. Move a service in, get it working on k3s, then repoint the proxy rule. The public surface never moved while the backend swapped underneath, and every service migrated as its own reversible step.
It worked. The tools did their jobs. ArgoCD reported Synced and Healthy for every Application. And then I undid it.
Everything went back to compose in one day: homepage, trilium, dify, vikunja, apprise, the LLM stack. The one k8s-native site with no compose predecessor went into compose too. What stayed on k3s was a short list: lab-environment, headlamp, and a namespace called pr-lanes that runs a two-tier practice app named hello.
I call it a reversal rather than a rollback, because a rollback means retreating from a change that failed. Nothing failed. Everything else the project built was kept — the cluster foundation, the GitOps loop, the supply-chain policy, the service mesh — and most of it is still running. Exactly one line of the plan was reversed: move the existing compose services into k8s.
The decision came down to one observation, and once I saw it, it explained everything else.
Everything I had migrated was software I didn't write.
I mean that literally: open-source images pulled from upstream registries. dify, vikunja, trilium, homepage, apprise, the model runners — none of it was code I wrote, iterated on, or shipped. That one fact knocked each of them out of every reason you'd run Kubernetes at all. There were three reasons, and they got worse in order.
The first is integration cost. A third-party image has no interest in your network policy, your CI pipeline, your internal DNS, or your admission rules. Whoever built it never had your enforcement layer in mind. So you either leave it half-integrated, or you do integration work whose entire output is making someone else's sealed box legible to your platform. That's real labor either way, and nobody ever uses what it produces. A stock open-source image has no need to be part of your cluster.
The second is worse, because it's where the value proposition evaporates. Canary releases, service meshes, mTLS between services, gradual rollouts — every one of them exists to give an owner fine-grained control over their own delivery. They're tools for a producer. A third-party image isn't something I release; it's something I consume. Someone else cuts the version tag, and my only decision is whether to take it. There's no rollout of mine to shape, no canary of mine to hold back, no service-to-service traffic of mine to secure inside an app whose internals upstream has already fixed. The mesh and the canary were built for a producer, and I showed up as a consumer.
The third is where it stopped being theory. Supply-chain policy was the part of the build I'd been proudest of: Trivy scans every image, and an admission policy refuses any pod carrying a critical CVE that has a fix available. For software I build, that loop closes itself — bump the base image, rebuild, redeploy. For software someone else builds, "fix the CVE" means maintaining a patched copy of their image.
I did that, once. A DoS bug in the tar library bundled inside npm itself turned up in homepage and trilium, and no upstream release of either app fixed it. So I built wrapper images: take the upstream image, replace the bundled npm, rebuild through a signed pipeline, scan clean. It worked. It also cost a new CI workflow, a signing setup, and a major-version upgrade of homepage, because one more copy of the vulnerable library was an app dependency no wrapper could reach. That was one CVE, on two images.
Then I went to flip the scan policy from audit to enforce, and found that ArgoCD, dify, and the lab's own components were all carrying fixable criticals nobody had noticed, because none of those pods had restarted since the reports landed. Enforcing it cluster-wide could have locked ArgoCD out of its own cluster the next time its pods were recreated. The only safe setting was to scope the gate to the images I build myself. A policy engine had reached the same conclusion I was about to: the gate only makes sense for software I author.
The day after the services went back to compose, I deleted the wrappers. Nothing enforced a clean scan on those two containers anymore, and the CVE they'd patched — DoS-only, no known exploit — was accepted as low-severity. The security policy hadn't failed. It was built for a producer, and I kept pointing it at things I consume.
None of this cost was hypothetical. It's sitting in my incident log from those weeks, and reading the entries together is what made the pattern visible. Each one looked like an unrelated one-off: memory overcommitted until the kernel OOM-killed containers while their pods kept reporting Running; a Trivy operator whose scan-concurrency setting sat in the wrong Helm values layer, was ignored, and hammered the disk; a Cilium flag, narrowed so the service mesh would work, that quietly cut the reverse proxy off from every k3s service behind it; and a packet from a pod to a compose container vanishing into a policy-routing blackhole, which I ended up designing around rather than fixing.
Four incidents, four root causes, one pattern underneath. Kubernetes isn't free on a single node that already runs Docker. It adds a second scheduler, a second network, a second security layer, a second source of truth, and then those layers have to agree with the ones already there, on the same kernel, on the same box. Compose fails too; one of my worst outages was a compose-side proxy refusing to start. But it fails inside one layer you can read top to bottom. On a shared box, Kubernetes fails across layers, and the failures arrive as "everything is green and it's still wrong."
For services whose code you're actively changing, that tax buys something real. For third-party images on a 4-core home box, it bought nothing, and I was paying it anyway.
The list of what stayed on k3s made the argument for me, and I didn't arrange it. Headlamp is someone else's software, but it's a Kubernetes dashboard: it exists to look at the cluster and has nowhere else to live. lab-environment was built for the cluster from day one and never had a compose version. That leaves pr-lanes, the two-tier hello app, and here the line between authored and consumed gets tighter, because only half of that pair is really mine. hello-backend is software I ship through my own CI. hello-frontend reuses a signed image inherited from an earlier, retired practice app. Its build context is gone, and the digest running today can't be rebuilt from the repo at all.
That isn't a counterexample. It's the reversal in miniature. The half I author lives on k3s and actually uses the lanes, the mesh, the canary, and the scan gate; the half that's a frozen, inherited artifact is exactly what doesn't belong there, and one day it will have to be replaced with something buildable. Even scaled down to one practice app, the lab draws the same line the migrations did.
Not one consumed production service stayed. On that box, k3s was never compose's successor. It's the lab: a place to practice the platform on the only kind of software the platform does anything for. Everything where I'm the operator and someone else is the author belongs on compose, where "run it and leave it alone" is the whole feature set, and that's the feature those services need.
Push the reasoning one level further and it stops being about Kubernetes or Compose at all. When I want some capability, there are three ways to get it, and each matches a kind of software.
Buy it as SaaS, when the data doesn't have to be mine and I want no operational burden. Self-host it on compose, when the data has to be mine but the software doesn't. Write it and run it on Kubernetes, when the software itself is mine and I'm going to keep changing it.
The mistake I had to undo was a fourth option, worse than all three: adopting the platform built for producing software to run software I consume. You pay the full cost of the platform for software that can never use the reason the platform exists. Everything Kubernetes offers is a way to shape software you own. If you've already decided not to shape it — and "I run someone else's image as-is" is exactly that decision — you gave all of it up before you started, and what's left is the operational cost.
The dividing line isn't important software versus unimportant software. It isn't even how often the software changes: dify ships new releases all the time. It's who owns the change. Software I author changes on my schedule, in ways I shape, so it earns infrastructure built for shaping change: canaries, lanes, gates. Software I consume changes on upstream's schedule, arriving as a new tag I either take or don't, and the right place for that is the boring one where taking it is docker compose pull && docker compose up -d. dify matters to me a great deal. None of its changes are mine. So it runs in compose.
That framework is what I actually walked away with. Moving up taught me what the platform does. Moving back taught me who it's for. The first half is all over the internet. This was the second half.