A practical, first-principles guide to the problems Kubernetes solves — and why Docker alone is not enough
Part 1 of the Kubernetes for MLOps series
Kubernetes exists because running one container is easy, but operating many containers across many machines is not.
The central idea is simple: you declare the state you want, and Kubernetes continuously works to make the real system match it.
What you will understand after this chapter:Why the industry converged on container orchestration, and what problem Kubernetes actually solves — from first principles, not marketing copy.
You are the sole ML engineer at a fintech startup. The payments team has trained an XGBoost model that detects fraudulent transactions with 94% precision. The model needs to run as a real-time inference service: every card swipe calls your API within 200ms and gets a fraud probability score. If the score exceeds a threshold, the transaction is blocked.
The model works. Now the infrastructure becomes your problem.
This chapter traces exactly how that problem evolves — from a Python script to a Kubernetes deployment — and at every step explains why the current approach broke down and what each new layer actually solved.
You start the only way an engineer should: the simplest thing that works.
You run it:
uvicorn fraud_detector:app --host 0.0.0.0 --port 8000 --workers 4
It works. The payments team integrates it. Transactions flow. Life is good for about six weeks.
Single point of failure. Your process is the only instance. When it crashes — due to a memory leak, an unexpected exception, a malformed input — every downstream payment attempt fails. At 3am on a Saturday.
No isolation. The fraud detector shares the OS, filesystem, CPU, and memory with every other process on that machine. A misconfigured apt upgrade can break your Python runtime. A different service leaking memory OOM-kills your process. You have no guarantees.
Manual deployments. Retraining the model means SSH-ing to the production server, copying a new fraud_model.json, and restarting uvicorn. Every deployment is a manual SSH session. Mistakes happen. There is no rollback.
No horizontal scaling. Transaction volume grows 5x after a marketing campaign. You cannot add capacity without significant manual intervention. The single instance becomes a latency bottleneck.
No resource limits. A bug in the feature extraction code causes a tight loop. Your process consumes 100% CPU. Other services on the same host degrade.
The first instinct is correct: isolate services. Virtual machines provide hard boundaries between workloads. The isolation story is real. A crash in VM 1 does not affect VM 2. The hypervisor enforces CPU and memory boundaries. You can snapshot, restore, and clone VMs. You have an audit trail.
Resource waste at scale. A Ubuntu 22.04 minimal install consumes roughly 2GB of RAM just to exist. Your XGBoost model with a FastAPI wrapper needs about 400MB of RAM to serve traffic. The VM tax means you are paying for 2GB of RAM per instance just to run a 400MB application. Across a fleet of 50 fraud-detection VMs, that is 100GB of RAM doing nothing but running OS daemons.
Boot time. A VM takes 30–90 seconds to boot. When traffic spikes suddenly — a flash sale, a bot attack, a news event — you cannot add capacity fast enough. By the time a new VM is healthy, the spike has passed.
Environment drift. Two VMs provisioned from the same Machine imagesix months apart will differ. Security patches, library updates, and manual configuration changes accumulate. You have experienced “it works on VM 2 but not VM 3” at the worst possible time.
Slow iteration. To deploy a new model version, you build a new Machine image(10–15 minutes), launch a new instance (2–3 minutes), wait for health checks (1–2 minutes), shift traffic. A deployment takes 30 minutes minimum. Rolling back is not faster.
The dependency conflict problem. The fraud detection service needs XGBoost 2.0. A new anomaly detection service needs XGBoost 1.7 because a legacy dependency pins it. On VMs, both services share the system Python. You either containerize the environments manually (virtualenv, conda) or run each service on its own VM — amplifying the waste problem.
Virtual machines solved isolation. They created a new category of problems around density, speed, and reproducibility.
Docker did not invent containers. Linux already provided the core technologies, especially namespaces and control groups (cgroups).
Docker’s main contribution was making containers easy to build, distribute, and run consistently across different environments.
Linux namespaces give a process its own view of system resources.
The container can also have its own hostname, filesystem, and network interface. However, it still shares the host’s Linux kernel.
Namespaces provide isolation, while cgroups control resource usage. With Docker, you can restrict how much CPU and memory a container can consume:
docker run \ --memory="512m" \ --cpus="1.0" \ fraud-detector:v1.2.0
This container can use up to:
If it exceeds its memory limit, the kernel can terminate the container’s process without directly stopping the other containers on the host.
A container image is built from read-only filesystem layers:
Layer 4: Application codeLayer 3: Python dependenciesLayer 2: System librariesLayer 1: Base image
For the fraud detection service, the layers might contain ( for example ):
Application: fraud_detector.py and fraud_model.jsonPython packages: XGBoost, FastAPI and NumPySystem libraries: libgomp and libstdc++Base image: python:3.11-slim
When only the application code changes, Docker can reuse the previous layers. This makes builds faster and ensures that the same application environment can run in development, testing, and production.
Docker therefore solved three important problems:
Docker solved packaging, reproducibility, and isolation for our fraud detector. But operating one container is very different from managing many containers across several machines.
Imagine that ten fraud-detection containers run on three virtual machines. One machine crashes, another still runs model v1.2.0, and the third already runs v1.3.0.
New questions appear:
Docker runs containers, but it does not coordinate an entire fleet. This is the container orchestration problem.
Docker Compose can define multiple services, health checks, volumes, and restart policies in one file. It is excellent for local development and some single-host deployments.
However, it does not provide fleet-wide scheduling. If a machine fails, Compose cannot automatically move its containers to another host. For that, we need an orchestrator.
Kubernetes manages containerized applications across multiple machines. It provides:
Its central idea is the reconciliation loop. Kubernetes continuously compares the state we requested with what is actually running.
If we request three replicas of the fraud detector but only two are running, Kubernetes creates a replacement.
Instead of manually starting and replacing containers, we declare the result we want:
apiVersion: apps/v1kind: Deploymentmetadata: name: fraud-detectorspec: replicas: 3 selector: matchLabels: app: fraud-detector template: metadata: labels: app: fraud-detector spec: containers: - name: fraud-detector image: fraud-detector:v1.3.0 ports: - containerPort: 8000 readinessProbe: httpGet: path: /health port: 8000
Apply it with:
kubectl apply -f fraud-detector-deployment.yaml
Kubernetes schedules the Pods, monitors them, and replaces failed instances. The readiness probe prevents traffic from reaching a Pod before it is ready.
Docker and Kubernetes are not competitors. Docker gives us a consistent container image; Kubernetes keeps the distributed application running.
In the next part, we will examine the Kubernetes control plane, worker nodes, API server, scheduler, controller manager, and kubelet.
Why Kubernetes Exists: From a Python Script to Production Orchestration was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.