A leading reason AI workloads fail in containers is the boundary between the container's isolated namespace and the host's hardware. Running inference on raw CPUs creates big latency bottlenecks, and exposing a GPU or accelerator takes deliberate host setup.
A container cannot talk to a PCIe device without driver support. The NVIDIA Container Toolkit supplies it by wrapping the standard runtime with a hardware-aware one (nvidia-container-runtime).
Diagram: FIG 1 · The NVIDIA Container Toolkit injects the host's driver libraries and device nodes into the container at launch. Your image does not bundle the kernel driver. See the animated version.
Three ideas make it work.
Diagram: FIG 2 · Decoupling the driver from the image is what makes accelerated containers portable. Compatibility between the host driver and your image's CUDA version still matters. See the animated version.
nvidia.ko). The image never contains it. libcuda.so) and the device nodes (/dev/nvidia*) into the container.
With the toolkit installed, register it with Docker and restart the daemon.
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
That edits /etc/docker/daemon.json. You can also set it by hand, and making it the default runtime is optional, since --gpus selects it per container.
{
"default-runtime": "nvidia",
"runtimes": {
"nvidia": { "path": "nvidia-container-runtime", "runtimeArgs": [] }
}
}
docker run -d --name local-inference \
--gpus all \
--shm-size="2g" \
-p 8000:8000 \
enterprise-model-server:latest
Verify it: docker run --rm --gpus all ubuntu nvidia-smi should print the same GPU table as the host.
--shm-size matters
PyTorch passes tensors between worker processes through shared memory (/dev/shm). Docker's default is only 64 MB, so parallel data can crash suddenly with bus errors. Raising it, for example to 2g, avoids that.
Diagram: FIG 4 · PyTorch uses shared memory to pass tensors between worker processes. Docker's 64 MB default is too small for that, so raise it with --shm-size. See the animated version.
An unconstrained container is a danger to the host. A multi-step prompt can grow the KV-cache and demand several extra gigabytes of RAM at once, so enforce limits with Linux cgroups.
docker run -d --name constrained-model \
--cpus="4.0" \
--memory="10g" \
--memory-swap="10g" \
-p 11434:11434 \
ollama/ollama:latest
Diagram: FIG 3 · An unconstrained model container can take the whole machine down. Kernel cgroup limits keep the damage inside the container. The 10 GB cap is an example for a 16 GB host. See the animated version.
--memory="10g" caps the container at 10 GB and leaves about 6 GB for the host OS, networking daemons and file cache.--memory-swap="10g" (equal to --memory) disables swap for the container. Swapping model weights to disk slows inference by orders of magnitude, so it is better to fail fast.--oom-kill-disable=false), so you do not need to set it. Never use --oom-kill-disable=true without a memory limit, because then a spike can hang the whole machine.--gpus only when a GPU is present. On a CPU-only laptop, the memory and CPU caps are what protect you.
Treat these flags as part of your deployment definition, and in Kubernetes you will express the same limits as resource requests and limits.
This is the exact example from the video. These are real docker run commands that read the limits from inside the container. There is no GPU on my laptop, so only the limits half of the lesson is shown.
docker run --rm --cpus=2 --memory=256m --memory-swap=256m \
--shm-size=2g alpine:3.20 sh -c '
cd /sys/fs/cgroup
for f in memory.max memory.swap.max cpu.max; do
echo "$f: $(cat $f)"
done
echo shm: $(df -h /dev/shm | tail -1 | tr -s " " | cut -d" " -f2)'
docker run --rm --memory=64m --memory-swap=64m alpine:3.20 \
sh -c 'x=$(dd if=/dev/zero bs=1M count=200 | tr "\0" x)'
echo "over the cap: exit code $?"
Run it:
bash limits.sh
Output I got while recording the video:
memory.max: 268435456
memory.swap.max: 0
cpu.max: 200000 100000
shm: 2.0G
over the cap: exit code 137
Day 4: Optimising model weights: quantization basics with GGUF and AWQ.
Originally published at https://sureshpallapothu.in/blog/ai-int-day-3-gpu-passthrough, where this post includes animated diagrams.