{"slug": "day-3-hardware-acceleration-in-docker-gpu-passthrough-and-limits-for-constrained", "title": "Day 3: Hardware Acceleration in Docker — GPU Passthrough and Limits for Constrained Hardware", "summary": "A developer documented how to run AI inference workloads in Docker with GPU passthrough via the NVIDIA Container Toolkit, registering nvidia-container-runtime with the Docker daemon and selecting GPUs per container with --gpus. The writeup stresses that the kernel driver stays on the host while driver libraries and /dev/nvidia* device nodes are injected at launch, and that Docker's 64 MB default /dev/shm must be raised (e.g. --shm-size=2g) to avoid PyTorch bus errors during parallel data loading. It also recommends cgroup caps such as --cpus, --memory and --memory-swap to keep KV-cache growth from taking down the host.", "body_md": "A leading reason AI workloads fail in containers is the boundary between the container's isolated namespace and the host's hardware. Running inference on raw CPUs creates big latency bottlenecks, and exposing a GPU or accelerator takes deliberate host setup.\n\nA container cannot talk to a PCIe device without driver support. The **NVIDIA Container Toolkit** supplies it by wrapping the standard runtime with a hardware-aware one (`nvidia-container-runtime`).\n\n**Diagram:** FIG 1 · The NVIDIA Container Toolkit injects the host's driver libraries and device nodes into the container at launch. Your image does not bundle the kernel driver. [See the animated version](https://sureshpallapothu.in/blog/ai-int-day-3-gpu-passthrough).\n\nThree ideas make it work.\n\n**Diagram:** FIG 2 · Decoupling the driver from the image is what makes accelerated containers portable. Compatibility between the host driver and your image's CUDA version still matters. [See the animated version](https://sureshpallapothu.in/blog/ai-int-day-3-gpu-passthrough).\n\n`nvidia.ko`). The image never contains it.` libcuda.so`) and the device nodes (`/dev/nvidia*`) into the container.\nWith the toolkit installed, register it with Docker and restart the daemon.\n\n```\nsudo nvidia-ctk runtime configure --runtime=docker\nsudo systemctl restart docker\n```\n\nThat edits `/etc/docker/daemon.json`. You can also set it by hand, and making it the default runtime is optional, since `--gpus` selects it per container.\n\n```\n{\n  \"default-runtime\": \"nvidia\",\n  \"runtimes\": {\n    \"nvidia\": { \"path\": \"nvidia-container-runtime\", \"runtimeArgs\": [] }\n  }\n}\ndocker run -d --name local-inference \\\n  --gpus all \\\n  --shm-size=\"2g\" \\\n  -p 8000:8000 \\\n  enterprise-model-server:latest\n```\n\nVerify it: `docker run --rm --gpus all ubuntu nvidia-smi` should print the same GPU table as the host.\n\n`--shm-size` matters\nPyTorch passes tensors between worker processes through shared memory (`/dev/shm`). Docker's default is only **64 MB**, so parallel data loading can crash suddenly with bus errors. Raising it, for example to `2g`, avoids that.\n\n**Diagram:** FIG 4 · PyTorch uses shared memory to pass tensors between worker processes. Docker's 64 MB default is too small for that, so raise it with --shm-size. [See the animated version](https://sureshpallapothu.in/blog/ai-int-day-3-gpu-passthrough).\n\nAn unconstrained container is a danger to the host. A multi-step prompt can grow the KV-cache and demand several extra gigabytes of RAM at once, so enforce limits with Linux **cgroups**.\n\n```\ndocker run -d --name constrained-model \\\n  --cpus=\"4.0\" \\\n  --memory=\"10g\" \\\n  --memory-swap=\"10g\" \\\n  -p 11434:11434 \\\n  ollama/ollama:latest\n```\n\n**Diagram:** FIG 3 · An unconstrained model container can take the whole machine down. Kernel cgroup limits keep the damage inside the container. The 10 GB cap is an example for a 16 GB host. [See the animated version](https://sureshpallapothu.in/blog/ai-int-day-3-gpu-passthrough).\n\n`--memory=\"10g\"` caps the container at 10 GB and leaves about 6 GB for the host OS, networking daemons and file cache.`--memory-swap=\"10g\"` (equal to `--memory`) disables swap for the container. Swapping model weights to disk slows inference by orders of magnitude, so it is better to fail fast.`--oom-kill-disable=false`), so you do not need to set it. Never use `--oom-kill-disable=true` without a memory limit, because then a spike can hang the whole machine.`--gpus` only when a GPU is present. On a CPU-only laptop, the memory and CPU caps are what protect you.\nTreat these flags as part of your deployment definition, and in Kubernetes you will express the same limits as resource requests and limits.\n\nThis is the exact example from the video. These are real docker run commands that read the limits from inside the container. There is no GPU on my laptop, so only the limits half of the lesson is shown.\n\n```\n# no GPU on this machine, so limits only\ndocker run --rm --cpus=2 --memory=256m --memory-swap=256m \\\n  --shm-size=2g alpine:3.20 sh -c '\n  cd /sys/fs/cgroup\n  for f in memory.max memory.swap.max cpu.max; do\n    echo \"$f: $(cat $f)\"\n  done\n  echo shm: $(df -h /dev/shm | tail -1 | tr -s \" \" | cut -d\" \" -f2)'\n\ndocker run --rm --memory=64m --memory-swap=64m alpine:3.20 \\\n  sh -c 'x=$(dd if=/dev/zero bs=1M count=200 | tr \"\\0\" x)'\necho \"over the cap: exit code $?\"\n```\n\nRun it:\n\n```\nbash limits.sh\n```\n\nOutput I got while recording the video:\n\n```\nmemory.max: 268435456\nmemory.swap.max: 0\ncpu.max: 200000 100000\nshm: 2.0G\nover the cap: exit code 137\n```\n\n**Day 4:** Optimising model weights: quantization basics with GGUF and AWQ.\n\n*Originally published at [https://sureshpallapothu.in/blog/ai-int-day-3-gpu-passthrough](https://sureshpallapothu.in/blog/ai-int-day-3-gpu-passthrough), where this post includes animated diagrams.*", "url": "https://wpnews.pro/news/day-3-hardware-acceleration-in-docker-gpu-passthrough-and-limits-for-constrained", "canonical_source": "https://dev.to/suresh_kumar_de3920bedd1c/day-3-hardware-acceleration-in-docker-gpu-passthrough-and-limits-for-constrained-hardware-b75", "published_at": "2026-11-09 23:08:11+00:00", "updated_at": "2026-11-09 23:23:44.930636+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "ai-tools", "developer-tools"], "entities": ["NVIDIA Container Toolkit", "Docker", "PyTorch", "Ollama", "Kubernetes", "nvidia-container-runtime"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/day-3-hardware-acceleration-in-docker-gpu-passthrough-and-limits-for-constrained", "markdown": "https://wpnews.pro/news/day-3-hardware-acceleration-in-docker-gpu-passthrough-and-limits-for-constrained.md", "text": "https://wpnews.pro/news/day-3-hardware-acceleration-in-docker-gpu-passthrough-and-limits-for-constrained.txt", "jsonld": "https://wpnews.pro/news/day-3-hardware-acceleration-in-docker-gpu-passthrough-and-limits-for-constrained.jsonld"}}