{"slug": "gpu-docker-space-fails-at-startup-mount-attr-idmap-on-dev-null-cpu-test-works", "title": "GPU Docker Space fails at startup: MOUNT_ATTR_IDMAP on /dev/null — CPU test works", "summary": "A Hugging Face Spaces user reported that Docker SDK Spaces requesting L4 (l4x1) and T4 Medium GPU hardware failed at container initialization with exit code 128 and the error \"failed to set MOUNT_ATTR_IDMAP on /dev/null: invalid argument,\" while the identical minimal Dockerfile started successfully on CPU Basic and served its page and health endpoint with HTTP 200. The failure occurred before application code ran, and neither restarting nor factory rebuilding fixed it; the user later said the problem appeared to resolve on its own without any changes, attributing it to Hugging Face internal issues.", "body_md": "My Space, builds successfully but fails during container initialization, before application code starts.\n\nIt uses the Docker SDK and currently requests L4 hardware (`l4x1`).\n\n```\nExit code: 128. Reason: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: failed to fulfil mount request: failed to set MOUNT_ATTR_IDMAP on /dev/null: invalid argument (maybe the filesystem used doesn't support idmap mounts on this kernel?)\n```\n\nRestarting and factory rebuilding have not resolved the issue.\n\nI also created a minimal Docker hello-world reproduction using the same base image and user ID, without any ML dependencies. With the same test code and Dockerfile:\n\n- **T4 Medium:** fails with the identical container-initialization error.\n- **CPU Basic:** starts successfully and serves the page and health endpoint with HTTP 200.\n\nThe reproduction Space is private, but I can provide the minimal Dockerfile and application code if helpful.\n\nCould you please ?\n\nThank you!\n\n \n \nPassing the Docker build and running cleanly on CPU Basic while failing before `ENTRYPOINT` on `l4x1` and T4 Medium with `failed to set MOUNT_ATTR_IDMAP on /dev/null: invalid argument` points directly to how the GPU runner attaches device nodes under user-namespace ID mapping.\n\nOn GPU nodes, `runc` and NVIDIA Container Toolkit inject `/dev/null` and `/dev/nvidia*` character devices into the container rootfs, and if the runtime attempts `mount_setattr(MOUNT_ATTR_IDMAP)` against a filesystem/device layer or UID setup that kernel devtmpfs idmapping rejects, container creation fails with `EINVAL` before your Python process starts.\n\nA few concrete things to inspect in your `Dockerfile`:\n\nCheck `USER` and UID alignment: Hugging Face Docker Spaces expect `USER 1000` (creating the non-root user with `-u 1000`). If the base image defines a different user, high UIDs/GIDs, or custom `/etc/subuid` shifts, drop back to standard `uid=1000` ownership on `/app` and `/home/user`.\n\nLook for conflicting `VOLUME` or device directives in the base image: inspect `docker history` / `docker inspect` of your `FROM` image for unexpected `VOLUME` mounts or embedded `nvidia-container-toolkit` / `systemd` / `udev` packages that fight the host runtime’s `/dev` hooks.\n\nSanity test with a minimal GPU Dockerfile: strip your `Dockerfile` down to `FROM python:3.11-slim` (or `nvidia/cuda:12.4.1-runtime-ubuntu22.04`) with `useradd -m -u 1000 user`, `USER user`, and a one-line HTTP server. If that minimal baseline **also** fails on `l4x1` / T4 Medium with `MOUNT_ATTR_IDMAP on /dev/null: invalid argument`, flag it to Hugging Face Spaces support with the Space URL because that exact kernel/runc idmap regression happens on the GPU node runtime side rather than inside your app code.\n\n \n \nIt seems to be resolved now. I didn’t change anything. Probably some HF internal issues.", "url": "https://wpnews.pro/news/gpu-docker-space-fails-at-startup-mount-attr-idmap-on-dev-null-cpu-test-works", "canonical_source": "https://discuss.huggingface.co/t/gpu-docker-space-fails-at-startup-mount-attr-idmap-on-dev-null-cpu-test-works/192204#post_3", "published_at": "2026-11-21 08:53:05+00:00", "updated_at": "2026-11-21 09:14:01.851656+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "developer-tools"], "entities": ["Hugging Face", "Hugging Face Spaces", "Docker", "NVIDIA Container Toolkit", "runc", "containerd"], "also_reported_by": [], "takeaway": "GPU Docker Spaces on l4x1 and T4 Medium failed before ENTRYPOINT with MOUNT_ATTR_IDMAP on /dev/null: invalid argument while the same image ran on CPU Basic, pointing to the GPU node's runc/NVIDIA device-injection and user-namespace ID mapping rather than app code.", "recommendation": {"advice": "When a Docker Space builds but dies before ENTRYPOINT on GPU hardware, reproduce with a minimal python:3.11-slim image at USER 1000; if it also fails, escalate to Hugging Face Spaces support with the Space URL instead of changing app code.", "playbook": "production-ops", "playbook_url": "https://wpnews.pro/playbooks/production-ops"}, "alternates": {"html": "https://wpnews.pro/news/gpu-docker-space-fails-at-startup-mount-attr-idmap-on-dev-null-cpu-test-works", "markdown": "https://wpnews.pro/news/gpu-docker-space-fails-at-startup-mount-attr-idmap-on-dev-null-cpu-test-works.md", "text": "https://wpnews.pro/news/gpu-docker-space-fails-at-startup-mount-attr-idmap-on-dev-null-cpu-test-works.txt", "jsonld": "https://wpnews.pro/news/gpu-docker-space-fails-at-startup-mount-attr-idmap-on-dev-null-cpu-test-works.jsonld"}}