GPU Docker Space fails at startup: MOUNT_ATTR_IDMAP on /dev/null — CPU test works A Hugging Face Spaces user reported that Docker SDK Spaces requesting L4 (l4x1) and T4 Medium GPU hardware failed at container initialization with exit code 128 and the error "failed to set MOUNT_ATTR_IDMAP on /dev/null: invalid argument," while the identical minimal Dockerfile started successfully on CPU Basic and served its page and health endpoint with HTTP 200. The failure occurred before application code ran, and neither restarting nor factory rebuilding fixed it; the user later said the problem appeared to resolve on its own without any changes, attributing it to Hugging Face internal issues. My Space, builds successfully but fails during container initialization, before application code starts. It uses the Docker SDK and currently requests L4 hardware l4x1 . Exit code: 128. Reason: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: failed to fulfil mount request: failed to set MOUNT ATTR IDMAP on /dev/null: invalid argument maybe the filesystem used doesn't support idmap mounts on this kernel? Restarting and factory rebuilding have not resolved the issue. I also created a minimal Docker hello-world reproduction using the same base image and user ID, without any ML dependencies. With the same test code and Dockerfile: - T4 Medium: fails with the identical container-initialization error. - CPU Basic: starts successfully and serves the page and health endpoint with HTTP 200. The reproduction Space is private, but I can provide the minimal Dockerfile and application code if helpful. Could you please ? Thank you Passing the Docker build and running cleanly on CPU Basic while failing before ENTRYPOINT on l4x1 and T4 Medium with failed to set MOUNT ATTR IDMAP on /dev/null: invalid argument points directly to how the GPU runner attaches device nodes under user-namespace ID mapping. On GPU nodes, runc and NVIDIA Container Toolkit inject /dev/null and /dev/nvidia character devices into the container rootfs, and if the runtime attempts mount setattr MOUNT ATTR IDMAP against a filesystem/device layer or UID setup that kernel devtmpfs idmapping rejects, container creation fails with EINVAL before your Python process starts. A few concrete things to inspect in your Dockerfile : Check USER and UID alignment: Hugging Face Docker Spaces expect USER 1000 creating the non-root user with -u 1000 . If the base image defines a different user, high UIDs/GIDs, or custom /etc/subuid shifts, drop back to standard uid=1000 ownership on /app and /home/user . Look for conflicting VOLUME or device directives in the base image: inspect docker history / docker inspect of your FROM image for unexpected VOLUME mounts or embedded nvidia-container-toolkit / systemd / udev packages that fight the host runtime’s /dev hooks. Sanity test with a minimal GPU Dockerfile: strip your Dockerfile down to FROM python:3.11-slim or nvidia/cuda:12.4.1-runtime-ubuntu22.04 with useradd -m -u 1000 user , USER user , and a one-line HTTP server. If that minimal baseline also fails on l4x1 / T4 Medium with MOUNT ATTR IDMAP on /dev/null: invalid argument , flag it to Hugging Face Spaces support with the Space URL because that exact kernel/runc idmap regression happens on the GPU node runtime side rather than inside your app code. It seems to be resolved now. I didn’t change anything. Probably some HF internal issues.