My Space, builds successfully but fails during container initialization, before application code starts.
It uses the Docker SDK and currently requests L4 hardware (l4x1).
Exit code: 128. Reason: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: failed to fulfil mount request: failed to set MOUNT_ATTR_IDMAP on /dev/null: invalid argument (maybe the filesystem used doesn't support idmap mounts on this kernel?)
Restarting and factory rebuilding have not resolved the issue.
I also created a minimal Docker hello-world reproduction using the same base image and user ID, without any ML dependencies. With the same test code and Dockerfile:
- T4 Medium: fails with the identical container-initialization error.
- CPU Basic: starts successfully and serves the page and health endpoint with HTTP 200.
The reproduction Space is private, but I can provide the minimal Dockerfile and application code if helpful.
Could you please ?
Thank you!
Passing the Docker build and running cleanly on CPU Basic while failing before ENTRYPOINT on l4x1 and T4 Medium with failed to set MOUNT_ATTR_IDMAP on /dev/null: invalid argument points directly to how the GPU runner attaches device nodes under user-namespace ID mapping.
On GPU nodes, runc and NVIDIA Container Toolkit inject /dev/null and /dev/nvidia* character devices into the container rootfs, and if the runtime attempts mount_setattr(MOUNT_ATTR_IDMAP) against a filesystem/device layer or UID setup that kernel devtmpfs idmapping rejects, container creation fails with EINVAL before your Python process starts.
A few concrete things to inspect in your Dockerfile:
Check USER and UID alignment: Hugging Face Docker Spaces expect USER 1000 (creating the non-root user with -u 1000). If the base image defines a different user, high UIDs/GIDs, or custom /etc/subuid shifts, drop back to standard uid=1000 ownership on /app and /home/user.
Look for conflicting VOLUME or device directives in the base image: inspect docker history / docker inspect of your FROM image for unexpected VOLUME mounts or embedded nvidia-container-toolkit / systemd / udev packages that fight the host runtime’s /dev hooks.
Sanity test with a minimal GPU Dockerfile: strip your Dockerfile down to FROM python:3.11-slim (or nvidia/cuda:12.4.1-runtime-ubuntu22.04) with useradd -m -u 1000 user, USER user, and a one-line HTTP server. If that minimal baseline also fails on l4x1 / T4 Medium with MOUNT_ATTR_IDMAP on /dev/null: invalid argument, flag it to Hugging Face Spaces support with the Space URL because that exact kernel/runc idmap regression happens on the GPU node runtime side rather than inside your app code.
It seems to be resolved now. I didn’t change anything. Probably some HF internal issues.