# GPU Docker Space fails at startup: MOUNT_ATTR_IDMAP on /dev/null — CPU test works

> Source: <https://discuss.huggingface.co/t/gpu-docker-space-fails-at-startup-mount-attr-idmap-on-dev-null-cpu-test-works/192204#post_3>
> Published: 2026-11-21 08:53:05+00:00

**Recommendation:** When a Docker Space builds but dies before ENTRYPOINT on GPU hardware, reproduce with a minimal python:3.11-slim image at USER 1000; if it also fails, escalate to Hugging Face Spaces support with the Space URL instead of changing app code.
More in the playbook "Running AI in production": <https://wpnews.pro/playbooks/production-ops>

My Space, builds successfully but fails during container initialization, before application code starts.

It uses the Docker SDK and currently requests L4 hardware (`l4x1`).

```
Exit code: 128. Reason: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: failed to fulfil mount request: failed to set MOUNT_ATTR_IDMAP on /dev/null: invalid argument (maybe the filesystem used doesn't support idmap mounts on this kernel?)
```

Restarting and factory rebuilding have not resolved the issue.

I also created a minimal Docker hello-world reproduction using the same base image and user ID, without any ML dependencies. With the same test code and Dockerfile:

- **T4 Medium:** fails with the identical container-initialization error.
- **CPU Basic:** starts successfully and serves the page and health endpoint with HTTP 200.

The reproduction Space is private, but I can provide the minimal Dockerfile and application code if helpful.

Could you please ?

Thank you!

 
 
Passing the Docker build and running cleanly on CPU Basic while failing before `ENTRYPOINT` on `l4x1` and T4 Medium with `failed to set MOUNT_ATTR_IDMAP on /dev/null: invalid argument` points directly to how the GPU runner attaches device nodes under user-namespace ID mapping.

On GPU nodes, `runc` and NVIDIA Container Toolkit inject `/dev/null` and `/dev/nvidia*` character devices into the container rootfs, and if the runtime attempts `mount_setattr(MOUNT_ATTR_IDMAP)` against a filesystem/device layer or UID setup that kernel devtmpfs idmapping rejects, container creation fails with `EINVAL` before your Python process starts.

A few concrete things to inspect in your `Dockerfile`:

Check `USER` and UID alignment: Hugging Face Docker Spaces expect `USER 1000` (creating the non-root user with `-u 1000`). If the base image defines a different user, high UIDs/GIDs, or custom `/etc/subuid` shifts, drop back to standard `uid=1000` ownership on `/app` and `/home/user`.

Look for conflicting `VOLUME` or device directives in the base image: inspect `docker history` / `docker inspect` of your `FROM` image for unexpected `VOLUME` mounts or embedded `nvidia-container-toolkit` / `systemd` / `udev` packages that fight the host runtime’s `/dev` hooks.

Sanity test with a minimal GPU Dockerfile: strip your `Dockerfile` down to `FROM python:3.11-slim` (or `nvidia/cuda:12.4.1-runtime-ubuntu22.04`) with `useradd -m -u 1000 user`, `USER user`, and a one-line HTTP server. If that minimal baseline **also** fails on `l4x1` / T4 Medium with `MOUNT_ATTR_IDMAP on /dev/null: invalid argument`, flag it to Hugging Face Spaces support with the Space URL because that exact kernel/runc idmap regression happens on the GPU node runtime side rather than inside your app code.

 
 
It seems to be resolved now. I didn’t change anything. Probably some HF internal issues.
