Based on the current Hugging Face docs, I would not assume that a fine-grained token by itself gives you a clean “user X can call endpoint A but not endpoint B” rule.
Fine-grained User Access Tokens are documented as being scopeable to specific Hub resources (for example specific models). Dedicated Inference Endpoints separately expose authentication modes such as private/org-member access, public, or authenticated-user access. Enterprise Resource Groups can restrict who may use the Inference Endpoints feature, but the docs explicitly describe that feature restriction as controlling who can use the feature, not where inside the organization.
So if your requirement is strict per-endpoint authorization between users, I’d put an authorization proxy/gateway in front of the endpoints (or use a stronger isolation boundary) and map verified principals to allowed endpoint URLs there.
I’d verify this against your current plan/API before committing to the design, because endpoint-access capabilities can evolve independently of Hub token scopes.