How to ensure safe usage?

Hi, I’m administrating a small server where people train AI models on. How can I make sure, people don’t download harmful models?

I thought of only allowing the usage of safe tensors or a whitelist of vendors. Is there anything like that?

Best, Sebastian

Bidram 2

Hello, I’m not an expert, but I’d like to help.

If you want to prevent users from training models on harmful data, maybe you could moderate the data using a moderation model. If you want a lighter or lower-cost option, you could also use random sampling and review the sampled data, although that would not guarantee that all harmful data is caught. I didn’t fully understand your setup, but you mentioned not wanting people to download harmful models, so I assume users may also be down models directly. If that is the case, you could restrict downloads by using a whitelist or blocklist of specific accounts, repositories, or model sources. For example, allowing only approved sources may help, because even safetensors models can still be harmful.

If by “harmful” you mean uncensored models, those checks may help. But if you mean models or repositories that could be risky for the server itself, then sandboxing and limiting permissions would also be important. So the question is really about models containing malware (i.e., data is not a concern).

The setup is, that people ssh onto the machine. They write python code in their repos. Some of them install the huggingface_hub package and download arbitrary models. I read in different news articles that there are many models that contain harmful code that is shipped easily via the pickled model file and is executed as soon as you load the model. I wanted to guard against such a case.

Now you also talked about the safe tensors format and white-/blacklisting of accounts. But I wonder, how this could be done on a technical level. I could imagine to write some kind of config file for hugging face which I could put into any users home directory upon creation or to set some environment variables. So I don’t need to guard against malicous intend of my users, I just want to make sure that they don’t do something stupid accidentally.

Bidram 4

Hello! Thanks for completing the design.

You said that you are worried about pickle-based models containing harmful things, and that is true. Pickle files can contain Python objects that may be dangerous when loaded.

I can give you two solutions that I found, though these are only my own suggestions and there may be better ones.

Allow safetensors where possible Since pickle files can contain harmful Python objects, it is better to allowsafetensors whenever possible. If needed, you can convert pickle-based models or other formats intosafetensors first. This is safer, but for very large models it may be slow and a bit inefficient. 2. Sandbox every user in a separate virtual machine You can sandbox each user inside a separate virtual machine. Shared drivers and other core components can help speed up VM creation. For each new user, you can create a separate VM with a fresh Python environment, or one that already contains some default libraries if needed. This is usually better for larger models and more general workloads, but it can become harder to maintain as the system grows and may be more complex overall.

So in short, these are the two solutions I found, but there could definitely be better ones depending on the exact setup.

Hi, not an expert but would like to share my suggestion for reference.

Since your goal is to prevent accidental execution rather than stopping determined malicious users, you can handle this locally without needing complex sandboxes.

Network-level filter via HF_ENDPOINT : The huggingface_hub library respects theHF_ENDPOINT environment variable. You can set this globally for all SSH users (e.g., in/etc/profile.d/hf.sh ) to point to a lightweight proxy (like a simple Nginx config) running locally on the server. Have the proxy forward all requests tohttps://huggingface.co , but return a 403 for file extensions associated with pickled code (.bin ,.pt ,.pth ,.pkl ). This automatically restricts users to down.safetensors and config files. 2. Python-level guardrail via sitecustomize.py : If you want to catch the execution itself, you can add asitecustomize.py file to your server’s global Python environment. Because this script executes automatically on Python startup, you can use it to monkey-patchtorch.load to raise a custom warning or error. This effectively acts as a tripwire, reminding users to passuse_safetensors=True when models viatransformers .

Relying on safetensors is exactly the right instinct, you just need to enforce it at the proxy or interpreter level.

VERIFY BEFORE POSTING:

Verify that all download traffic from huggingface_hub strictly routes throughHF_ENDPOINT in the version your users are running (it generally does, but it is worth testing in your specific environment). #

If you go the `sitecustomize.py` route, ensure monkey-patching`torch.load` won’t break users’ legitimate local training workflows (e.g., saving and  their own optimizer states or mid-training checkpoints, which often default to pickle).

 
 
[mfenx](https://discuss.huggingface.co/u/mfenx)

6

We hit the same problem and ended up treating a model file the way you’d treat a binary release:

1. Pin by digest, not by name. Hash the exact artifact (SHA-256 + byte count) when you admit it. A name or a version tag can be re-pointed; a digest can’t.

2. Run your checks against a pinned contract. “Passed evals” only means something if the checks themselves can’t drift — same suite, same thresholds, recorded against that digest.

3. Keep the evidence. Sign the result (digest + contract + pass/fail + who issued it) into a receipt you can re-check later, offline, without re-running the model. That’s what turns a one-time check into an admission policy you can enforce at load time.

I build MFENX, which packages exactly that flow (free verification, no account: MFENX). Even if you hand-roll it with sha256sum + a signed manifest, steps 1–3 close most of the hole — the part most setups miss is #2: binding the checks to the bytes.

For your setup specifically: a whitelist of vendors is a good start, but pair it with digest pinning — a whitelisted vendor’s repo can still be re-uploaded or compromised, and the digest is what tells you the bytes changed. I think the best approach is to treat model outputs as untrusted, especially when dealing with user inputs or external data. Use proper input validation, avoid running generated code blindly, and apply permission limits wherever possible. Testing with edge cases also helps catch potential security issues early.