{"slug": "how-to-use-model-locally-using-in-cli-for-testing-purposes-no-gui-tool", "title": "How to use Model locally using in CLI for testing purposes no GUI tool", "summary": "A community forum answer lays out a two-step method for running a language model locally from the command line without a GUI: download Qwen/Qwen2.5-0.5B-Instruct once with `hf download ... --local-dir ./qwen` from the huggingface_hub package, then load it via the transformers `pipeline` with `local_files_only=True` and `device_map=\"auto\"`. The answer attributes most \"hundreds of errors\" to pointing the loader at a single file instead of the folder containing config.json, or to missing torch and accelerate installs, and notes that `transformers chat Qwen/Qwen2.5-0.5B-Instruct` offers a no-code terminal chat. A second reply recommends Ollama as a single-install, single-command CLI alternative requiring no Python environment.", "body_md": "Hello everybody, How can I download the models locally, and how inference them smoothly without having hundered of errors.\n\nand it’s too hard to use transfomer lib to and forcing him to load the local model. can anyone help?\n\n \n \nHi! Easiest way is two steps.\n\n**1. Download the model once**\n\n```\npip install -U huggingface_hub\nhf download Qwen/Qwen2.5-0.5B-Instruct --local-dir ./qwen\n```\n\n**2. Load it from that folder**\n\n```\npip install torch transformers accelerate\npython\nfrom transformers import pipeline\n\npipe = pipeline(\n    \"text-generation\",\n    model=\"./qwen\",          # folder that contains config.json\n    device_map=\"auto\",\n    local_files_only=True,   # never hits the internet\n)\n\nmessages = [{\"role\": \"user\", \"content\": \"Hello!\"}]\nout = pipe(messages, max_new_tokens=100)\nprint(out[0][\"generated_text\"][-1][\"content\"])\n```\n\nMost “hundreds of errors” come from a wrong path (it must be the folder with `config.json`, not a single file) or from missing `torch`/` accelerate`.\n\nIf you just want to chat in the terminal with no code, this also works (see the “Chat basics” docs):\n\n```\ntransformers chat Qwen/Qwen2.5-0.5B-Instruct\n```\n\nIf you still get an error, paste the last lines of the traceback and the model name, and we can look at it.\n\n \n \nOne more beginner friendly option if the transformers route gives you trouble: Ollama gives you a pure CLI chat with a single install and a single command to pull and run a model, no Python environment needed. It is worth trying when you just want to test a model quickly. If you stick with transformers, make sure the local folder really contains config.json and the model weights, since a wrong path is the most common error.", "url": "https://wpnews.pro/news/how-to-use-model-locally-using-in-cli-for-testing-purposes-no-gui-tool", "canonical_source": "https://discuss.huggingface.co/t/how-to-use-model-locally-using-in-cli-for-testing-purposes-no-gui-tool/190954#post_3", "published_at": "2026-10-24 06:05:42+00:00", "updated_at": "2026-10-24 06:11:25.688313+00:00", "lang": "en", "topics": ["ai-tools", "large-language-models", "developer-tools"], "entities": ["Qwen2.5-0.5B-Instruct", "Hugging Face", "transformers", "Ollama", "torch", "accelerate"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-use-model-locally-using-in-cli-for-testing-purposes-no-gui-tool", "markdown": "https://wpnews.pro/news/how-to-use-model-locally-using-in-cli-for-testing-purposes-no-gui-tool.md", "text": "https://wpnews.pro/news/how-to-use-model-locally-using-in-cli-for-testing-purposes-no-gui-tool.txt", "jsonld": "https://wpnews.pro/news/how-to-use-model-locally-using-in-cli-for-testing-purposes-no-gui-tool.jsonld"}}