{"slug": "hugging-face-transformers-load-and-run-a-model", "title": "Hugging Face transformers: Load and Run a Model", "summary": "A developer explains how to load and run Hugging Face transformers models, highlighting two often-skipped steps: applying the chat template and choosing precision deliberately. The post provides code examples and formulas for estimating memory requirements, including weights and KV cache, to avoid common pitfalls.", "body_md": "Running a model with transformers is three objects — a tokenizer, a model, a generation call — and two steps that most tutorials skip and that account for most of the confusing results: applying the chat template, and choosing the precision deliberately rather than accepting the default.\n\nThe `from_pretrained`\n\npattern has been the library’s interface for years and is as stable as anything in this ecosystem gets. The auto classes inspect the repository’s configuration and instantiate the right implementation, so the same four lines work across architectures.\n\n```\ntokenizer = AutoTokenizer.from_pretrained(MODEL_ID)\nmodel     = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map=\"auto\")\n\ninputs  = tokenizer(prompt, return_tensors=\"pt\").to(model.device)\noutputs = model.generate(**inputs, max_new_tokens=256)\nprint(tokenizer.decode(outputs[0], skip_special_tokens=True))\n```\n\nOne detail in that snippet causes a steady stream of confusion: `generate`\n\nreturns the prompt tokens followed by the new ones, so decoding the whole tensor prints your prompt back at you. Slice off the input length, or use the tokenizer’s batch decode on the generated portion only.\n\n```\nprompt_len = inputs[\"input_ids\"].shape[-1]\nnew_tokens = outputs[0][prompt_len:]\nprint(tokenizer.decode(new_tokens, skip_special_tokens=True))\n```\n\nThis is the highest-value paragraph on the page. An instruction-tuned model was trained with a specific format of special tokens marking system, user and assistant turns. Feeding it a bare string skips that format entirely, and the model responds — badly, in a way that looks like the model being poor rather than like a formatting bug.\n\n```\nmessages = [\n    {\"role\": \"system\", \"content\": \"You are terse.\"},\n    {\"role\": \"user\",   \"content\": \"Explain KV caching in two sentences.\"},\n]\n\ntext = tokenizer.apply_chat_template(\n    messages,\n    tokenize=False,\n    add_generation_prompt=True,   # opens the assistant turn\n)\ninputs = tokenizer(text, return_tensors=\"pt\").to(model.device)\n```\n\nThe template ships with the model, so switching models switches formats automatically — which is exactly why hand-writing the format strings is a mistake. Two symptoms tell you the template is missing: the model continues your prompt rather than answering it, and it rambles past where an answer should end because the stop token it was trained to emit never appears in a format it recognises. If a repository has no template, the model is probably a base model rather than an instruction-tuned one, which is a different tool — see [base versus instruct models](https://multigrid.ai/learn/base-vs-instruct-models).\n\nWork this out before downloading forty gigabytes. Every term is countable and none of it depends on a benchmark.\n\n```\nweights = parameters × bytes_per_parameter\n\n  float32  4 bytes    bfloat16  2 bytes    8-bit  1 byte    4-bit  0.5 bytes\n\n  a 7B model:   7e9 × 2 = 14.0 GB at bfloat16\n                7e9 × 4 = 28.0 GB at float32      <- the trap\n                7e9 × 1 =  7.0 GB at 8-bit\n\nKV cache per token, per sequence:\n\n  bytes = 2 × layers × kv_heads × head_dim × bytes_per_element\n          ^ one each for K and V\n\n  worked example: 32 layers, 8 kv heads, head_dim 128, bfloat16\n  = 2 × 32 × 8 × 128 × 2 = 131,072 bytes ≈ 128 KB per token\n\n  4,000 tokens of context  ≈  0.5 GB\n  32,000 tokens            ≈  4.0 GB\n\nTotal ≈ weights + KV cache + activations + framework overhead\n      (allow 1–2 GB for the last two on a single-stream run)\n```\n\nRead the layer count, head counts and head dimension from the model’s config file rather than guessing; they are all in it. Note that models using grouped-query attention have far fewer key-value heads than attention heads, which is the difference between a KV cache that fits and one that does not — the mechanism is covered in [the KV cache](https://multigrid.ai/learn/kv-cache).\n\nThe arithmetic is exact for weights and close for the cache; real usage adds allocator fragmentation and framework buffers. If the calculation says you have 300 MB of headroom, you do not have headroom.\n\nTwo arguments decide whether the model loads and how fast it runs. Both have caused enough confusion to be worth stating carefully.\n\n**Precision.** Loading in the model’s native [reduced precision](https://multigrid.ai/learn/floating-point-formats) rather than upcasting to float32 halves the memory and is almost always what you want on modern hardware. The keyword argument that controls this was renamed in recent versions of the library — older code passes `torch_dtype`\n\n, newer code passes `dtype`\n\n— so check which one your installed version expects rather than assuming. The value `\"auto\"`\n\ntakes the precision recorded in the model’s own config, which is usually the right answer.\n\n**Placement.** `device_map=\"auto\"`\n\nspreads the model across available devices and, if it does not fit, will place layers on CPU or disk. That is a feature and a trap: the model loads, and generation is thirty times slower than expected because half of it is being read from CPU memory on every token. If throughput collapses, check where the layers actually went before blaming anything else.\n\nThe generation arguments have been stable for years and a handful of them account for nearly all the behaviour.\n\n| Argument | Description |\n|---|---|\n| max_new_tokens | How many tokens to generate, not counting the prompt. Prefer this over the total-length argument, which includes the prompt and therefore silently shortens answers to long prompts. |\n| do_sample | Off means greedy decoding: the highest-probability token every time, deterministic given the same input. On enables sampling and makes temperature and the top-p and top-k cutoffs meaningful. Setting temperature while sampling is off does nothing, which is a common source of confusion. |\n| temperature, top_p, top_k | The sampler's shape. Only relevant with sampling enabled. Defaults come from the model's own generation config, so two models with identical arguments can behave differently. |\n| repetition_penalty | Discourages repeating tokens already produced. Useful against degenerate loops on smaller models; heavy values distort ordinary text, particularly lists and code. |\n| eos_token_id / stopping criteria | When to stop. Chat models emit an end-of-turn token that may differ from the tokenizer's default end-of-sequence token, which is the usual reason a model keeps talking after finishing its answer. |\n\nBatching several prompts into one call is the easiest throughput win available, and there is one rule that decides whether it works: for decoder-only generation, padding must be on the left.\n\nThe reason is mechanical. Generation continues from the last position of the sequence. With right padding, the last positions are pad tokens, so the model continues from padding and produces nonsense for every sequence shorter than the longest. Set the tokenizer’s padding side to left before batching, and set a padding token if the tokenizer has none — many causal models ship without one, and reusing the end-of-sequence token for the purpose is the conventional fix.\n\nThe symptom is distinctive and worth memorising: the longest prompt in the batch produces a good answer and the shorter ones produce garbage. That is left-padding, every time.\n\nText generation is the thing transformers is worst at relative to the alternatives, and it is the only thing most tutorials show. Two other uses have no comparable alternative, and both are reasons to keep it installed even after you serve models elsewhere.\n\n**Raw logits.** Calling the model directly rather than through the generation helper returns scores over the vocabulary for every position. That gives you a probability for a specific continuation — which is how you build a classifier that outputs a calibrated score rather than a word, and how you compute [perplexity](https://multigrid.ai/learn/perplexity-explained) over a corpus. No hosted API exposes this fully, and where one exposes logprobs it is a truncated view. The mechanics of what those numbers mean are in [logprobs explained](https://multigrid.ai/learn/logprobs-explained).\n\n```\nout = model(**inputs)          # no generate(); one forward pass\nout.logits.shape               # (batch, sequence, vocab)\n\n# Score of a specific next token, as a probability:\nprobs = out.logits[0, -1].softmax(dim=-1)\nprobs[tokenizer.encode(\" yes\", add_special_tokens=False)[0]]\n```\n\n**Hidden states.** Requesting them returns the internal representations at every layer, which is the starting point for probing, feature extraction and any analysis of what a model is representing. For embeddings specifically, use a model trained for it rather than pooling a generative model’s hidden states — the reasons are on [the Sentence Transformers page](https://multigrid.ai/learn/sentence-transformers-guide) — but the mechanism is the same.\n\nThe third is fine-tuning, which lives in the surrounding ecosystem rather than in this library alone but starts from the same loaded model object. Whether it is the right move at all is a separate question, addressed in [should you fine-tune](https://multigrid.ai/learn/should-you-fine-tune).\n\nIt is a research and experimentation library. For serving — concurrent users, continuous batching, paged attention, a stable HTTP endpoint — a dedicated inference server is not an optimisation, it is a different category of software. A naive loop over `generate`\n\nbehind a web framework will serve requests one at a time and idle the GPU between tokens.\n\nUse transformers to test whether a model can do the task, to compute embeddings or logits, and to fine-tune. Then serve it with [vLLM](https://multigrid.ai/learn/vllm-guide) or [TGI](https://multigrid.ai/learn/tgi-guide), both of which load the same checkpoints and speak HTTP. The throughput difference between those and a naive loop is not marginal; it is the reason those projects exist, and the underlying mechanism is [continuous batching](https://multigrid.ai/learn/continuous-batching).", "url": "https://wpnews.pro/news/hugging-face-transformers-load-and-run-a-model", "canonical_source": "https://dev.to/multigrid/hugging-face-transformers-load-and-run-a-model-2e8n", "published_at": "2026-08-12 18:01:23+00:00", "updated_at": "2026-08-12 18:18:04.571699+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools"], "entities": ["Hugging Face", "AutoTokenizer", "AutoModelForCausalLM", "multigrid.ai"], "alternates": {"html": "https://wpnews.pro/news/hugging-face-transformers-load-and-run-a-model", "markdown": "https://wpnews.pro/news/hugging-face-transformers-load-and-run-a-model.md", "text": "https://wpnews.pro/news/hugging-face-transformers-load-and-run-a-model.txt", "jsonld": "https://wpnews.pro/news/hugging-face-transformers-load-and-run-a-model.jsonld"}}