# Issue with LlamaSdpaAttention Not Being Utilized

> Source: <https://discuss.huggingface.co/t/issue-with-llamasdpaattention-not-being-utilized/140678#post_3>
> Published: 2026-10-25 13:31:56+00:00

A version-specific clarification for [@jacobvinje](https://discuss.huggingface.co/u/jacobvinje) and future readers: in **Transformers 4.48.3**, seeing `LlamaAttention` in `print(model)` does **not** by itself mean SDPA is disabled.

In that version, `LlamaAttention.forward` selects its attention function internally using `self.config._attn_implementation`. The implementation is visible in the [v4.48.3 source](https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/llama/modeling_llama.py).

For an existing loaded model, this is a useful first check:

```
print(model.config._attn_implementation)
```

That reports the configured implementation, not proof of the GPU kernel actually executed. In this version, requesting `output_attentions=True` with SDPA triggers an eager-attention fallback. A profiler is needed to verify the executed operations and any memory implications.

So the unchanged class name is consistent with SDPA being selected; it is not, on its own, a reproducible backend failure. This clarification does not establish the cause of the separate memory issue you reported.

For a fuller explanation, I also maintain a free [Transformer guide (English and Spanish)](https://karlesmarin.github.io/transformers-guide/). The [KV-cache chapter](https://karlesmarin.github.io/transformers-guide/en/chapters/20-kv-cache.html) explains the memory budget and why fitting in memory, retaining attention mass and preserving task quality are separate questions.