Issue with LlamaSdpaAttention Not Being Utilized

In Transformers 4.48.3, a model printing LlamaAttention in print(model) does not by itself indicate that SDPA is disabled, because LlamaAttention.forward selects its attention function internally via self.config._attn_implementation, according to a Hugging Face forum clarification. The post advises checking print(model.config._attn_implementation) as a first step, noting that this reports the configured implementation rather than proof of the GPU kernel actually executed, and that requesting output_attentions=True with SDPA triggers an eager-attention fallback in that version. The clarification states the unchanged class name is consistent with SDPA being selected and does not establish the cause of the separate memory issue reported.

A version-specific clarification for @jacobvinje https://discuss.huggingface.co/u/jacobvinje and future readers: in Transformers 4.48.3 , seeing LlamaAttention in print model does not by itself mean SDPA is disabled. In that version, LlamaAttention.forward selects its attention function internally using self.config. attn implementation . The implementation is visible in the v4.48.3 source https://github.com/huggingface/transformers/blob/v4.48.3/src/transformers/models/llama/modeling llama.py . For an existing loaded model, this is a useful first check: print model.config. attn implementation That reports the configured implementation, not proof of the GPU kernel actually executed. In this version, requesting output attentions=True with SDPA triggers an eager-attention fallback. A profiler is needed to verify the executed operations and any memory implications. So the unchanged class name is consistent with SDPA being selected; it is not, on its own, a reproducible backend failure. This clarification does not establish the cause of the separate memory issue you reported. For a fuller explanation, I also maintain a free Transformer guide English and Spanish https://karlesmarin.github.io/transformers-guide/ . The KV-cache chapter https://karlesmarin.github.io/transformers-guide/en/chapters/20-kv-cache.html explains the memory budget and why fitting in memory, retaining attention mass and preserving task quality are separate questions.