{"slug": "last-layer-hidden-state-gpt2", "title": "Last layer hidden state: GPT2", "summary": "A user investigating GPT-2's last hidden state in Hugging Face Transformers 3.1.0 found that the final hidden state (layer 12) is reported after the last decoder block but before the final LayerNorm (ln_f), contrary to some expectations. The user observed that applying ln_f manually produced values within precision limits of hidden state 12, suggesting the normalization has negligible effect.", "body_md": "Hi, I am trying to understand exactly where the last last hidden output in GPT2 stems from. The following code outputs hidden states from the embedding layer and all layers (1+12=13 layers):\n\ntokenizer = GPT2Tokenizer.from_pretrained(‘gpt2’)\n\nmodel = GPT2Model.from_pretrained(‘gpt2’)\n\ninput_ids = torch.tensor(tokenizer.encode(sent))\n\noutputs = model(input_ids, output_hidden_states=True, return_dict=True)\n\nhidden_states = result_model[‘hidden_states’]\n\nAs GPT2 has a LayerNorm layer *after* the very last decoder block (ln_f: LayerNorm(768)), are the last hidden states extracted before this final normalization layer (after the last decoder block) or after this final normalization layer?\n\nI am running Transformers 3.1.0.\n\nThanks so much!\n\nFrom what I read, it seemed like GPT-2’s last hidden state would not have gone through a layer norm and that I would have to hook ln_f to have a comparable highest layer. However, when I did that, I found that every value from ln_f was within precision limitations of hidden state 12 (over 1290x16x768). Can anyone give a definitive answer to what is being reported as hidden state 12?", "url": "https://wpnews.pro/news/last-layer-hidden-state-gpt2", "canonical_source": "https://discuss.huggingface.co/t/last-layer-hidden-state-gpt2/5016#post_2", "published_at": "2026-07-25 01:07:06+00:00", "updated_at": "2026-07-25 01:32:03.572329+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools"], "entities": ["GPT-2", "Hugging Face Transformers", "LayerNorm"], "alternates": {"html": "https://wpnews.pro/news/last-layer-hidden-state-gpt2", "markdown": "https://wpnews.pro/news/last-layer-hidden-state-gpt2.md", "text": "https://wpnews.pro/news/last-layer-hidden-state-gpt2.txt", "jsonld": "https://wpnews.pro/news/last-layer-hidden-state-gpt2.jsonld"}}