Last layer hidden state: GPT2

A user investigating GPT-2's last hidden state in Hugging Face Transformers 3.1.0 found that the final hidden state (layer 12) is reported after the last decoder block but before the final LayerNorm (ln_f), contrary to some expectations. The user observed that applying ln_f manually produced values within precision limits of hidden state 12, suggesting the normalization has negligible effect.

Hi, I am trying to understand exactly where the last last hidden output in GPT2 stems from. The following code outputs hidden states from the embedding layer and all layers 1+12=13 layers : tokenizer = GPT2Tokenizer.from pretrained ‘gpt2’ model = GPT2Model.from pretrained ‘gpt2’ input ids = torch.tensor tokenizer.encode sent outputs = model input ids, output hidden states=True, return dict=True hidden states = result model ‘hidden states’ As GPT2 has a LayerNorm layer after the very last decoder block ln f: LayerNorm 768 , are the last hidden states extracted before this final normalization layer after the last decoder block or after this final normalization layer? I am running Transformers 3.1.0. Thanks so much From what I read, it seemed like GPT-2’s last hidden state would not have gone through a layer norm and that I would have to hook ln f to have a comparable highest layer. However, when I did that, I found that every value from ln f was within precision limitations of hidden state 12 over 1290x16x768 . Can anyone give a definitive answer to what is being reported as hidden state 12?