Choose number of labels in WhisperForAudioClassification

Hugging Face forum user dpizza reported that the WhisperForAudioClassification class from the transformers library ignores the num_labels argument when loading openai/whisper-tiny, leaving the classifier with two output labels instead of one. A solution involves copying the model config, editing the id2label parameter to {0: 'prob'}, and reloading the model with the modified config, which changes the classifier's out_features to 1.

dpizza https://discuss.huggingface.co/u/dpizza 1 I would like to use Whisper to fine tune it for a binary audio classification task, so I’m using the WhisperForAudioClassification class and loading the pre-trained model like this: WhisperForAudioClassification.from pretrained "openai/whisper-tiny" The resulting architecture is as follows WhisperForAudioClassification encoder : WhisperEncoder conv1 : Conv1d 80, 384, kernel size= 3, , stride= 1, , padding= 1, conv2 : Conv1d 384, 384, kernel size= 3, , stride= 2, , padding= 1, embed positions : Embedding 1500, 384 layers : ModuleList 0-3 : 4 x WhisperEncoderLayer self attn : WhisperAttention k proj : Linear in features=384, out features=384, bias=False v proj : Linear in features=384, out features=384, bias=True q proj : Linear in features=384, out features=384, bias=True out proj : Linear in features=384, out features=384, bias=True self attn layer norm : LayerNorm 384, , eps=1e-05, elementwise affine=True activation fn : GELUActivation fc1 : Linear in features=384, out features=1536, bias=True fc2 : Linear in features=1536, out features=384, bias=True final layer norm : LayerNorm 384, , eps=1e-05, elementwise affine=True layer norm : LayerNorm 384, , eps=1e-05, elementwise affine=True projector : Linear in features=384, out features=256, bias=True classifier : Linear in features=256, out features=2, bias=True The issue here is that the last layer outputs two labels. How do I change the last layer so that it outputs just a single one? I have already tried to include a keyword argument num labels=1 but it seems to get ignored. The documentation talks about a config argument that can be added to the from pretrained method that could solve my problem, but I’m not sure what to put there exactly. dpizza https://discuss.huggingface.co/u/dpizza 2 After fiddling around, I found a possible solution: 1 - Copy config from the Whisper model: config = WhisperForAudioClassification.from pretrained 'openai/whisper-tiny' .config 2 - Edit the id2label parameter: config.id2label {0: 'LABEL 0', 1: 'LABEL 1'} config.id2label = {0: 'prob'} 3 - Re-download the model with the new config: model = WhisperForAudioClassification.from pretrained 'openai/whisper-tiny', config=config This is the result, there is just one output as expected. WhisperForAudioClassification encoder : WhisperEncoder conv1 : Conv1d 80, 384, kernel size= 3, , stride= 1, , padding= 1, conv2 : Conv1d 384, 384, kernel size= 3, , stride= 2, , padding= 1, embed positions : Embedding 1500, 384 layers : ModuleList 0-3 : 4 x WhisperEncoderLayer self attn : WhisperAttention k proj : Linear in features=384, out features=384, bias=False v proj : Linear in features=384, out features=384, bias=True q proj : Linear in features=384, out features=384, bias=True out proj : Linear in features=384, out features=384, bias=True self attn layer norm : LayerNorm 384, , eps=1e-05, elementwise affine=True activation fn : GELUActivation fc1 : Linear in features=384, out features=1536, bias=True fc2 : Linear in features=1536, out features=384, bias=True final layer norm : LayerNorm 384, , eps=1e-05, elementwise affine=True layer norm : LayerNorm 384, , eps=1e-05, elementwise affine=True projector : Linear in features=384, out features=256, bias=True classifier : Linear in features=256, out features=1, bias=True Just use num labels when defining, eg: model = WhisperForAudioClassification.from pretrained model name, num labels=2 mohamed https://mohamed.com Just use num labels when defining, eg: model = WhisperForAudioClassification.from pretrained model name, num labels=2 model = WhisperForAudioClassification.from pretrained model name, num labels=2 Can someone give me a simple example on how to train Wav2Vec2 for audio frame classificationDDDAAA?Can someone give me a simple example on how to train Wav2Vec2 for audio frame classification?