{"slug": "choose-number-of-labels-in-whisperforaudioclassification", "title": "Choose number of labels in WhisperForAudioClassification", "summary": "Hugging Face forum user dpizza reported that the WhisperForAudioClassification class from the transformers library ignores the num_labels argument when loading openai/whisper-tiny, leaving the classifier with two output labels instead of one. A solution involves copying the model config, editing the id2label parameter to {0: 'prob'}, and reloading the model with the modified config, which changes the classifier's out_features to 1.", "body_md": "[dpizza](https://discuss.huggingface.co/u/dpizza)\n1\nI would like to use Whisper to fine tune it for a binary audio classification task, so I’m using the `WhisperForAudioClassification`\n\nclass and loading the pre-trained model like this:\n\n```\nWhisperForAudioClassification.from_pretrained(\"openai/whisper-tiny\")\n```\n\nThe resulting architecture is as follows\n\n```\nWhisperForAudioClassification(\n  (encoder): WhisperEncoder(\n    (conv1): Conv1d(80, 384, kernel_size=(3,), stride=(1,), padding=(1,))\n    (conv2): Conv1d(384, 384, kernel_size=(3,), stride=(2,), padding=(1,))\n    (embed_positions): Embedding(1500, 384)\n    (layers): ModuleList(\n      (0-3): 4 x WhisperEncoderLayer(\n        (self_attn): WhisperAttention(\n          (k_proj): Linear(in_features=384, out_features=384, bias=False)\n          (v_proj): Linear(in_features=384, out_features=384, bias=True)\n          (q_proj): Linear(in_features=384, out_features=384, bias=True)\n          (out_proj): Linear(in_features=384, out_features=384, bias=True)\n        )\n        (self_attn_layer_norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n        (activation_fn): GELUActivation()\n        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n        (final_layer_norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n      )\n    )\n    (layer_norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n  )\n  (projector): Linear(in_features=384, out_features=256, bias=True)\n  (classifier): Linear(in_features=256, out_features=2, bias=True)\n)\n```\n\nThe issue here is that the last layer outputs two labels. How do I change the last layer so that it outputs just a single one? I have already tried to include a keyword argument `num_labels=1`\n\nbut it seems to get ignored.\n\nThe documentation talks about a `config`\n\nargument that can be added to the `from_pretrained`\n\nmethod that could solve my problem, but I’m not sure what to put there exactly.\n\n[dpizza](https://discuss.huggingface.co/u/dpizza)\n2\nAfter fiddling around, I found a possible solution:\n\n1 - Copy config from the Whisper model:\n\n```\nconfig = WhisperForAudioClassification.from_pretrained('openai/whisper-tiny').config\n```\n\n2 - Edit the `id2label`\n\nparameter:\n\n```\nconfig.id2label\n>>> {0: 'LABEL_0', 1: 'LABEL_1'}\n\nconfig.id2label = {0: 'prob'}\n```\n\n3 - Re-download the model with the new config:\n\n```\nmodel = WhisperForAudioClassification.from_pretrained('openai/whisper-tiny', config=config)\n```\n\nThis is the result, there is just one output as expected.\n\n```\nWhisperForAudioClassification(\n  (encoder): WhisperEncoder(\n    (conv1): Conv1d(80, 384, kernel_size=(3,), stride=(1,), padding=(1,))\n    (conv2): Conv1d(384, 384, kernel_size=(3,), stride=(2,), padding=(1,))\n    (embed_positions): Embedding(1500, 384)\n    (layers): ModuleList(\n      (0-3): 4 x WhisperEncoderLayer(\n        (self_attn): WhisperAttention(\n          (k_proj): Linear(in_features=384, out_features=384, bias=False)\n          (v_proj): Linear(in_features=384, out_features=384, bias=True)\n          (q_proj): Linear(in_features=384, out_features=384, bias=True)\n          (out_proj): Linear(in_features=384, out_features=384, bias=True)\n        )\n        (self_attn_layer_norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n        (activation_fn): GELUActivation()\n        (fc1): Linear(in_features=384, out_features=1536, bias=True)\n        (fc2): Linear(in_features=1536, out_features=384, bias=True)\n        (final_layer_norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n      )\n    )\n    (layer_norm): LayerNorm((384,), eps=1e-05, elementwise_affine=True)\n  )\n  (projector): Linear(in_features=384, out_features=256, bias=True)\n  (classifier): Linear(in_features=256, out_features=1, bias=True)\n)\n```\n\nJust use `num_labels`\n\nwhen defining, eg:\n\n`model = WhisperForAudioClassification.from_pretrained(model_name, num_labels=2)`\n\n[mohamed](https://mohamed.com)\n\nJust use `num_labels`\n\nwhen defining, eg:\n\n`model = WhisperForAudioClassification.from_pretrained(model_name, num_labels=2)`\n\nmodel = WhisperForAudioClassification.from_pretrained(model_name, num_labels=2)\n\nCan someone give me a simple example on how to train Wav2Vec2 for audio frame classificationDDDAAA?Can someone give me a simple example on how to train Wav2Vec2 for audio frame classification?", "url": "https://wpnews.pro/news/choose-number-of-labels-in-whisperforaudioclassification", "canonical_source": "https://discuss.huggingface.co/t/choose-number-of-labels-in-whisperforaudioclassification/58194#post_6", "published_at": "2026-08-31 19:37:43+00:00", "updated_at": "2026-08-31 19:53:11.520943+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models"], "entities": ["Hugging Face", "dpizza", "WhisperForAudioClassification", "openai/whisper-tiny"], "alternates": {"html": "https://wpnews.pro/news/choose-number-of-labels-in-whisperforaudioclassification", "markdown": "https://wpnews.pro/news/choose-number-of-labels-in-whisperforaudioclassification.md", "text": "https://wpnews.pro/news/choose-number-of-labels-in-whisperforaudioclassification.txt", "jsonld": "https://wpnews.pro/news/choose-number-of-labels-in-whisperforaudioclassification.jsonld"}}