Long Live Fine-Tuning: Task-Specific Transformers Outperform Zero-Shot LLMs for Misinformation Response Classification on Reddit

Fine-tuned RoBERTa achieved a 0.62 macro-F1 score for classifying misinformation responses on Reddit, outperforming the best zero-shot large language model (Claude Haiku 4.5) at 0.50 while costing a fraction per query. The supervised advantage concentrated on detecting belief comments—the implicit category every zero-shot model under-detected—and scaling model size did not improve zero-shot performance, with Claude Sonnet 4.6 collapsing belief detection to 0.17 due to safety-alignment artifacts. The findings demonstrate that task-specific fine-tuning remains more reliable than zero-shot LLMs for misinformation response classification, particularly when missing belief comments is the costlier error.

arXiv:2606.04274v1 Announce Type: new Abstract: As large language models LLMs become default tools for online information verification, an implicit assumption follows them: that scale and general capability are sufficient for nuanced classification of misinformation discourse. We test this assumption directly on 900 Reddit comments spanning three PolitiFact-verified misinformation claims environment, health, immigration , labelled as belief propagates the claim , fact-check corrects it , or other. We compare nine models across three paradigms -- BART-MNLI, three Llama variants, three commercial frontier LLMs Claude Haiku 4.5, Gemini Flash Lite 2.5, Claude Sonnet 4.6 , and fine-tuned DistilBERT and RoBERTa -- under universal and topic-specific label schemas. The assumption does not hold. Fine-tuned RoBERTa reaches 0.62 macro-$F 1$ against a best zero-shot result of 0.50 Claude Haiku 4.5 , at a fraction of the per-query cost; the supervised advantage is concentrated on the belief class, the implicit, affective category every zero-shot model under-detects. Scaling does not help: Llama-3-8B matches Llama-3-70B, and Claude Sonnet 4.6 underperforms the smaller Haiku under generic labels, collapsing belief detection to 0.17 and refusing outright on a subset of comments flagged as sensitive. This is a safety-alignment artefact, not a capacity limit. Label schema and topic jointly shape zero-shot performance, with the same model varying by more than 0.13 macro-$F 1$ across topics under matched labels. In a verification context, where missing belief is the costlier error, task-specific fine-tuning remains the more reliable choice despite the proliferation of large generative models.