{"slug": "sentiment-analysis-harder-than-it-looks", "title": "Sentiment Analysis: Harder Than It Looks", "summary": "A developer's analysis reveals that sentiment analysis is harder than it appears, with challenges stemming from underspecified labels, structural issues like negation and sarcasm, and domain-specific vocabulary. The post highlights that simple models like bag-of-words fail on negation, while rule-based systems like VADER offer practical solutions for high-volume social text, and contextual models like transformers handle complex cases better. It emphasizes that human annotators often need context to detect irony, making it a separate task, and warns that public dataset accuracy does not transfer to real-world applications.", "body_md": "“Sentiment analysis” sounds like a solved classification problem, and the demos support that impression. The difficulty is not in the model. It is that the label is underspecified, the phenomena that break it are structural, and the accuracy you see on a public dataset does not transfer to your inbox.\n\nBefore choosing a method, notice which of these you actually want, because they have different labels and different ceilings.\n\nMost disappointment comes from picking a tool built for the first while wanting the second or third.\n\nA bag-of-words model sees *the room was not clean* as the terms `room`\n\n, `not`\n\n, `clean`\n\n— and if your pipeline removed stop words, just `room`\n\nand `clean`\n\n. The strongest positive feature in the sentence survives; the word that reverses it does not. This is the most consequential interaction in the whole preprocessing debate and it is why [the standard stop list](https://multigrid.ai/learn/stop-words) containing `not`\n\nis not a trivia point.\n\nThree fixes exist, in ascending order of cost. Bigrams capture `not_clean`\n\nas one feature and handle the common cases — Wang and Manning’s *Baselines and Bigrams* (ACL 2012) is the reference for how strong a simple bigram model with Naive Bayes features remains on this task. Explicit negation scoping tags every token between a negator and the next punctuation, which VADER (Hutto and Gilbert, ICWSM 2014) does as part of a rule-based system alongside intensifiers, capitalisation and emoji. And a contextual model reads the sentence, which is what makes transformers straightforwardly better here.\n\nNote what the rule-based option buys: VADER is a lexicon plus a handful of grammatical rules, runs in microseconds with no model to load, and handles the specific constructions of short social text it was built for. On tweets and chat messages at high volume it is a completely reasonable production choice, and it is free.\n\n“The model cannot detect sarcasm” is usually stated as a modelling limitation. The research suggests it is upstream of that. Wallace et al., *Humans Require Context to Infer Ironic Intent* (ACL 2014), found that human annotators frequently could not identify irony from the text alone and requested surrounding context before they would commit — and that the cases annotators needed context for were the same cases classifiers failed on.\n\nSemEval-2018 Task 3 made irony detection its own shared task, which is itself the point: it is hard enough to warrant separate treatment rather than being folded into polarity. The practical consequence for a product is not “buy a better model”. It is that if your annotators disagree on a class of examples, no model will be reliable on that class, and your evaluation set cannot measure it either. Measure inter-annotator agreement before you measure a model.\n\nSentiment vocabulary is domain-specific in ways that look absurd once you see them. *Unpredictable* is praise for a plot and a defect in a car. *Small* is positive for a phone and negative for a hotel room. *Sick* and *insane* invert in some registers. A classifier fitted on movie reviews carries all of these the wrong way, and the domain-adaptation literature for sentiment goes back to Blitzer, Dredze and Pereira’s multi-domain Amazon review work (ACL 2007) for exactly this reason.\n\nThis is also why the accuracy number on a public leaderboard is close to useless as a prediction for you, and why an off-the-shelf sentiment API is a bigger gamble than it looks: you are inheriting someone else’s domain assumptions with no way to inspect them.\n\nNobody can tell you what accuracy you will get, so here is the cheapest honest way to find out. It takes an afternoon and it is worth more than any benchmark.", "url": "https://wpnews.pro/news/sentiment-analysis-harder-than-it-looks", "canonical_source": "https://dev.to/multigrid/sentiment-analysis-harder-than-it-looks-2p25", "published_at": "2026-08-12 17:27:43+00:00", "updated_at": "2026-08-12 17:47:12.256105+00:00", "lang": "en", "topics": ["natural-language-processing", "machine-learning", "artificial-intelligence"], "entities": ["VADER", "Wang and Manning", "Hutto and Gilbert", "Wallace et al.", "Blitzer", "Dredze", "Pereira", "SemEval-2018"], "alternates": {"html": "https://wpnews.pro/news/sentiment-analysis-harder-than-it-looks", "markdown": "https://wpnews.pro/news/sentiment-analysis-harder-than-it-looks.md", "text": "https://wpnews.pro/news/sentiment-analysis-harder-than-it-looks.txt", "jsonld": "https://wpnews.pro/news/sentiment-analysis-harder-than-it-looks.jsonld"}}