{"slug": "sentiment-analysis-using-vader", "title": "Sentiment Analysis using VADER", "summary": "A developer demonstrated a sentiment-analysis pipeline that uses NLTK's VADER lexicon and rule-based scorer to classify IMDB movie reviews from a CrowdFlower dataset as positive, negative or neutral. The workflow lowercases text, strips English stopwords, tokenizes sentences, and derives neg, neu, pos and compound scores, then maps compound values to labels using thresholds of 0.05 and -0.05.", "body_md": "Sentiment Analysis is a supervised Machine Learning technique used to determine if a chunk of text is positive, negative or neutral. In text analytics, natural\n\nlanguage processing (NLP) and machine learning (ML) techniques are combined to assign sentiment scores to topics, categories or entities within a phrase.\n\nSentiment analysis is a powerful tool that businesses can leverage to better understand the overall opinions of their customer, gain insights and make\n\ndata-driven decisions.\n\nIn this tutorial we will classify text articles using VADER on a dataset from CrowdFlower\n\n``` python\nimport pandas as pd\nimport nltk\nfrom nltk.corpus import stopwords\nfrom nltk.probability import FreqDist\nfrom nltk.tokenize import sent_tokenize, RegexpTokenizer\nfrom nltk.sentiment.vader import SentimentIntensityAnalyzer\nnltk.download('stopwords')\nnltk.download('vader_lexicon')\nnltk.download('punkt')\n```\n\nFor the sake of this tutorial we’re going to use a simple dataset from CrowdFlower via data.world (IMDB Sentiment Sampled).\n\nFor more info visit the link: | [https://data.world/robbertb/imdb-sentiment-sampled](https://data.world/robbertb/imdb-sentiment-sampled)\n\nThe NLTK module is a massive tool kit, aimed to help you with the entire Natural Language Processing (NLP) methodology.\n\nNLTK will aid you with everything from splitting sentences from paragraphs, splitting up words, recognizing the part of speech of those words.\n\nIn order to apply polarity score we need to prepare your data, we start by the converting our data to lowercase using the built-in function “lower”.\n\n```\ndf['review'] = df['review'].apply(lambda txt: txt.lower())\n```\n\nRemoving stop word using NLTK by downloading the english stopwords.\n\n```\nstop_words=stopwords.words('english')\ndf['review'] = df['review'].apply(lambda txt: ' '.join([word for word in txt.split() if word not in stop_words]))\n```\n\nSentences Tokenization\n\n```\ndf['review'] = df['review'].apply(lambda txt: sent_tokenize(txt))\n```\n\nJoin the tokenized data into text.\n\n```\ndf['review'] = df['review'].apply(lambda txt: ' '.join(txt))\n```\n\nVADER (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed\n\nin social media.\n\nIt is used for sentiment analysis of text which has both the polarities i.e. positive/negative. VADER is also used to quantify how much positive or negative\n\nemotion the text has and also the intensity of emotion.\n\nThe VADER library returns 4 values such as:\n\nNotice that the pos, neu and neg probabilities add up to 1. Also, the compound score is a very useful metric in case we want a single measure of sentiment.\n\nTypical threshold values are the following:\n\nInstantiate a new object with NLTK SentimentIntensityAnalyzer\n\n```\nsid = SentimentIntensityAnalyzer()\n```\n\nNow we can create a new column to the original DataFrame to store the polarity_scores dictionary, the scores extracted will have the keys “neg”, “neu”, “pos”,\n\n“compound” derived from the composite score.\n\n```\ndf['score'] = df['review'].apply(lambda txt: sid.polarity_scores(txt))\ndf['negative'] = df['score'].apply(lambda txt: txt['neg'])\ndf['neutral'] = df['score'].apply(lambda txt: txt['neu'])\ndf['positive'] = df['score'].apply(lambda txt: txt['pos'])\ndf['compound'] = df['score'].apply(lambda txt: txt['compound'])\n```\n\nWe will follow by creating a function called “polarity_score” to calculate the accuracy test for each review in our dataframe. We can finally apply the function\n\nby creating a new column called “sentiment”.\n\nThe reviews in this column will be classified into positive, negative and neutral.\n\n``` python\ndef polarity_score(compound):\n    if compound > 0.05:\n        return \"positive\"\n    elif compound < -0.5:\n        return \"negative\"\n    elif compound >= -0.05 and compound < 0.05:\n        return \"neutral\"\n\ndf['sentiment'] = df['compound'].apply(lambda val: polarity_score(val))\ndf.head()\ndf['sentiment'].value_counts()\n```\n\nVADER classifies the sentiments very well. It is easy to use, the ready-made model which can be used across multiple domains, social-media texts.\n\nThanks for reading and happy web scraping everyone!\n\nYou can find my Jupyter Notebook for this on my Github.\n\nThank you for reading! ❤️", "url": "https://wpnews.pro/news/sentiment-analysis-using-vader", "canonical_source": "https://dev.to/akladyous/sentiment-analysis-using-vader-2la1", "published_at": "2026-11-01 19:08:43+00:00", "updated_at": "2026-11-01 19:22:44.310090+00:00", "lang": "en", "topics": ["natural-language-processing", "machine-learning", "artificial-intelligence"], "entities": ["VADER", "NLTK", "CrowdFlower", "data.world", "IMDB"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/sentiment-analysis-using-vader", "markdown": "https://wpnews.pro/news/sentiment-analysis-using-vader.md", "text": "https://wpnews.pro/news/sentiment-analysis-using-vader.txt", "jsonld": "https://wpnews.pro/news/sentiment-analysis-using-vader.jsonld"}}