Sentiment Analysis using VADER

A developer demonstrated a sentiment-analysis pipeline that uses NLTK's VADER lexicon and rule-based scorer to classify IMDB movie reviews from a CrowdFlower dataset as positive, negative or neutral. The workflow lowercases text, strips English stopwords, tokenizes sentences, and derives neg, neu, pos and compound scores, then maps compound values to labels using thresholds of 0.05 and -0.05.

Sentiment Analysis is a supervised Machine Learning technique used to determine if a chunk of text is positive, negative or neutral. In text analytics, natural language processing NLP and machine learning ML techniques are combined to assign sentiment scores to topics, categories or entities within a phrase. Sentiment analysis is a powerful tool that businesses can leverage to better understand the overall opinions of their customer, gain insights and make data-driven decisions. In this tutorial we will classify text articles using VADER on a dataset from CrowdFlower python import pandas as pd import nltk from nltk.corpus import stopwords from nltk.probability import FreqDist from nltk.tokenize import sent tokenize, RegexpTokenizer from nltk.sentiment.vader import SentimentIntensityAnalyzer nltk.download 'stopwords' nltk.download 'vader lexicon' nltk.download 'punkt' For the sake of this tutorial we’re going to use a simple dataset from CrowdFlower via data.world IMDB Sentiment Sampled . For more info visit the link: | https://data.world/robbertb/imdb-sentiment-sampled https://data.world/robbertb/imdb-sentiment-sampled The NLTK module is a massive tool kit, aimed to help you with the entire Natural Language Processing NLP methodology. NLTK will aid you with everything from splitting sentences from paragraphs, splitting up words, recognizing the part of speech of those words. In order to apply polarity score we need to prepare your data, we start by the converting our data to lowercase using the built-in function “lower”. df 'review' = df 'review' .apply lambda txt: txt.lower Removing stop word using NLTK by downloading the english stopwords. stop words=stopwords.words 'english' df 'review' = df 'review' .apply lambda txt: ' '.join word for word in txt.split if word not in stop words Sentences Tokenization df 'review' = df 'review' .apply lambda txt: sent tokenize txt Join the tokenized data into text. df 'review' = df 'review' .apply lambda txt: ' '.join txt VADER Valence Aware Dictionary and sEntiment Reasoner is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media. It is used for sentiment analysis of text which has both the polarities i.e. positive/negative. VADER is also used to quantify how much positive or negative emotion the text has and also the intensity of emotion. The VADER library returns 4 values such as: Notice that the pos, neu and neg probabilities add up to 1. Also, the compound score is a very useful metric in case we want a single measure of sentiment. Typical threshold values are the following: Instantiate a new object with NLTK SentimentIntensityAnalyzer sid = SentimentIntensityAnalyzer Now we can create a new column to the original DataFrame to store the polarity scores dictionary, the scores extracted will have the keys “neg”, “neu”, “pos”, “compound” derived from the composite score. df 'score' = df 'review' .apply lambda txt: sid.polarity scores txt df 'negative' = df 'score' .apply lambda txt: txt 'neg' df 'neutral' = df 'score' .apply lambda txt: txt 'neu' df 'positive' = df 'score' .apply lambda txt: txt 'pos' df 'compound' = df 'score' .apply lambda txt: txt 'compound' We will follow by creating a function called “polarity score” to calculate the accuracy test for each review in our dataframe. We can finally apply the function by creating a new column called “sentiment”. The reviews in this column will be classified into positive, negative and neutral. python def polarity score compound : if compound 0.05: return "positive" elif compound < -0.5: return "negative" elif compound = -0.05 and compound < 0.05: return "neutral" df 'sentiment' = df 'compound' .apply lambda val: polarity score val df.head df 'sentiment' .value counts VADER classifies the sentiments very well. It is easy to use, the ready-made model which can be used across multiple domains, social-media texts. Thanks for reading and happy web scraping everyone You can find my Jupyter Notebook for this on my Github. Thank you for reading ❤️