Sentiment Analysis is a supervised Machine Learning technique used to determine if a chunk of text is positive, negative or neutral. In text analytics, natural
language processing (NLP) and machine learning (ML) techniques are combined to assign sentiment scores to topics, categories or entities within a phrase.
Sentiment analysis is a powerful tool that businesses can leverage to better understand the overall opinions of their customer, gain insights and make
data-driven decisions.
In this tutorial we will classify text articles using VADER on a dataset from CrowdFlower
import pandas as pd
import nltk
from nltk.corpus import stopwords
from nltk.probability import FreqDist
from nltk.tokenize import sent_tokenize, RegexpTokenizer
from nltk.sentiment.vader import SentimentIntensityAnalyzer
nltk.download('stopwords')
nltk.download('vader_lexicon')
nltk.download('punkt')
For the sake of this tutorial we’re going to use a simple dataset from CrowdFlower via data.world (IMDB Sentiment Sampled).
For more info visit the link: | https://data.world/robbertb/imdb-sentiment-sampled
The NLTK module is a massive tool kit, aimed to help you with the entire Natural Language Processing (NLP) methodology.
NLTK will aid you with everything from splitting sentences from paragraphs, splitting up words, recognizing the part of speech of those words.
In order to apply polarity score we need to prepare your data, we start by the converting our data to lowercase using the built-in function “lower”.
df['review'] = df['review'].apply(lambda txt: txt.lower())
Removing stop word using NLTK by down the english stopwords.
stop_words=stopwords.words('english')
df['review'] = df['review'].apply(lambda txt: ' '.join([word for word in txt.split() if word not in stop_words]))
Sentences Tokenization
df['review'] = df['review'].apply(lambda txt: sent_tokenize(txt))
Join the tokenized data into text.
df['review'] = df['review'].apply(lambda txt: ' '.join(txt))
VADER (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed
in social media.
It is used for sentiment analysis of text which has both the polarities i.e. positive/negative. VADER is also used to quantify how much positive or negative
emotion the text has and also the intensity of emotion.
The VADER library returns 4 values such as:
Notice that the pos, neu and neg probabilities add up to 1. Also, the compound score is a very useful metric in case we want a single measure of sentiment.
Typical threshold values are the following:
Instantiate a new object with NLTK SentimentIntensityAnalyzer
sid = SentimentIntensityAnalyzer()
Now we can create a new column to the original DataFrame to store the polarity_scores dictionary, the scores extracted will have the keys “neg”, “neu”, “pos”,
“compound” derived from the composite score.
df['score'] = df['review'].apply(lambda txt: sid.polarity_scores(txt))
df['negative'] = df['score'].apply(lambda txt: txt['neg'])
df['neutral'] = df['score'].apply(lambda txt: txt['neu'])
df['positive'] = df['score'].apply(lambda txt: txt['pos'])
df['compound'] = df['score'].apply(lambda txt: txt['compound'])
We will follow by creating a function called “polarity_score” to calculate the accuracy test for each review in our dataframe. We can finally apply the function
by creating a new column called “sentiment”.
The reviews in this column will be classified into positive, negative and neutral.
def polarity_score(compound):
if compound > 0.05:
return "positive"
elif compound < -0.5:
return "negative"
elif compound >= -0.05 and compound < 0.05:
return "neutral"
df['sentiment'] = df['compound'].apply(lambda val: polarity_score(val))
df.head()
df['sentiment'].value_counts()
VADER classifies the sentiments very well. It is easy to use, the ready-made model which can be used across multiple domains, social-media texts.
Thanks for reading and happy web scraping everyone!
You can find my Jupyter Notebook for this on my Github.
Thank you for reading! ❤️