SENTIMENT ANALYSIS ON IMDB

INTRODUCTION:

Sentiment analysis is the process of using natural language processing, text analysis, and statistics to analyze customer sentiment. The best businesses understand the sentiment of their customers—what people are saying, how they’re saying it, and what they mean. Customer sentiment can be found in tweets, comments, reviews, or other places where people mention your brand. Sentiment Analysis is the domain of understanding these emotions with software, and it’s a must-understand for developers and business leaders in a modern workplace.As with many other fields, advances in deep learning have brought sentiment analysis into the foreground of cutting-edge algorithms. Today we use natural language processing, statistics, and text analysis to extract, and identify the sentiment of words into positive, negative, or neutral categories.

ABSTRACT :

This project is to analyze the sentiment given by a person  who are watching movies. For this we need a dataset which is provided by kaggle or http://ai.stanford.edu/~amaas/data/sentiment/.
This datasets contains 50,000 movie reviews that have been pre-labeled with “positive” and “negative” or 0 or 1 sentiment class labels based on the review content.Negative reviews have scores less or equal than 4 out of 10 while a positive review have score greater or equal than 7 out of 10. Neutral reviews are not included. The 50,000 reviews are divided evenly into the training and test set. Besides this, there are additional movie reviews that are unlabeled. We will only be using the raw labeled movie reviews for our analyses.

DATA PREPROCESSING:

Following steps were done for text preprocessing:

• Remove punctuation
• Tokenize sentence
• Remove stopwords
• Lemmatize words
• Calculate TFIDF vectorizer
• Train ML models

        LEMMATIZE WORDS:

         Lemmatization change words based on the dictionary from different algorithms, such as "went" to "go". Based on the different type of the word (verb, noun), it can change to different meaning of word which solve the disambiguation problem. While it demands more computational power. (It can be used if you want to build a dictionary world: NLP system)                    

           TRAIN ML MODELS:

                In this project we have used two patterns to test it:

        1) Supervised Learning

        2) Unsupervised Learning

          The models used in Supervised learning are:

1. Logistic Regression

2. Stochastic gradient descent

3.Random Forest Classifier

4.Ada Boost Classifier.

          The models used in Unsupervised Learning are Lexicon based:

1. AFINN Lexicon

2. VADER Lexicon

COMPARING THE MODELS:

• Logistic Regression

    

           

• Stochastic gradient descent

• Random Forest Classifier

• Ada Boost Classifier

• AFINN Lexicon

• VADER Lexicon

        CONCLUSION:

We can observe that both Logistic Regression and SGDClassifier are performing well compared to other classifiers.