A Fast, Effective, and Simple Classification Model for Customer Reviews

Summary

This post presents a new, simple, fast, and effective model for predicting a Yelp review’s star rating from its review text, including an interactive demonstration app and a downloadable R package. After motivating the project I present the theoretical model, describe how I implemented and trained it in the R programming language, and present empirical results. It is approximately 84.4% accurate when tested on a balanced set of roughly 249,000 Yelp reviews (50% positive, 50% negative), and its ROC curve shows good performance. The model is based on a simple logistic regression and runs extremely quickly when vectorised in R: it takes seconds to train on hundreds of thousands of inputs, takes seconds to test on hundreds of thousands of inputs, and takes seconds to download and install.

This work was done as part of an MBA-level directed reading course on Natural Language Processing under the supervision of Dr. Peter Rabinovitch at the University of Ottawa’s Telfer School of Management. Thank you Peter!

We also experimented with other advanced methods like lasso regression and support vector machines following Silge and Hvitfeldt (2020) and Silge and Robinson (2020), but the logistic regression model worked better and was about 1200 times faster. My entire lab notebook is available here.

Interactive Shiny App

Let’s start with the fun stuff: a Shiny app that takes user-provided input text and applies the model to predict a rating. Although this is just a toy example, it shows how you could deploy this kind of model interactively for real-time use.

If the app doesn’t load in the window above, see it in action at this link.

Motivation

An NLP classification model takes a string of text and assigns it to one of two or more classes. The problem is general and classification models have many applications in industry and academia. For example, you could use a classification model to decide whether a document is an invoice, an essay was plagiarised, or a tweet was generated by a bot.

Businesses also have a specific interest in knowing what customers are saying about their products, and more so what customers are feeling about their products. Many companies offer real-time sentiment monitoring for social media and product reviews, and these services can sift through thousands or hundreds of thousands of texts and quickly distill what the “animal spirits” are saying about a company’s products.

As a step in this direction, this project’s goal is to build a model that can accurately predict whether a Yelp review’s star rating is positive or negative from the text of the review. I used a dataset of roughly 5 million reviews that Yelp makes freely available, since using this large set of real-world data would let me build a model with a legitimate business use-case.¹

The Model

This model classifies Yelp reviews as either positive or negative based on their review text. Yelp reviews actually come as integers from 1 to 5, so following common practice I classified reviews with 1 or 2 stars as negative (“NEG”), reviews with 4 or 5 stars as positive (“POS”), and discarded reviews with 3 stars (Liu 2015).

Intuitively, the model simply says that a review is more likely to be favourable if it uses more words with positive sentiment and doesn’t use the words “but” or “not” too much. The model has three pre-processing steps and a logistic function, and chooses a set of coefficients based on the input text’s length.

The first pre-processing step is to calculate the input text’s sentiment. Sentiment analysis is a rich field, but this model uses a simple dictionary-based approach based on assigning scores to positive and negative words. I’ve used a sentiment dictionary called AFINN, which is a list of 2477 words and associated sentiment scores. The word “bad,” for example, has an AFINN score of -3, the word “good” has a score of +3, and a neutral word like “because” has a score of 0. The model uses the mean AFINN score of all words in a review. In my experiments the mean worked better than the sum.

We next count the number of times “but” and “not” appear in the text. AFINN is a “bag-of-words” approach that looks words out of context without considering syntax, so it can’t tell if a word has been negated. For example, the two brief reviews “Good!” and “Not good!” will both get a score of +3, even though the second one is negative. Counting up the number of “but”s and “not”s in the text gives us a rough way of putting this information back into the model. Basically, if a text has lots of high-sentiment words but also lots of instances of the word “not,” it’s probably not a positive review.

For a given text, the recipe is as follows:

• Pre-processing:
  • Calculate the mean AFINN score for the text, i.e. the sum of each word’s AFINN score divided by the number of words, and call this value afinn_mean.
  • Count how many times the words “but” and “not” appear in the text: call this total buts_nots.
  • Count how many words are in the text, and then find which quintile of a specified distribution it falls in. Call this value quintile.
• Predicting:
  • Choose the model coefficients given by quintile.
  • Find the value of the logistic function with the given coefficients and values of afinn_mean and buts_nots.
  • Interpret this function value as the probability that the text is positive.
  • If the probability of being positive is > 0.5, predict the text is positive; otherwise predict it is negative.

If we let \(\beta_n^q\) denote the \(n\)th coefficient for the \(q\)th-quintile model, the equation can be written as:

\[p(\text{text}) = \frac{1}{e^{-(\beta_0^q + \beta_1^q\times\text{afinn_mean} + \beta_2^q\times\text{buts_nots})}}\]

With the coefficients given in the following table:

The quintiles were derived empirically from the training set of approximately 211,000 Yelp reviews, and are given in the following table:

Demonstration

The model is implemented in an R package that’s easy to install and try out. You can get the development package from GitHub with:

# install.packages("devtools")
devtools::install_github("chris31415926535/yelpredict")

Here’s a simple example that runs some test reviews through the model. I’ve written three straightforward reviews, and then one tricky one with lots of negations to try to fool the model.

library(tibble)
library(yelpredict)

review_examples <- tribble(
  ~"review_text", ~"true_rating",
  "This place was awful!", "NEG",
  "The service here was great I loved it it was amazing.", "POS",
  "Meh, it was pretty good I guess but not the best.", "POS",
  "Not bad, not bad at all, really the opposite of terrible. I liked it.", "POS"
)

review_examples %>%
  prepare_yelp(review_text) %>%
  get_prob() %>%
  predict_rating() %>%
  knitr::kable(col.names= c("Review Text", "True Rating", "AFINN Mean", "# Buts/Nots", "Qtile", "Log Odds", "Prob. POS", "Prediction"),
               caption = "Sample review texts with true ratings, plus additional columns created by the model: The text's mean AFINN sentiment score, the number of buts/nots in the text, the text's word-length quantile, the log-odds and probability that the text is positive, and a final prediction. ")

The first function, prepare_yelp(), takes the name of the text column and gives back a new tibble with additional columns for each review’s mean AFINN score, the number of buts/nots, and its word-length quintile. The function get_prob() takes this augmented tibble and calculates the log-odds and probability that each review is positive. Then predict_rating() turns these probabilities into predictions of either “POS” or “NEG”.

Finally, the function get_accuracy() takes the column of true ratings as an input, and will boil our final results down into one number representing the model’s predictive accuracy on the input data:

review_examples %>%
  prepare_yelp(review_text) %>%
  get_prob() %>%
  predict_rating() %>%
  get_accuracy(true_rating) %>%
  knitr::kable(col.names = "Accuracy on Sample Reviews")

Here we can see that the model was 75% accurate, because it got the wrong answer for the negative review that used a lot of positive words with negators (e.g. “Not bad at all”) that AFINN really can’t handle.

Note that some of these functions have optional parameters described in the package documentation.

Empirical Results

yelp_test$rating_factor %>% summary() %>%
  knitr::kable(col.names = "n",
               caption = "Distribution of negative and positive reviews in test data (n=249,053).")

test_accuracy <- yelp_test %>%
  prepare_yelp(text, qtiles) %>%
  get_prob()  %>%
  predict_rating() %>%
  get_accuracy(rating_factor)

test_accuracy %>%
  knitr::kable()

Reflections on Approaches to Machine Learning

This model shows that you can get great results from combining a simple and supervised approach with domain-specific knowledge. Logistic regression probably isn’t even in most people’s top 5 approaches to machine learning, but in my experiments it ran circles around more complicated and general methods like lasso regression and support vector machine classification.

The key to all this is that we knew something about reviews–good ones tend to use positive words–and used that as a starting point for experimentation. Through trial and error we learned that longer and shorter reviews are different, so we augmented the model to treat reviews differently based on how long they are. Then through some qualitative analysis we noticed that reviews with lots of negations were being misclassified, so we worked negations into the model. At each step I was able to find some new information and work it into the model to increase its accuracy.

This piecewise tinkering and improvement was only possible because logistic regression is fast. It takes a few seconds to train it on hundreds of thousands of inputs, and then a few seconds more to cross-validate it and see how it does. In contrast, testing a support vector machine on text tokens would often take over half an hour. This is a very long time to wait, especially when you’re just learning: there’s nothing like letting a training cycle run overnight only to find that it crashed halfway through because you set it up wrong. It’s not an exaggeration to say that using logistic regression let me speed up my design iterations by a factor of 1000.

To be clear, I’m not claiming that logistic regression will work in all use cases, or that other methods have no value. On the contrary, other methods like lasso regression, support vector machines, or deep-learning methods have their advantages, especially in cases where we may not have domain-specific knowledge we can bring to bear on the problem. I’m as interested and excited about these methods as anyone else, and there’ll be a blog post about them soon enough!

But this project makes it clear that if you’re looking for results, sometimes simpler is better.

Next Steps

It could be promising to extend this model using a more sophisticated sentiment-analysis algorithm. AFINN is extremely simple and doesn’t include any syntax; as I mentioned above, it can’t tell the difference between “good” and “not good.” A more complex way of measuring sentiment might therefore give even better results. However, at present I’m not aware of any better sentiment-detection algorithms that run in R at reasonable speeds.

It could also be interesting to train this model on different datasets, to see whether this approach is capturing something general about online behaviour or something specific about Yelp reviews.

Conclusions

In this blog post I’ve presented a fast and effective model for predicting Yelp ratings from text reviews. The model is 84.4% effective on a large test set, and its ROC curve shows good classification abilities. I also presented an implementation of this model in the R programming language, including an interactive Shiny app and a user-friendly package that can be downloaded and installed from GitHub. In addition, the source code is all up on GitHub and can be inspected or modified.

Resources

• Package code on GitHub: https://github.com/chris31415926535/yelpredict
• Full (unpolished) lab notebook for MBA6292: https://chris31415926535.github.io/mba6292/
• Yelp review dataset: https://www.kaggle.com/yelp-dataset/yelp-dataset
• Supervised Machine Learning for Text Analysis in R: https://smltar.com/