The other day, I hovered over a movie on Netflix and was trying to decide whether to watch it. So I quickly checked its rating on Rotten Tomatoes to see if it was worth my time. Certified fresh or for the trash bin?
Sure, I could have given the movie a few minutes and made my own decision. But in this case I relied on the wisdom of crowds.
As a consumer, do you believe in such ratings? Many of us rely on them when deciding on whether to purchase or engage with all sorts of goods, services and information. But rating systems, and the comments left, may not communicate as much or as powerfully as we imagine.
Among the experts who study rating systems, here are some critiques. Call them their ratings of the ratings:
- A thumb up or thumb down may seem like a clean way to measure people’s feelings. After all, it’s a simple binary choice. Then again, maybe that’s the problem. It doesn’t allow for nuance. Making things quicker and easier does increase rating activity, but what does an all-or-nothing system really mean?
- A three-star system – reflecting positive, neutral and negative – isn’t much better. What’s considered positive? What’s considered negative? Again, no granularity.
- A four-star system receives a lot of hostility, believe it or not, mainly because it leaves no room for neutrality.
- A five-star system gets the most praise. A scale that correlates roughly to terrible, bad, neutral, good and great offers more shades of meaning. Go beyond five choices, and it becomes harder to land on a number with any precision.
So is the five-star rating system a better way to size things up? Not necessarily.
Aggregate numbers alone don’t tell the whole story or sway us. Nor, even, do comments – unless they are tinged with emotion.
That was a conclusion of a new study from Northwestern University’s Kellogg School of Management and the University of Massachusetts Boston. Researchers discovered that they could predict the success of various offerings (movies, books and restaurants) by relying on the emotionality of reviews, rather than the star rating.
The researchers looked at box office revenue of 2,400 movies, sales of 1.6 million books and reservations at more than 1,000 restaurants. What were the rating like and how did this correlate to success?
Overall, higher star ratings were either a weaker signal of success, or were negatively related to success.
It seems counterintuitive, yet the researchers noted that user ratings pose a positivity problem.
“Most reviews are positive. In this sea of positivity, how do people distinguish products?” said study co-author Derek Rucker, a Professor of Entrepreneurial Studies in Marketing at Kellogg.
In fact, the study noted that the average star rating on Amazon is 4.2 out of 5 (a majority of reviews are five star), almost half of Yelp reviews are five-star ratings, and nearly 90% of Uber ratings are five stars.
If everything is above average, what does average (or good) even mean anymore?
The study found that what users really respond to is the language that other consumers use in their feedback. The researchers used computational linguistics to evaluate whether the reviewer had an emotional reaction.
Take two Yelp reviews. “This restaurant is excellent, dinner was flawless.” Or, “I love this restaurant, it is absolutely wonderful.” Both are highly positive, but the second is considered more emotional.
As the researchers explain, emotional responses are better predicters of behaviours and marketplace success.
Which shouldn’t really be surprising. We respond to words more than numbers, and to emotion more than logic. So I may dig deeper next time I check user ratings. In the meantime, feel free to give this blog a five-star review.
Stuart Foxman is a Toronto-based freelance writer, who helps clients’ products, services, ideas and organizations to come alive. Follow me on Twitter @StuartFoxman, connect with me here on LinkedIn, or check me out at foxmancommunications.com. I would love to hear from you. More articles like this coming, with original posts every week about communications, information, motivation, writing, branding, creativity, media, marketing, persuasion, messages, learning, etc.
April 21, 2021