A large number of reviews is not a reliable indicator of a product’s quality.

By MediaStreet Staff Writers

When we’re trying to decide which mobile phone case to buy or which hotel room to book, we often rely on the ratings and reviews of others to help us choose. But new research suggests that we tend to use this information in ways that can actually work to our disadvantage.

The findings, published in Psychological Science, indicate that people tend to favour a product that has more reviews, even when it has the same low rating as an alternative product.

“It’s extremely common for websites and apps to display the average score of a product along with the number of reviews. Our research suggests that, in some cases, people might take this information and make systematically bad decisions with it,” says researcher Derek Powell of Stanford University, lead author on the study.

“We found that people were biased toward choosing to purchase more popular products and that this sometimes led them to make very poor decisions,” he explains.

As opportunities to buy products and services online multiply, we have greater access than ever before to huge amounts of first-hand information about users’ experiences.

“We wanted to examine how people use this wealth of information when they make decisions, and specifically how they weigh information about other people’s decisions with information about the outcomes of those decisions,” says Powell.

Looking at actual products available on Amazon.com, Powell and colleagues Jingqi Yu (Indiana University Bloomington), Melissa DeWolf and Keith Holyoak (University of California, Los Angeles) found no relationship between the number of reviews a product had and its average rating. In other words, real-world data show that a large number of reviews is not a reliable indicator of a product’s quality.

With this in mind, the researchers wanted to see how people would actually use review and rating information when choosing a product. In one online experiment, 132 adult participants looked at a series of phone cases, presented in pairs. The participants saw an average user rating and total number of reviews for each phone case and indicated which case in each pair they would buy.

Across various combinations of average rating and number of reviews, participants routinely chose the option with more reviews. This bias was so strong that they often favoured the more-reviewed phone case even when both of the options had low ratings, effectively choosing the product that was, in statistical terms, more likely to be low quality.

A second online experiment that followed the same design and procedure produced similar results.

“By examining a large dataset of reviews from Amazon.com, we were able to build a statistical model of how people should choose products. We found that, faced with a choice between two low-scoring products, one with many reviews and one with few, the statistics say we should actually go for the product with few reviews, since there’s more of a chance it’s not really so bad,” Powell explains. “But participants in our studies did just the opposite: They went for the more popular product, despite the fact that they should’ve been even more certain it was of low quality.”

The researchers found that this pattern of results fit closely with a statistical model based on social inference. That is, people seem to use the number of reviews as shorthand for a product’s popularity, independent of the product’s average rating.

According to Powell, these findings have direct implications for both retailers and consumers:

“Consumers try to use information about other people’s experiences to make good choices, and retailers have an incentive to steer consumers toward products they will be satisfied with,” he says. “Our data suggest that retailers might need to rethink how reviews are presented and consumers might need to do more to educate themselves about how to use reviews to guide their choices.”