Researchers urged to hone methods for mining social-media data, or investment in marketing will be wasted.

By MediaStreet Staff Writers

A growing number of people, from marketers to academic researchers, are mining social media data to learn about both online and offline human behaviour. In recent years, studies have claimed the ability to predict everything from summer blockbusters to fluctuations in the stock market.

But mounting evidence of flaws in many of these studies points to a need for researchers to be wary of serious pitfalls that arise when working with huge social media data sets. This is according to computer scientists at McGill University and Carnegie Mellon University.

Such erroneous results can have huge implications on data gleaned from social media. A lot of marketing investment could be placed in the wrong areas.

The challenges involved in using data mined from social media include:

  • Different social media platforms attract different users – Pinterest, for example, is dominated by females aged 25-34 – yet researchers rarely correct for the distorted picture these populations can produce.
  • Publicly available data feeds used in social media research don’t always provide an accurate representation of the platform’s overall data – and researchers are generally in the dark about when and how social media providers filter their data streams.
  • The design of social media platforms can dictate how users behave and, therefore, what behaviour can be measured. For instance, on Facebook the absence of a “dislike” button makes negative responses to content harder to detect than positive “likes.”
  • Large numbers of spammers and bots, which masquerade as normal users on social media, get mistakenly incorporated into many measurements and predictions of human behaviour.
  • Researchers often report results for groups of easy-to-classify users, topics, and events, making new methods seem more accurate than they actually are. For instance, efforts to infer political orientation of Twitter users achieve barely 65% accuracy for typical users – even though studies (focusing on politically active users) have claimed 90% accuracy.

Many of these problems have well-known solutions from other fields such as epidemiology, statistics, and machine learning. The common thread in all these issues is the need for researchers to be more acutely aware of what they’re actually analysing when working with social media data.

Social scientists have honed their techniques and standards to deal with this sort of challenge before. Says Derek Ruths, an assistant professor in McGill’s School of Computer Science, “The infamous ‘Dewey Defeats Truman’ headline of 1948 stemmed from telephone surveys that under-sampled Truman supporters in the general population. Rather than permanently discrediting the practice of polling, that glaring error led to today’s more sophisticated techniques, higher standards, and more accurate polls. Now, we’re poised at a similar technological inflection point. By tackling the issues we face, we’ll be able to realise the tremendous potential for good promised by social media-based research.”