The source of statistics: New frontiers in survey methodology

The United Nations Statistical Commission celebrates the fifth annual World Statistics Day today. Over 60 countries have joined together to coordinate activities based on the theme of "Better data. Better lives." The National Science Foundation (NSF) funds multiple research projects aimed at fostering new statistical methods and strengthening statistical systems. This article is one of two published for World Statistics Day highlighting NSF's work. News from the World of Statistics has published a guide listing World Statistics Day outreach efforts by other organizations.

Policymakers around the world rely on statistics as the best evidence to inform their decisions--big and small.

The Federal Reserve and Wall Street firms need data on the unemployment rate, wage growth numbers and jobs. Municipal governments require demographic information in order to decide how to allocate public resources. The health care industry pays close attention to--and shifts resources based on--health insurance coverage and usage rates.

But where do those statistics come from? A surprising amount of the data that influences decisions by policymakers comes from surveys, whether via interviews, questionnaires or other means. Survey methodology therefore has a huge influence on getting accurate data.

In today's rapidly changing technological landscape, the options for reaching people look much different than they did even a few years ago. A group of researchers, supported by NSF's Social, Behavioral and Economics (SBE) directorate, is working to map out this landscape.

Frederick Conrad, professor at the University of Michigan's Institute for Social Research and Michael Schober, professor and associate provost for research at The New School, are cognitive psychologists who specialize in survey methodology. They say the field not only helps improve estimates from surveys, it also offers a fertile, large-sample testing ground for theories from psychology.

In one study, funded by NSF's Methodology, Measurement, and Statistics program, they explored how the widespread use of smartphones alters the survey landscape. Smartphones allow users to respond to a survey by email, text, voice, or through a web browser, a video chat application like Skype, or a specialized survey app. Each of these modes is available in your hand, on the go, and each has implications in terms of the quality and completeness of the data collected.

The first article from this research, "Precision and Disclosure in Text and Voice Interviews on Smartphones," was published in June in the journal PLOS ONE.

"We compared the survey responses in text messaging versus voice interviews and found that text actually had some terrific qualities," Conrad said. "Respondents seemed to invest more effort and be more thoughtful about their responses. They more often responded with exact numbers instead of rounding up or down and they took more time to answer. They also seemed to answer more embarrassing questions more candidly. Texting reduces the time pressure and the respondent doesn't experience the judgment of the interviewer on the other end of the line in the same way."

In another study, Schober added, the researchers found that respondents were more satisfied when they were able to choose the mode of response themselves.

"Survey fatigue and declining response rates are growing problems," he said. "If allowing responses in some of these new modes can help overcome this, that would be a major advance. One way or another, it's very important to better understand the implications and opportunities associated with the changing technology."

When asked about the future of survey research and survey methodology, Schober said that in addition to the work on new response modes, researchers are also interested in finding ways to use big data to collect information on a given topic, as opposed to using surveys that rely on sample populations.

Surveys are costly and burdensome, he said, and the possibility of crunching large amounts of data from sources like social media to augment or even replace survey estimates has generated excitement--and controversy.

"For example, in some studies, sentiment analysis of Twitter by tracking the use of negative vs. positive words in relation to a certain topic (like 'jobs') has been shown to correspond to survey data on consumers' attitudes about the economy," he said.

In other words, the information gathered inexpensively through analyzing tweets may be another way of getting some of the same data that surveys now produce. But, Schober added, "other times, researchers have not found this kind of relationship."

Conrad said big data holds potential, but requires much more study.

"The current survey model may not be sustainable in the long term because of cost and survey fatigue, but right now no one understands enough about when and how we can use new data sources instead of--or in addition to--survey data," he said. "Making progress on this issue should be a high priority. Enthusiasts tend to portray big data analyses as a panacea, sometimes even claiming that we won't even need surveys in the future, but there really isn't yet enough theory or evidence to know why or when social media data or other data sources will produce reliable results."

The strength of the methods behind a survey determines whether it will produce a complete and representative picture of a population. Without good numbers, policies will not address the true needs of society; they will be aiming at the wrong targets.

The theme of this year's World Statistics Day is "Better Data, Better Lives". Perhaps it should be expanded to "Better Surveys, Better Data, Better Lives".