Incentive Experiments: NSF Experiences
1. Monetary Incentive Versus Nonmonetary Incentive
One postulated reason for survey nonresponse is that the sampled person lacks motivation to respond to the survey. This lack of motivation can be due to the sampled person's perception of the lack of relevance of the survey or any sense of obligation to respond. Some sampled persons would answer the survey if he/she felt the survey was relevant to them and it was not burdensome to fill out. Others will answer the survey only if some type of incentive is offered. According to the norm of reciprocity (Gouldner 1960), people are more likely to help those who provide a favor to them. The act of providing a favor builds a psychological obligation to reciprocate. Therefore, nonmonetary incentives, such as brochures, flyers, CD containing survey information, etc., are often used to promote the importance of a survey (and therefore increase the importance of a person responding to the survey). Monetary incentives are often used to encourage the sampled person's reciprocal response. To evaluate the effectiveness of both nonmonetary and monetary incentives to SESTAT surveys, NSF conducted an experiment during the 2003 SDR data collection period to assess the impact of nonmonetary and monetary incentives on response rates and data quality. The following is a summary of this study, mainly drawn from Grigorian and Sederstrom (2006); Broach and Grigorian (2005); and Fecso, Broach, and Grigorian (2006).
The Survey of Doctorate Recipients is a longitudinal panel survey of individuals who have received doctoral level degrees from U.S. institutions in SEH fields and resided in the United States at the time of the survey. The SDR has been conducted biennially since 1973. In this survey, participants are followed throughout their careers from the year of award of the doctoral degree through age 75. Every 2 years, a sample of new SEH doctoral degree earners is added to the SDR sample from another NSF survey, the Survey of Earned Doctorates. For the 2003 cycle, a sample of individuals under the age of 76 who earned doctoral degrees in SEH fields from U.S. institutions through academic year 2002 was surveyed. For further details regarding the SDR, refer to the SRS website: http://www.nsf.gov/statistics/srvydoctoratework/.
Data collection for the 2003 SDR began in October 2003 with 40,000 sample members. In May 2004, after 7 months of data collection, 5,764 sample members had not yet responded. In all of these cases, the individuals had received the full contacting treatment protocol, which included up to 4 mail contacts, 3 e-mail contacts, and 20 computer-assisted telephone interview (CATI) attempts. From this reluctant portion of the sample, 5,578 sample members who had no locating problem were included in the 2003 SDR Incentive Study. This experiment included four treatment groups, including a control group, as follows:
Table 1 summarizes the sample sizes, response rates, and data quality measures. The results show that
The 2003 SDR Incentive Study showed not only that the brochure as an incentive might have had a negative impact on data quality, but also that monetary incentives had significant positive effects on response rates and data quality. This latter pattern was observed in another incentive study with much lower levels of money incentives, conducted by the U.S. Census Bureau for NSF: the 2006 NSRCG Prepaid Incentive Experiment. The following is a summary of this study. For detailed information, see Zukerberg (2007) and Zukerberg, Hall, and Henly (2007).
The National Survey of Recent College Graduates provides information about individuals who recently obtained bachelor's or master's degrees in an SEH field from a U.S. institution, were living in the United States during the survey reference week, and were under age 76. The NSRCG sample is a two-stage sample: in the first stage, a sample of institutions is selected; in the second stage, a sample of graduates is selected from lists provided by the sampled institutions. For detailed information about the 2006 NSRCG, please refer to the SRS website: http://www.nsf.gov/statistics/srvyrecentgrads/.
The goal of this study was to increase the response rates during the early mail phase among traditionally low-responding demographic groups, defined as low-responding degree fields. A degree field was considered a low-responding field if its 2003 NSRCG estimated response rate was lower than 68.6%. Nine degree fields were identified accordingly. Five of the fields were collapsed with other fields to form four groups (see table 2). The complete 2006 NSRCG sample consisted of 27,000 cases. After removing 151 cases that could not be contacted, 391 cases that were known foreign cases, and 134 cases with no name or Social Security number, the eligible frame for this study consisted of 26,324 cases. Of these cases, 13,802 were identified as cases in the low-responding degree fields defined in table 2 below. The sampled 2006 NSRCG cases from the low-responding degree fields were allocated to one of the following groups:
It should be noted that only the cases in the low-responding degree fields were included in the incentive study. Cases in the non-low-responding degree fields did not receive any type of prepaid incentive.
The 2006 NSRCG began with a paper questionnaire that was mailed to sampled individuals. Nonrespondents received a follow-up mailing with a replacement questionnaire 5 weeks later; if there was still no response, a CATI follow-up was conducted. The incentives (prepaid gift cards) were mailed along with the first questionnaire to members of the incentive groups.
Table 2 summarizes the incentive group sizes. Table 3 summarizes the response rates of all groups and the results of pairwise comparisons—the data show that monetary incentives as low as $5 can significantly increase response rates. Both tables show results through August 2006, 4 months into the data collection process; afterward, another incentive experiment with a sample that overlaps this study was implemented.