nsf.gov - NCSES Incentive Experiments: NSF Experiences - US National Science Foundation (NSF)
text-only page produced automatically by LIFT Text
Transcoder Skip all navigation and go to page contentSkip top navigation and go to directorate navigationSkip top navigation and go to page navigation
National Science Foundation National Center for Science and Engineering Statistics
Incentive Experiments: NSF Experiences

1. Monetary Incentive Versus Nonmonetary Incentive



 

One postulated reason for survey nonresponse is that the sampled person lacks motivation to respond to the survey. This lack of motivation can be due to the sampled person's perception of the lack of relevance of the survey or any sense of obligation to respond. Some sampled persons would answer the survey if he/she felt the survey was relevant to them and it was not burdensome to fill out. Others will answer the survey only if some type of incentive is offered. According to the norm of reciprocity (Gouldner 1960), people are more likely to help those who provide a favor to them. The act of providing a favor builds a psychological obligation to reciprocate. Therefore, nonmonetary incentives, such as brochures, flyers, CD containing survey information, etc., are often used to promote the importance of a survey (and therefore increase the importance of a person responding to the survey). Monetary incentives are often used to encourage the sampled person's reciprocal response. To evaluate the effectiveness of both nonmonetary and monetary incentives to SESTAT surveys, NSF conducted an experiment during the 2003 SDR data collection period to assess the impact of nonmonetary and monetary incentives on response rates and data quality. The following is a summary of this study, mainly drawn from Grigorian and Sederstrom (2006); Broach and Grigorian (2005); and Fecso, Broach, and Grigorian (2006).

Top of page. Back to Top

1.1 2003 SDR Incentive Survey

The Survey of Doctorate Recipients is a longitudinal panel survey of individuals who have received doctoral level degrees from U.S. institutions in SEH fields and resided in the United States at the time of the survey. The SDR has been conducted biennially since 1973. In this survey, participants are followed throughout their careers from the year of award of the doctoral degree through age 75. Every 2 years, a sample of new SEH doctoral degree earners is added to the SDR sample from another NSF survey, the Survey of Earned Doctorates. For the 2003 cycle, a sample of individuals under the age of 76 who earned doctoral degrees in SEH fields from U.S. institutions through academic year 2002 was surveyed. For further details regarding the SDR, refer to the SRS website: http://www.nsf.gov/statistics/srvydoctoratework/.

Data collection for the 2003 SDR began in October 2003 with 40,000 sample members. In May 2004, after 7 months of data collection, 5,764 sample members had not yet responded. In all of these cases, the individuals had received the full contacting treatment protocol, which included up to 4 mail contacts, 3 e-mail contacts, and 20 computer-assisted telephone interview (CATI) attempts. From this reluctant portion of the sample, 5,578 sample members who had no locating problem were included in the 2003 SDR Incentive Study. This experiment included four treatment groups, including a control group, as follows:

  1. $30 prepaid: a $30 personalized check sent via first-class mail with a gaining cooperation letter on NSF stationery. One week later, interviewers contacted sample members by telephone, requesting their participation, and an e-mail message was also sent to the sample members, referencing the $30 check and repeating the gaining cooperation text.
  2. $50 postpaid: a gaining cooperation letter on NSF stationery sent via first-class mail promising a $50 incentive payment for those who completed the survey by telephone or by the experimental online version of the survey. Concurrently with the mailing, interviewers contacted sample members by telephone requesting their participation and offering the $50 incentive, and an e-mail message was sent to the members, promising the $50 check and repeating the gaining cooperation text.
  3. NSF brochure: a gaining cooperation letter on NSF stationery and an informational, four-color, SESTAT brochure with a data CD included sent via first-class mail. Concurrently with the mailing, interviewers contacted sample members by telephone, requesting their participation, and an e-mail message was also sent to the members, repeating the gaining cooperation text.
  4. Control: a gaining cooperation letter on NSF stationery sent via first-class mail. Simultaneously with the mailing, interviewers contacted sample members by telephone, requesting their participation, and an e-mail message was sent to the sample members, repeating the gaining cooperation text.

Table 1Excel table. summarizes the sample sizes, response rates, and data quality measures. The results show that

  • The nonmonetary group (NSF brochure) had a response rate similar to that of the control group. In other words, the information brochure about SESTAT did not establish better rapport than the cooperation letter on NSF stationery.
  • Both monetary incentive groups ($30 prepaid group and $50 postpaid group) had significantly higher response rates than the control group. However, although the $50 postpaid group received a higher amount of money, its response rate (33.2%) is not significantly higher than the $30 prepaid group (37.7%).
  • The nonmonetary group needed significantly more editing and imputation than the control group.
  • Both monetary groups needed significantly less data editing and imputation than the control group.

Top of page. Back to Top

1.2 2006 NSRCG Prepaid Incentive Experiment

The 2003 SDR Incentive Study showed not only that the brochure as an incentive might have had a negative impact on data quality, but also that monetary incentives had significant positive effects on response rates and data quality. This latter pattern was observed in another incentive study with much lower levels of money incentives, conducted by the U.S. Census Bureau for NSF: the 2006 NSRCG Prepaid Incentive Experiment. The following is a summary of this study. For detailed information, see Zukerberg (2007) and Zukerberg, Hall, and Henly (2007).

The National Survey of Recent College Graduates provides information about individuals who recently obtained bachelor's or master's degrees in an SEH field from a U.S. institution, were living in the United States during the survey reference week, and were under age 76. The NSRCG sample is a two-stage sample: in the first stage, a sample of institutions is selected; in the second stage, a sample of graduates is selected from lists provided by the sampled institutions. For detailed information about the 2006 NSRCG, please refer to the SRS website: http://www.nsf.gov/statistics/srvyrecentgrads/.

The goal of this study was to increase the response rates during the early mail phase among traditionally low-responding demographic groups, defined as low-responding degree fields. A degree field was considered a low-responding field if its 2003 NSRCG estimated response rate was lower than 68.6%. Nine degree fields were identified accordingly. Five of the fields were collapsed with other fields to form four groups (see table 2Excel table.). The complete 2006 NSRCG sample consisted of 27,000 cases. After removing 151 cases that could not be contacted, 391 cases that were known foreign cases, and 134 cases with no name or Social Security number, the eligible frame for this study consisted of 26,324 cases. Of these cases, 13,802 were identified as cases in the low-responding degree fields defined in table 2 below. The sampled 2006 NSRCG cases from the low-responding degree fields were allocated to one of the following groups:

  • $10 prepaid incentive
  • $5 prepaid incentive
  • $0 no incentive (control)

It should be noted that only the cases in the low-responding degree fields were included in the incentive study. Cases in the non-low-responding degree fields did not receive any type of prepaid incentive.

The 2006 NSRCG began with a paper questionnaire that was mailed to sampled individuals. Nonrespondents received a follow-up mailing with a replacement questionnaire 5 weeks later; if there was still no response, a CATI follow-up was conducted. The incentives (prepaid gift cards) were mailed along with the first questionnaire to members of the incentive groups.

Table 2Excel table. summarizes the incentive group sizes. Table 3Excel table. summarizes the response rates of all groups and the results of pairwise comparisons—the data show that monetary incentives as low as $5 can significantly increase response rates. Both tables show results through August 2006, 4 months into the data collection process; afterward, another incentive experiment with a sample that overlaps this study was implemented.

 
Incentive Experiments: NSF Experiences
Working Paper | SRS 11-200 | November 2010