nsf.gov - NCSES Incentive Experiments: NSF Experiences - US National Science Foundation (NSF)
text-only page produced automatically by LIFT Text
Transcoder Skip all navigation and go to page contentSkip top navigation and go to directorate navigationSkip top navigation and go to page navigation
National Science Foundation National Center for Science and Engineering Statistics
Incentive Experiments: NSF Experiences

4. Timing of the Monetary Incentives



 

As seen above, a monetary incentive—as little as $5—can have positive effects on response rates and data quality. But when in the data collection process is offering an incentive most effective? Will offering an incentive early in the data collection process result in a higher response rate and better data quality than offering it later in the data collection process? To answer these questions, an incentive study was conducted to the 2006 SDR. The following is a summary of this study. For more details on this study, see Grigorian, Brown, and Jennings (2007).

Top of page. Back to Top

4.1 2006 SDR Incentive Experiment

The 2006 SDR Incentive Experiment was composed of an early-stage prepaid incentive distribution and a late-stage prepaid incentive distribution. The experiment sought to determine the effect of offering an incentive at different times on completion rate, response rate, and survey quality.

In September 2006, 5 months after 2006 SDR data collection began, four experiment groups were selected from 15,592 nonresponse cases from the 42,955 cases in the original sample (table 10Excel table.):

  • Early control group
  • Early incentive group ($25 prepaid check)
  • Late control group
  • Late incentive group ($25 prepaid check)

Sample members assigned to the early incentive and late incentive groups were mailed an envelope containing a gaining cooperation cover letter on SDR letterhead and a $25 check via first-class mail. The early control and late control groups were mailed an envelope containing only a gaining cooperation cover letter on SDR letterhead via first-class mail.

The incentive experiment schedule was the following:

September 22, 2006
  • Gaining cooperation letter with $25 check mailed to early incentive group.
  • Gaining cooperation letter mailed to early control group.
  • Late control group and late incentive group did not receive any mailing at this time.
October 23, 2006
  • Gaining cooperation letter with $25 check mailed to late incentive group.
  • Gaining cooperation letter mailed to late control group.
  • Early control group and early incentive group did not receive mailing at this time.

The early control and early incentive groups received gaining cooperation letters only in September; the late control group and late incentive group received gaining cooperation letters only in October. Tables 11–16 summarize the results of the experiment and are explained below.

The second column of table 12Excel table. summarizes the completion rates by October 22, 2006, 1 month after the cooperation letter/incentives were sent to the early control and early incentive groups. As of October 22, neither the late control group nor the late incentive group had received anything for the experiment. By that date, however, 6 months into the data collection process, both late groups had received several mailings for the survey before their portion of the experiment began.

The third column of table 12Excel table. summarizes the completion rates by the end of the field period (December 2006). By then, the late control and late incentive groups already had their cooperation letter/incentives for at least 1 month. During this period of time, the early control and early incentive groups did not receive any additional mailings. At the end of the data collection period, the two control groups had similar completion rates, as did the two incentive groups.

Table 11Excel table. indicates that although both incentive groups' response rates are statistically significantly higher than their corresponding control groups, at the end of the field period the two incentive groups had similar response rates. Hence, the difference in timing of offering an incentive for these two groups did not result in differences in response rates. Operationally, offering an incentive early may mean offering it to all individuals in the sample (potential respondents and nonrespondents), while offering an incentive late means offering it to fewer sampled persons (potential nonrespondents only). As shown in table 12Excel table., however, early incentives also can result in earlier completion, which may reduce the follow-up efforts and associated cost. As such, the potential extra costs for early incentives being offered to many people may be offset by cost savings in follow-up efforts. Further study is needed to determine whether the savings offsets the costs when offering incentives early in the data collection process.

Table 13Excel table. indicates the two incentive groups also had similar imputation rates. The late incentive group had a significant lower imputation rate (3.36%) than the late control group (5.23%).

A concern with prepaid incentives is that many people may cash incentive checks without completing the survey. As table 14Excel table. shows, this occurred with 3.8% of individuals in the experiment, while 13.5% completed the survey and did not cash their check.

The cooperation rate is the percentage of the respondents in the sample after removing the cases with locating problems. Table 15Excel table. indicates that the early incentive and late incentive groups had similar cooperation rates across all sample types and had significantly higher cooperation rates than the corresponding control groups for more than half of the comparison groups.

 
Incentive Experiments: NSF Experiences
Working Paper | SRS 11-200 | November 2010