Incentive Experiments: NSF Experiences
4. Timing of the Monetary Incentives
As seen above, a monetary incentive—as little as $5—can have positive effects on response rates and data quality. But when in the data collection process is offering an incentive most effective? Will offering an incentive early in the data collection process result in a higher response rate and better data quality than offering it later in the data collection process? To answer these questions, an incentive study was conducted to the 2006 SDR. The following is a summary of this study. For more details on this study, see Grigorian, Brown, and Jennings (2007).
The 2006 SDR Incentive Experiment was composed of an early-stage prepaid incentive distribution and a late-stage prepaid incentive distribution. The experiment sought to determine the effect of offering an incentive at different times on completion rate, response rate, and survey quality.
In September 2006, 5 months after 2006 SDR data collection began, four experiment groups were selected from 15,592 nonresponse cases from the 42,955 cases in the original sample (table 10):
Sample members assigned to the early incentive and late incentive groups were mailed an envelope containing a gaining cooperation cover letter on SDR letterhead and a $25 check via first-class mail. The early control and late control groups were mailed an envelope containing only a gaining cooperation cover letter on SDR letterhead via first-class mail.
The incentive experiment schedule was the following:September 22, 2006
The early control and early incentive groups received gaining cooperation letters only in September; the late control group and late incentive group received gaining cooperation letters only in October. Tables 11–16 summarize the results of the experiment and are explained below.
The second column of table 12 summarizes the completion rates by October 22, 2006, 1 month after the cooperation letter/incentives were sent to the early control and early incentive groups. As of October 22, neither the late control group nor the late incentive group had received anything for the experiment. By that date, however, 6 months into the data collection process, both late groups had received several mailings for the survey before their portion of the experiment began.
The third column of table 12 summarizes the completion rates by the end of the field period (December 2006). By then, the late control and late incentive groups already had their cooperation letter/incentives for at least 1 month. During this period of time, the early control and early incentive groups did not receive any additional mailings. At the end of the data collection period, the two control groups had similar completion rates, as did the two incentive groups.
Table 11 indicates that although both incentive groups' response rates are statistically significantly higher than their corresponding control groups, at the end of the field period the two incentive groups had similar response rates. Hence, the difference in timing of offering an incentive for these two groups did not result in differences in response rates. Operationally, offering an incentive early may mean offering it to all individuals in the sample (potential respondents and nonrespondents), while offering an incentive late means offering it to fewer sampled persons (potential nonrespondents only). As shown in table 12, however, early incentives also can result in earlier completion, which may reduce the follow-up efforts and associated cost. As such, the potential extra costs for early incentives being offered to many people may be offset by cost savings in follow-up efforts. Further study is needed to determine whether the savings offsets the costs when offering incentives early in the data collection process.
Table 13 indicates the two incentive groups also had similar imputation rates. The late incentive group had a significant lower imputation rate (3.36%) than the late control group (5.23%).
A concern with prepaid incentives is that many people may cash incentive checks without completing the survey. As table 14 shows, this occurred with 3.8% of individuals in the experiment, while 13.5% completed the survey and did not cash their check.
The cooperation rate is the percentage of the respondents in the sample after removing the cases with locating problems. Table 15 indicates that the early incentive and late incentive groups had similar cooperation rates across all sample types and had significantly higher cooperation rates than the corresponding control groups for more than half of the comparison groups.