Incentive Experiments: NSF Experiences
2. Prepaid Versus Postpaid
The previous section indicates a prepaid incentive as low as $5 can have a positive impact on the response rate. Does a postpaid incentive work equally as well as a prepaid incentive? If so, the use of a postpaid incentive can avoid the expense of having sampled persons taking the money without responding to the survey. During the 2003 NSRCG, an incentive experiment was conducted to test the effectiveness of different communication strategies and incentive methods. Below is a summary of this study. For more information about this experiment, refer to the 2003 NSRCG methodology report (Wilson et al. 2005); for more information about NSRCG, refer to the SRS website at http://www.nsf.gov/statistics/srvyrecentgrads/.
On April 21, 2004, more than 7 months after the beginning of the 2003 NSRCG data collection process, following two questionnaire mailings, CATI follow-up, and multiple e-mail/postcard reminders, an incentive experiment was conducted to test the effectiveness of various communication strategies (letter versus postcard) and payment methods (prepaid, postpaid, or no payment). For this purpose, 2,000 individuals who had not responded to the survey by April 21, 2004, were randomly assigned to five conditions (table 4).
Nonrespondents to the initial mailing for all groups were sent follow-up reminders by e-mail and by telephone. During the follow-ups, the incentive was mentioned to the members of incentive groups (groups 1, 2, and 4). Table 5below shows the results of the incentive experiment 1 week after the mailing.
Table 5 summarizes the response rates and sample counts for the three letter groups followed by the two postcard groups. The data indicate that there seems to be a communication effect: the postpaid–postcard group 4 seems to have a lower response rate than the postpaid–letter group 1 (5.4% versus 10.2%), and the control–postcard group 5 appears to have a lower response rate than the control–letter group 3 (2.6% versus 6.3%). These results suggest that a postcard may not be an optimal form of communication. Because of this communication effect, the two postpaid groups (postpaid–letter and postpaid–postcard) were not combined to make a comparison with the only prepaid group: the prepaid–letter group. The prepaid–letter group seems to have a similar response rate as the postpaid–letter group (11.5% vs. 10.2%), but a careful examination of the conditions shows the postpaid group had an option of getting a $30 incentive by responding via the Web. Indeed, 37 out of the 38 postpaid–letter group members responded using the Web to get the $30 incentive instead of the $20 CATI incentive. Consequently, the 10.2% response rate of the postpaid–letter group could reflect the option of a higher incentive; in contrast, the prepaid–letter group does not have the $30 Web option—thus, if not for the $10 higher incentive for the Web response option, the postpaid–letter group might have had a lower response rate. This reinforces the earlier similar observation in the 2003 SDR Incentive Study (section 1.1) in which the $50 postpaid group response rate was 33.2% while the $30 prepaid group response rate was 37.7%.
All comparisons in this experiment were not tested because the response rates were based on only 1 week of results in order to make an operational decision about treatment of all remaining nonrespondents. On June 2, 2004, all nonrespondents were mailed a letter offering a postpaid incentive to encourage response: $20 for CATI response or $30 for Web response. This combination of incentives (group 1 in the experiment) was chosen because it yielded the highest proportion of Web responses and ensured that payment would be reserved for actual respondents.