Incentive Experiments: NSF Experiences
6. Slippery Slope?
Two of the SESTAT surveys, SDR and NSCG, are panel surveys—most sampled persons in these two surveys have responded to multiple waves of the surveys. Will a panel member who received a monetary incentive in a previous survey round develop an expectation of a monetary incentive for participating in the next round of the survey? Accordingly, will a panel member who received an incentive in the last round be less likely to respond to the survey if he or she does not receive an incentive in the next round? These are the questions that need to be answered before applying incentives to members of panel surveys.
Using existing information from SESTAT surveys, Michael White of the U.S. Census Bureau performed an ad hoc analysis of the possible slippery-slope effect on panel members in 2008 (White 2008). The following is a summary of that report.
The 2006 NSCG sampling frame consisted of the following:
During the 2003 NSRCG, an incentive was offered to all nonrespondents toward the end of data collection (see section 2.1). The 2006 NSCG followed a sample of respondents to the 2003 NSRCG; as such, some of the 2006 NSCG sampled persons had received an incentive in the 2003 NSRCG data collection. During the 2006 NSCG, an incentive experiment also was conducted (see section 3.2). Therefore, it is possible to identify four groups of sampled persons from which to assess the possible effect of expectation of an incentive:
Table 17 summarizes the number of cases and response rate of each of the four groups.
The analysis indicates there may be an expectation effect of incentives. This can be seen in various ways, e.g., Group 1 versus Group 2, Group 2 versus Group 4, etc. But it should be emphasized that the experiment was not controlled—the 2006 treatment group was not a simple random sample of the 2003 treatment sample (only potential and final nonrespondents received incentives in 2006). Consequently, no statistical test or comparison was attempted. The data strongly indicate, however, that a controlled experiment should be conducted to further assess the expectation effect.