The FY 2016 NPRA Survey was a multi-year project that began in 2013 with the development of the pilot survey and involved extensive input from and testing with the nonprofit community throughout the pilot test development and preparation for the FY 2016 survey. Given the challenge of surveying a population in which the majority of those surveyed did not have R&D activity and thus did not see the survey as salient, this effort had many successes that can be built upon for future surveys.
One of the major successes was the questionnaire content. Due to the development work, both the screener and the questionnaire were extremely well-received by respondents and no significant cognitive issues arose during the data collection period or the debriefing interviews. Respondents reported that the definitions and examples provided for R&D were exceptionally clear. There was also no significant item nonresponse because only questions which had been thoroughly cognitively tested were placed on the final questionnaire.
The key issues encountered for FY 2016 were in the areas of sample design and organization contact. The major sample design issue was the amount of unanticipated organizational duplication among sampling units, particularly for the hospital sector. This led to ad hoc consolidation and deactivation of units during data collection, which complicated the eventual weighting and variance estimation methodology.
Finding updated contact information for the organizations also proved more difficult than anticipated, as was reaching the correct offices even when contact information was available. Finally, the reluctance of many organizations to respond to any survey, given the present environment of declining response rates, was and will remain a major challenge.
Each of these issues contributed to a lower than desired response rate and nonresponse bias, which led to increased pressure on the final weighting methodology to deliver valid results. The next survey will need to achieve a higher response rate and have lower nonresponse bias to meet the NCSES criteria for official statistics.
To mitigate these issues for the next fielding of a nonprofit R&D survey, significant resources should be invested at the sampling stage to establish a clean, unduplicated sample with identified points of contact at each organization. These points of contact should be screened in advance to ensure they will be able to answer questions about their organization's R&D activity and to appoint a backup point of contact in case of staff turnover before fielding. To encourage response from the organizations who do not find the survey salient, more work should be done to find and incorporate topics that would be salient into the survey to incentivize response from those with no R&D activity.