Appendix C. Expert Panel Meeting Summary
An expert panel meeting was held at NSF on 5 December 2000, to discuss sampling design issues for SESTAT. (An agenda of the meeting is given in appendix A.) Most of the experts invited were from other federal agencies and were either familiar with the potential data sources or were experts in sampling methods.
Panel participants consisted of eight senior staff from the U.S. Census Bureau, three staff members from BLS, and one each from the National Center for Education Statistics, the Office of Management and Budget, the Energy Information Agency, NCHS, INS, and the Federal Reserve Board. An expert on similar survey efforts at Statistics Canada also attended. Presentations were made to this group by some of the many individuals at NSF or who are under contract to NSF who have studied SESTAT sampling issues. Among the speakers were Westat representatives, who presented the content of this report in some detail.
Expert panelists were asked to comment on the analysis and conclusions made by the researchers at NSF and in this report and to offer suggestions for future actions regarding NSF sampling activities. Some experts responded at the meeting or sent a short response after the meeting. The experts gave the issues substantial consideration and asked many questions, but in the end, no one suggested any better solution or had any better recommendations for approaching these challenges than those already proposed.
At the conclusion of the expert panel meeting, the co-chairs, Ron Fecso and Graham Kalton, offered the following summary of the expert panel's deliberations and recommendations for action:
- How well could other, existing large-scale surveys work for NSF's scientist, engineer, and technician population sampling needs?
(1) Establishment surveys probably would not satisfy the SESTAT data needs for several reasons. First, the problem of the employer as an intermediary screener may be difficult to overcome. Second, there is the non-representativeness of the population that is employed, and particularly the underrepresentation of the population employed in establishments above a minimum size, if the establishment survey has a minimum size cut-off.
(2) The sample size for the Census Bureau's SIPP is too small. Also, the SIPP would be difficult to use because of its complicated, nested rotating panel structure and the dedicated thematic foci of each wave.
(3) The small sample sizes of the CPS and NHIS are also severe limitations on their use as a screener survey. There are some possibilities of accumulating cases over time, but that would introduce complications.
(4) The large sample size of the NIS is a positive feature in considering its use as a screener survey. However, there would be problems using it as a screener survey given its focus on families with children and its disproportionate allocation to the IAP areas.
(5) The size, breadth, and currency of the Census Bureau's planned ACS make it a promising possibility to serve as a screener survey. However, at the time of the workshop, its funding situation was uncertain; whether it will be funded, the level of funding, and the continuation of funding into the future were unclear. Ongoing use of the ACS, particularly if a field of degree question is added, would provide opportunities for
changes in the design and scope of the NSCG and, to a lesser extent, the NSRCG.
(6) Some possibilities for sampling new immigrants in S&E occupations may present themselves in the future at the INS, and lines of communication should be kept open. Currently, occupation is recorded on an INS form upon arrival, but it can be ambiguous or changed. Efforts to improve the currency and reliability of this information may be made in the near future. A frame with current and reliable information on degree and/or occupation of immigrants could help fill an important coverage gap in the current SESTAT structure.
- The panel felt that it would be unwise for NSF to rely solely on the ACS as its screener survey. The two main alternative strategies are as follows:
(1) Continue with the current panel, used throughout the 1990s, with freshening of the panel for new entrants to the S&E population. This option would retain all the past problems, such as response rate declines over the decade and the resultant degradation in the quality of the panel.
(2) Repeat the process of the last decade, starting with a sample drawn from the long form of the 2000 census to produce the first wave of a new NSCG in 2003.
With either of these strategies, there would still be a need to conduct the NSRCG to freshen the sample. NSF could switch to the ACS later in the decade, as soon as it is practicable. Alternatively, NSF could continue with one of the strategies throughout the rest of the decade, perhaps conducting pilot surveys with the ACS and then switching to the ACS at the beginning of the 2010 decade.
- If NSF adopts one of these strategies, it will probably need to continue its research on cost-effective quality improvements for the SESTAT system, which might include the following:
(1) Research on nonsampling error issues
(2) New technology options regarding cell phones and web-based data collection
(3) Improved ways to maintain current panels (e.g., better tracing and searching)
The main SESTAT policy decisions facing NSF at the time of the meeting were as follows:
- Sample quality considerations
(1) NSF is not conducting the NSCG in 2001. Thus, if the current panels are maintained and these panel members are next contacted in 2003, it will have been 4 years since the panel was last contacted.
(2) If a sample from the 2000 census long form is not selected and contacted soon, it will become difficult to find the individuals, and that difficulty will increase as time goes on.
- Budget considerations
(1) If NSF decides to wait for the ACS option to work out, and it then fails, budget limitations may make it impossible to restart the current survey later.
(2) The NSCG response rate is declining as the panel ages. It now stands at about
65%, and further declines could threaten the quality reputation of the survey. Maintaining the current panels for a while is the least expensive option.
Other important SESTAT policy decisions facing NSF follow.
- Subgroup considerations
(1) Any insistence on producing precise estimates for very small domains, e.g., for very small population groups, such as subdomains within American Indians and the disabled, can substantially increase costs. Resources spent on accumulating adequate sizes of rare subpopulations from the already rare population of scientists, engineers, and technicians are resources unavailable for other survey needs. An assessment needs to be made of the necessity for all the current survey categories. Some may be less cost effective to sample than others.
(2) It may be adequate to combine some small groups for detailed categories of responses and to report on them individually only for major categories of
responses. Perhaps NSF could establish a cost-effective threshold population size (which some small groups may not meet).
(3) The population of the SDR is covered by other surveys but not at high enough sampling rates. It is therefore highly likely that the SDR will continue in its current form. Besides, the SDR has a dedicated user group that will likely not want to risk major changes.