A SUMMARY OF FINDINGS FROM THE 1999
SRS CUSTOMER SATISFACTION SURVEY
In 1999, the Division of Science Resources Studies (SRS) within the National Science Foundation (NSF) sponsored a survey to determine who used data on science and engineering (S&E) personnel and resources, and how satisfied they were with SRS' products and services. The survey of roughly 1,000 potential S&E data users was conducted by mail, with telephone followup of those who did not respond. The survey built on an earlier SRS survey conducted in 1996 and, where possible, compared the 1999 data with those from 1996. The survey intentionally included both users and nonusers of SRS data, since the opinions of nonusers may provide SRS with information about the need for its products and services. However, respondents who said they did not use S&E data were not asked to complete most of the questionnaire. The statistics on satisfaction with S&E data refer only to S&E data users, not to the full sample, which includes nonusers.
The satisfaction of three different but overlapping groups is discussed here: government policymakers, the sample that was surveyed in 1996, and known users of SRS data. The first group consisted of 273 government policymakers who were likely to use S&E data based on their positions (e.g., as congressional staff for related committees, as state R&D technology officials, and as participants in government-sponsored science panels). The sample that was surveyed in 1996 was based on a list from the American Association for the Advancement of Science (AAAS) of people who had declared an interest in science policy; only people who responded to the 1996 survey and said they were users of S&E data were included. The sample of known users of SRS data users was compiled from SRS' RADAR tracking system, which is used by SRS to maintain records on requests for products or services.
Generally, the three groups showed similar patterns of responses. To simplify the presentation of the survey findings, the results are presented as the range of responses that appeared across all three groups (for example, "80-85 percent" would indicate that depending on the specific group, between 80 and 85 percent of the respondents chose a particular response). Following are some of the major findings from the survey.
Overall, respondents were positive in their evaluations of SRS, whether evaluating specific SRS products, evaluating SRS data by topic area, or naming the data source they used most. Government policymakers, the 1996 sample, and known users of SRS data all showed similar levels of satisfaction. When asked to evaluate specific SRS products, 80 percent or more of the respondents described SRS products as either excellent or good for most of the dimensions that were examined, with respondents more often rating the products as good than as excellent. Respondents were also asked their satisfaction with SRS data by topic area. Between 47 and 89 percent described themselves as either very satisfied or somewhat satisfied, while no more than 7 percent expressed dissatisfaction in any topic area. Another indication of respondents' positive evaluation of SRS products was that they reported getting their S&E data more frequently from SRS than from any other source. When asked to write down the name and source of a product they considered one of the best they had used, between 50 and 65 percent named an SRS publication. The fact that the survey was sponsored by SRS may have led some people to think of SRS products first. Still, the result suggests that respondents considered SRS publications to be among the best.
There also was evidence that satisfaction with SRS has improved since 1996. Depending on the area that was examined, the percentage who described SRS-produced information as excellent increased by a range of 4 to 34 percentage points from 1996 to 1999. The increase occurred across all five topic areas where sufficient data were available to measure change (R&D funding, R&D infrastructure, S&E work force, S&E education, and technology/innovation) and across all four aspects of quality that were examined in both surveys (clarity, relevance, accuracy, and currency/timeliness).
SRS's Science and Engineering Indicators stood out both as the single publication that was used most frequently from any source (i.e., either SRS or non-SRS sources), with 46 to 58 percent naming it, and as the SRS product that most often was described as very useful (65-75 percent). Other SRS products that were frequently described as very useful were National Patterns of R&D Resources (49-63 percent), the SRS web site (54-62 percent), and responses to special requests (45-53 percent).
The topic areas in which people were most often very or somewhat satisfied with the usefulness of SRS information were R&D funding (83-89 percent of those who used such information), S&E education (80-87 percent), and the S&E work force (77-81 percent). Lower levels of satisfaction (but still generally with a majority being satisfied) were reported for international S&E degree and personnel information (53-57 percent), public attitudes toward S&E (60-62 percent), scientific publications and citations (57-59 percent), patents (47-62 percent), and technology/innovation (59-67 percent).
The SRS product or service most often considered excellent was communications with SRS staff. Roughly half or more of the respondents said the communications were excellent with regard to staff expertise, courtesy, meeting special needs, and handling complaints, and 37 to 45 percent said communications were excellent with regard to the time needed to reach knowledgeable staff.
Respondents were asked their preferences in terms of the topic areas that were most important to them, the aspects of information that they considered most important, and the reporting format that they most preferred.
The topic areas where respondents most often thought detailed and accurate information would be very useful to them were Federal R&D funding (63-84 percent), academic R&D funding (61-75 percent), industry R&D funding (50-66 percent), the S&E work force (50-64 percent), and S&E education at the graduate level (53-61 percent).
The two aspects that respondents most often considered necessary when examining S&E data were their ability to find what they needed (76-85 percent) and the accuracy of the data (79-84 percent). Other qualities that were frequently described as necessary were objectivity (55-69 percent), timeliness (49-58 percent), and relevance (47-52 percent).
The five reporting formats that a majority of respondents considered very useful were detailed statistical tables in electronic format (62-69 percent), charts and graphs (61-68 percent), topical briefs and reports (61-65 percent), highlights (58-62 percent), and electronic data files (56-58 percent). Both government policymakers and the 1996 sample said their method of first choice for receiving S&E data was through the World Wide Web (62-68 percent), while SRS users were split between the web (48 percent) and printed reports (47 percent). More specifically, the survey asked for the preferred format for distributing reports consisting of detailed statistical tables, because those reports are expensive to print. Most respondents (63-69 percent) said the ideal method of distributing them was through the Internet, and almost all of the remainder considered the Internet a workable approach (26-31 percent). However, 63 to 68 percent said it would be a major inconvenience if printed reports were eliminated entirely.
Respondents tended to be most satisfied with SRS products with regard to their accuracy and objectivity. The area where they were least satisfied was currency/timeliness. Since efforts to improve timeliness could affect accuracy, it is useful to see how important these two dimensions were to customers. The survey indicated a greater perceived need for accuracy than for timeliness (i.e., 79-84 percent said that accuracy was necessary, compared with 49-58 percent for timeliness).
Another area of great importance to respondents was electronic access to SRS data. Respondents indicated that the Internet was their first choice for obtaining data and showed a high familiarity with the SRS web site (72-78 percent had visited it). In three areas a large majority of respondents were satisfied with the web site: ease of use (85-93 percent), the types of information offered (92-94 percent), and clarity (79-90 percent). In two other areas most respondents were satisfied, but there was room for improvement as the satisfaction levels were lower: respondents' ability to obtain reports from the SRS web site (74-79 percent) and their ability to obtain data (69-81 percent).
The survey was intentionally sent to people other than known SRS users in order to find out how much SRS data were used. SRS data could fail to reach all of its intended audience in three ways: people might not use any S&E data (regardless of the source) even when they could benefit from the data; people might use S&E data but not use SRS data; or people might use SRS data but not as fully as possible.
Many people (28-40 percent) reported they did not use any S&E data in the last year. Of these, two groups gave responses indicating that they might have been potential users of S&E data: they did not use S&E data either because they did not know where to find such information (9-12 percent of the nonusers), or because the information was not organized to meet their needs (7-14 percent). Such people (16-26 percent of nonusers) represent a potential target audience for SRS. The others not using S&E data in the last year (74-84 percent) either did not report why they did not use the data or reported the information was not useful or they were not interested.
Another potential target audience is those who were S&E data users but did not use SRS data at all. However, few people fell in this category. Between 82 and 90 percent of S&E data users indicated that they used SRS publications. S&E data users also often made use of SRS staff (36-44 percent), and some may also have used SRS when performing web searches (69-85 made web searches, and 72-78 percent said they had visited the SRS web site). Only 6 percent of respondents who reported using S&E data had not used at least one of these three methods of accessing SRS data (SRS publications, SRS staff, and visiting the SRS web site), i.e., 94 percent of S&E data users had used at least one SRS source.
Finally, some S&E data users may not have used SRS data as fully as possible (e.g., for all topic areas that were relevant) or as frequently as the data could have been useful. The survey did collect data on the frequency of use, and only a minority of respondents could be considered frequent users (i.e., 8-15 percent of all respondents, including nonusers, used S&E data once a week or more, and 14-16 percent used S&E data once to three times a month). Whether this is a sign that S&E data users did not use the data as fully as they might is unclear, since no data were collected on the level of need for S&E data among the respondents.
By many measures, customers were satisfied with SRS products and services, and that satisfaction has increased since 1996. Customers typically described SRS products as either excellent or good, and they generally obtained S&E data more often from SRS than from any other source. The SRS products and services that generally received the highest ratings included Science and Engineering Indicators, National Patterns of R&D Resources, the SRS web site, responses to special requests, and communications with SRS staff.
The survey also provided useful information about ways SRS could enhance its services and products to be responsive to its customers. It might give special attention to enhancing services, data, and reports provided through the Internet, which has become S&E data users' first choice for obtaining data. SRS also might enlarge its target audience by helping potential users know how to find the information they need and by organizing the information in a way that would help users access the data more easily.
Division of Science Resources Studies
National Science Foundation
by Bradford Chaney, Westat