Title: The ERC Program: An Assessment of Benefits and Outcomes Date: February 18, 1998 Engineering Education and Centers Division Directorate for Engineering National Science Foundation Arlington, VA 22230 THE ENGINEERING RESEARCH CENTERS (ERC) PROGRAM: AN ASSESSMENT OF BENEFITS AND OUTCOMES December 1997 Linda Parker Project Director Engineering Education and Centers Division Directorate for Engineering National Science Foundation Arlington, Virginia Table of Contents Page I. Executive Summary_____________________________________ i II. Background: The ERC Program__________________________ 1 III. The Studies: Program Results After Ten Years_________ 3 IV. ERC-Industry Interaction______________________________ 6 V. Effectiveness of ERC Graduates in Industry and Other Sectors _______________________________________ 17 VI. Findings and Conclusions______________________________ 29 Appendix A: Engineering Research Centers_________ 32 Appendix B: Participation ERC Alumni in Graduate Training Activities ________ 34 EXECUTIVE SUMMARY The Engineering Research Centers (ERC) Program was created by the National Science Foundation (NSF) in 1985 to develop a government- industry-university partnership to strengthen the competitive position of U.S. firms in world trade and change the culture of engineering research and education in the U.S. The 26 ERCs* (as of August 1997) discover new industry-relevant knowledge at the intersections of the traditional disciplines and transfer that knowledge to industry, while preparing a new generation of engineering leaders who are capable of leading in industry by engaging successfully in team-based, cross-disciplinary engineering to advance technology. __________________ *19 ERCs have active awards; seven have completed successfully a full 11-year award. In 1994, the NSF's Engineering Education and Centers (EEC) Division initiated study of key aspects of the ERC Program: ERC- industry interaction and the effectiveness of former ERC graduate students in all sectors of employment. The primary intent of this two-part study was to examine the extent to which the Program was making progress toward its goals. The purposes of the ERC-Industry Interaction study were to (1) examine the patterns of interaction that have emerged between ERCs and industry; (2) determine which types of interaction were most useful to industry and brought firms the greatest benefits; and (3) assess the value of these to the companies. The purpose of the ERC Graduate Effectiveness Study was to examine; (1) the extent to which masters and doctoral graduates with substantial ERC experience are more effective than their peers; (2) what the graduates did while at an ERC; and (3) the impact of ERC activities on the graduates' effectiveness on the job. Information was obtained from industrial representatives of ERC member companies, supervisors of ERC graduates in industry and other sectors, and ERC graduates themselves through written surveys and telephone interviews. Interaction with ERCs provides 90% of ERC companies with a wide range of benefits such as those in the following table: _________________________________________________________________ Table A: Examples Of Significant Benefits Received By Firms Benefit Percent of Firms Access to new ideas, know-how, or technologies 84 Receiving technical assistance 63 Interaction with other firms participating in the ERC 50 Access to ERC equipment and facilities 40 Hiring ERC students and graduates 40 __________________________________________________________________ Not all companies received every type of benefit; but of the benefits realized by particular companies, among the highest- valued were the employment of ERC students and graduates, gains in intellectual property (product/process development or improvement, patents, and copyrights), and access to specialized equipment and facilities. Nearly a quarter of all firms reported having developed a new product or process as a result of their interaction with an ERC. The impacts on the firms derived from these benefits are threefold: A majority of the respondents indicated that their ERC involvement had influenced their firm's research agenda. Two- thirds of the corporate representatives reported that their firm's competitiveness had increased as a result of benefits received and the level reached 80% for firms involved with an ERC for eight to ten years. Finally, corporate personnel in firms hiring ERC students or graduates rated these employees as more productive and effective engineers than peers in the same firms. These are the basic impacts that the Program was designed to achieve. The ERC-industry interaction study demonstrated very strongly that company benefit is directly related to the length and level of active involvement in a center. The longer a firm participates and the more direct personal interaction there is between corporate and center personnel, the more direct benefits the firm will have received and the greater the effect on company competitiveness. A key finding was that "More Is Better" for these partnerships. The more centers can expose the industry representatives and their colleagues to the variety of ways in which they can be actively involved in the center, the more the firms, students, and centers will benefit. Three factors in particular are strongly related to the realization of positive outcomes for companies from their interactions with an ERC: the existence of a "champion" for the ERC within the company, the receptivity of company technical staff to ERC ideas and/or results, and management support for the ERC partnership within the company. Significant involvement in ERC activities during graduate study builds upon, or is comple-mentary to, traditional graduate education. Masters and doctoral degree recipients with ERC experience take the knowledge, skills, capabilities, and techniques they learned in ERCs with them to their subsequent jobs. Regardless of the nature of their jobs and their sector of employment, ERC graduates continue working in ways they learned in ERCs, e.g., in interdisciplinary teams and by engaging in industry-university collaboration to advance technology. ERC firms employing ERC students and graduates value this benefit of ERC interaction more than any other type of benefit. As the following table illustrates, supervisors and other industry representatives of firms employing ERC graduates judged the ERC- trained employees to be superior to non-ERC employees on a number of key performance dimensions. The same was true for ERC graduates working in other employment sectors. _______________________________________________________________ Table B: Supervisors' and Representatives' Rating of ERC Graduates As Superior to Peers Dimension Supervisors Representatives Overall preparedness 89.4% 80.2% Contribution to technical work 84.8 77.3 Depth of technical understanding 85.0 80.2 Ability to work in interdisciplinary teams 80.3 64.3 Breadth of technical understanding 80.7 74.3 Ability to apply knowledge and use technology 69.9 72.3 SCALE: 1 = "much worse"; 2 = "somewhat worse"; 3 = about the same"; 4 = "somewhat better"; 5 = "much better" NOTE: Values shown in table are percent responding at level 4 or 5 ________________________________________________________________ 1. BACKGROUND: THE ERC PROGRAM Program Mission The Engineering Research Centers (ERC) Program was created by the National Science Foundation (NSF) in 1985 to develop a government- industry-university partnership to strengthen the competitive position of U.S. firms in world trade. At the heart of this problem was the need to increase the productivity of U.S. industry in an increasingly global economy. The ERC program was designed to create long-term collaborations between universities and industry, create new industry-relevant knowledge at the intersections of the traditional disciplines, and prepare a new generation of engineering leaders who are more capable of engaging successfully in team-based, cross-disciplinary engineering practice. Individual ERCs provide an integrated environment for academe and industry to focus on next-generation advances in complex engineered systems important for the Nation's future. Activity within ERCs lies at the cusp of the discovery-driven culture of science and the innovation-driven culture of engineering, creating a synergy between science, engineering, and industrial practice. ERCs provide the intellectual foundation for industry to collaborate with faculty and students on resolving generic, long-range challenges, thus producing the knowledge base needed to generate steady advances in technology and transition those technologies to the marketplace. Background and Goals The ERC Program grew out of concern expressed in the early 1980s by the National Academy of Engineering (NAE) and NSF that: (1) rapid technological advances were occurring at the intersection of engineering and other disciplines, requiring a cross- disciplinary approach to engineering that had not been incorporated into engineering research or practice; and (2) a mismatch had developed between the way engineering was carried out in industry and the way students were being trained. At the request of NSF Director Edward Knapp in 1983, NAE advised NSF on the mission and operations of the new ERC Program (NAE, 1983).* The Academy identified two goals for the Program: - to improve engineering research so that U.S. engineers will be better prepared to contribute to engineering practice; and - to assist U.S. industry in becoming more competitive in world markets. In addition, NAE advised that there should be three common elements across all ERCs: - Provide continual interaction of academic researchers, students, and faculty with their peers - namely, the engineers and scientists in industry - to ensure that the research programs in the centers remain relevant to the needs of the engineering practitioner and that they facilitate and promote the flow of knowledge between the academic and industrial sectors; - Emphasize the synthesis of engineering knowledge; that is, the research programs should seek to integrate different disciplines in order to bring together the requisite knowledge, methodologies, and tools to solve problems important to engineering practitioners; and - Contribute to the increased effectiveness of all levels of engineering education. _____________________ *NAE, 1983. Guidelines for Engineering Research Centers. Washington DC: National Academy of Engineering. Program Operations The ERC Program is managed out of NSF's Engineering Education and Centers (EEC) Division. Since 1985, 29 ERC awards have been made; as of August 1997, 26 centers are receiving ERC Program support or have successfully completed a full 11-year ERC award period. (see Appendix A) Each new center receives a five-year cooperative agreement stipulating that a formal review including external site visitors be conducted in the third year for renewal of support. Centers passing the third-year review receive a new five-year cooperative agreement requiring a formal review in each center's sixth year. The full term of an ERC is 11 years. As a center begins its seventh year, if not before, it should begin to generate even higher levels of non-NSF funding to lay down a pathway to self-sufficiency after Year 11. Funding is phased down in an ERC's ninth and tenth years. The center is expected to preserve the ERC culture as it moves into self-sufficiency. Sub-groups within an ERC may choose to regroup with others to compete to form a new ERC in the final two years of the ERC's award. From the beginning of the Program, each active center has submitted data about its activities and outputs annually to ERC Program staff. For example, in the 1994-95 reporting year, which included 21 centers, ERCs received a total of $51.7 million from the ERC Program and $53.7 million from firms in cash, in-kind donations, and associated grants and contracts. (Of this, $17 million was cash.) Other Federal agencies, state governments, universities, and other organizations provided the ERCs with $73.5 million. In that year, there were 708 industrial partnerships with ERCs, involving 523 companies or industrial consortia. As of Fall 1996, 228 patents have been granted for intellectual property emerging from ERC activities and 1142 licenses to patents and software products originating at the centers were negotiated. By the end of the 1995-96 year, 1,808 bachelors degrees, 1677 masters degrees, and 1587 doctoral degrees had been granted to students who had been involved in ERC activities. 2. THE STUDIES: PROGRAM RESULTS AFTER TEN YEARS In 1994, the EEC Division initiated two studies of key aspects of the ERC Program: ERC-industry interaction and the effectiveness of former ERC graduate students in all sectors of employment. The primary intent of these studies was to examine the extent to which the Program was making progress toward its goals of developing long- term relationships between academe and industry and preparing a "new generation" of engineers who are able to come up to speed more quickly and effectively in industry. The studies were intended principally for NSF program management purposes, as a component of continuous improvement of the Program and to extend EEC's knowledge of the benefits, outcomes, and impacts of the data that centers report in the aggregate about interactions with industry and student involvement in ERCs. These initial studies constitute the beginning of a series that will take a comprehensive look at the Program over several years. Data for the first two studies were collected in 1995 and early 1996 as the Program entered its second decade of operation. This report summarizes and integrates the results of these studies. The methodology for both studies is described below. Results of the ERC-industry interaction are presented in Section 3, followed by those of the ERC graduate effectiveness study in Section 4. Overall findings and conclusions are presented in Section 5. Methodology: ERC-Industry Interaction Study The purposes of this study were to: (1) examine the patterns of interaction that have emerged between ERCs and industry; (2) determine which types of interaction were most useful to industry; and (3) assess the value of these impacts to the companies. The intellectual framework for the study was derived from the hypothesis that the greater and broader the types of involvement between firms and ERCs, the more the firms would benefit. This was a founding principle of the ERC Program that could not be tested until sufficient time had passed for the impacts of these interactions to become evident. Because industry-university interaction is by definition dynamic, the study was designed to reflect this fact, rather than simply focus on retrospective consideration of results of the Program to date. The study contractor, SRI International, began by conducting a series of case studies within a small number of firms -- each involved with a different ERC -- to obtain information on activities within firms that related to their participation in one or more ERCs. Information obtained through visits with corporate representatives was compiled into a preliminary "process model" of ERC-industry interaction. The case studies, the model, and results from a focus group with additional corporate representatives to ERCs were all used in development of the survey instruments and in specification of groups of individuals to be surveyed. Given the organizational complexity and decentralization of many firms, the target for the study was the unit that was the primary direct beneficiary of the firm's participation in a given center. The Primary Participant questionnaire was sent to the person from this unit whom the center identified as the corporate representative to the center. Names and addresses of company representatives, corresponding to the list of companies each center reported in the 1993-94 reporting year, were requested from each of the 18 ERCs in operation at the beginning of the study. (See Appendix A for a list of centers.) In a number of instances, a single firm had multiple partnerships across several centers through different corporate divisions. In each case, a separate point-of-contact was requested from centers; they provided 581 names and addresses to the contractor. Of those, 81 were not usable due to bad addresses, retirement, and similar reasons. Consequently, the survey population was reduced to 497, of whom 355 returned valid questionnaires for a response rate of 71%. To broaden the perspective within a single company, a related but shorter Secondary Participant questionnaire was sent to two additional people whom primary participants identified as being knowledgeable about the nature of the firm's involvement in the center and the impact of participation on the firm. Of the 266 people suggested by the primary participants, 138, or 52%, submitted valid responses. Surveying of both populations took place during 1995 and, in the case of the secondary participants, into early 1996. After preliminary review of the survey data, telephone interviews were conducted with 20 survey respondents to discuss in greater detail certain questions and themes that had emerged from the data. The results were assembled as mini-case studies and incorporated in the survey results. Methodology: ERC Graduates Effectiveness Study At the inception of the ERC Program, there was an expectation that involvement of graduate and undergraduate students in the full range of ERC activities would expand upon abilities and knowledge gained in traditional degree programs and result in more effective new entrants to engineering practice in industry. A companion goal was to produce new courses and course materials based on the ERCs' research, so as to expose a broader base of students to cross-disciplinary, systems-level courses. As it happens, some centers have developed new degree programs as well. The second ERC study in this series, conducted by Abt Associates Inc., examined: the extent to which masters and doctoral graduates with substantial ERC experience are more effective employees than are their peers; what the graduates did while at an ERC; and the impact of ERC activities on the graduates' effectiveness on the job. The emphasis of the study was on those who were employed in the industry. However, two additional ERC graduate populations were surveyed as well: those working in academia and those in Federal or other sectors. Questionnaires with slight differences were prepared for each of the three populations. Two assumptions determined which centers and which former students were surveyed. First, it takes a center approximately three years to achieve full operations and, second, substantial ERC experience as a student would require at least two years of involvement for there to be significant impact. Thus, only individuals who had graduated from the first 14 centers (see Appendix A) in these centers' third year or later with two or more years of ERC experience were eligible for the study. The ERCs provided the names and available contact information for eligible graduates. Using several approaches to find correct addresses and phone numbers, 720 eligible graduates were contacted initially by phone; of these, 554 provided the names of their supervisor(s) and agreed to allow their names to be used when contacting the supervisor(s). Sixty percent (433) of the graduates responded to a written survey about their ERC experiences, other educational background, and present job. Eighty-six percent (477) of the supervisors responded to a short telephone interview about the effectiveness of their employees with ERC experience in graduate school. As with the graduates, slightly different questionnaires were used for the supervisor in industry, academia, and Federal or other sectors to reflect differences among the sectors. Legal reasons prevented inclusion of control group(s) of engineering students in this study. It would have been ideal if individuals with similar graduate education without exposure to anything like an ERC also could have been tracked and information about their effectiveness in their respective sector of employment compared with that of individuals with ERC experience. However, it is illegal to obtain information from a college or university about the activities of someone who has graduated from the institution without the expressed permission of that individual. But in order to determine whose permission would be required, it would be necessary to see the academic records first. An alternative approach, selecting a sample of individuals from large national statistical databases at the Department of Education and at NSF and then contacting these individuals, proved to be methodologically insupportable. Statisticians in charge of the relevant databases advised that an appropriate sample could not be drawn. Thus, the supervisors' questionnaires were designed to obtain a "proxy comparison," rather than a direct comparison between ERC graduates and a pre-defined control or comparison group. In the proxy, supervisors assess the extent to which ERC graduates are less or more effective than their peers in the same firm on a series of capabilities related to the foci of ERC education, as well as capabilities associated with traditional graduate education. To provide a second perspective on the effectiveness of graduates in employed industry, industrial representatives surveyed in the ERC-Industry Interaction Study also rated ERC graduates whom their firm had hired against peers in their firms. The results of this additional component are presented in Section 3. 3. ERC-INDUSTRY INTERACTION Relationships of Firms to ERCs At the core of the ERC Program is the concept of firms having membership in a center, which involves them in strategic planning, joint research projects, mentoring of students, access to the center's facilities and equipment, access to center researchers and students, and a number of other things, depending on the center. Center membership usually involves payment of a fixed annual fee that is pooled with cash from other members and sponsors for support of the center's research and research- related activities. In some cases, centers have accepted specific in-kind donations of such things as software and equipment in lieu of cash to establish membership. Centers set their own membership rates and often have associate memberships for small firms that cannot afford the cost of full membership, or for larger firms that are not yet ready to make a commitment for full membership. Member companies may also sponsor specific research projects and some form consortia around larger projects and testbeds. Occasionally, centers will introduce potential member firms to the ERC through project-level support. A total of 355 valid responses were submitted to the Primary Participants Survey. Table 3-1 shows the firms' distribution across the primary research or product line, as designated by the respondents: _________________________________________________________________ Table 3-1 Primary Research/Product Line of ERC Participants Percent of Primary Research/Product Line Respondents' Firms Electrical, electronic, telecommunications 21.6 Chemicals and allied products, including pharmaceuticals 15.8 Surgical/medical instruments and supplies 9.1 Petroleum refining and extraction 6.7 Aircraft and missiles 4.4 Machinery 3.5 Building construction 3.5 Motor vehicles and motor vehicle equipment 3.2 Primary metals 2.9 Professional and scientific instruments 2.6 Other non-manufacturing (e.g., engineering design/service and computer software companies) 9.4 Other 17.3 _________________________________________________________________ Industry representatives surveyed reported that their firms had been associated with the ERC for an average of about four years. Nearly a third had been involved for five to seven years, while 44.9% had newer partnerships that ranged from two to four years old. Involvement included providing financial support to centers. In 1994, nearly two-thirds of the firms had paid center membership fees, which averaged $21,028. The remaining firms provided in- kind support at the center's accepted in lieu of cash memberships. Most fees were between $10,000 and $100,000; however, 19.6% were less than $10,000 and 1.4% were over $100,000. In the same year, one firm in five had contracts with a center, averaging $125,000, for research related to the center's focus; nearly as many firms provided in-kind donations of such things as software and equipment averaging $55,000 in value. Additionally, a small percentage of firms provided unrestricted grants (averaging $45,000) and other support not specified (averaging $130,000). (The latter category is distorted by a $2 million donation to one center. Without this amount, the average would be about $60,000.) ERC-Firm Interactions Respondents and other members of their firm were involved with ERCs in a variety of ways; patterns of involvement also varied considerably across firms. Keeping in mind that the centers themselves identified the corporate representatives who were included in this study, there were some differences in the "official" roles of these individuals in a given center, independent of how they actually interacted with that center. Nearly 82% of those surveyed acted as their company unit's main point of contact with the center, while more than two-thirds coordinated or helped coordinate the participation of their firm's technical staff in the ERC as well as being responsible for preparing budgets or justifications/recommendations for continued annual membership in the ERC. Nearly a third were responsible for approving the budget for the firm's membership in the center. Slightly more than a third were voting members of the center's advisory body that is responsible for making recommendations regarding the center's management or research plans. Most corporate representatives to ERCs were personally involved in their ERC in numerous ways. Nearly three-quarters (71.9%) had attended or participated in one or more ERC seminars or workshops, while almost as many (67.9%) had reviewed ERC research results or publications. Looking at more active forms of involvement, more than half (59.0%) had received technical advice or consultation from either ERC faculty or other ERC researchers. On a smaller scale, respondents reported being involved in personnel exchanges. More than a quarter (30.4%) of the respondents indicated that they had collaborated with ERC researchers at the company's site, while 23.8% participated in research at the ERC's site. About 18% supervised a former or current ERC student working at his or her company and nearly 15% co-authored a publication or developed invention(s) jointly with ERC researchers. Relatively few respondents (6.9%) had served on an ERC student's thesis or dissertation committee. Industry representatives reported that the principal vehicles for other members of their firms to interact with the ERC were the same as those for the respondents: reading ERC reports or publications, consulting with ERC faculty or researchers, and participating in ERC seminars and workshops. Again, however, a notable proportion were more directly involved: over a third reported that at least one other member of the firm had participated in a joint ERC project at the ERC's site and roughly the same proportion indicated that a colleague had been involved in a joint project at the firm's site. In addition, slightly less than a third mentioned that at least one colleague had used ERC facilities and equipment. Benefits to Firms Nearly 90% of industry sponsors determined that their firm had received benefits from ERC participation and over two-thirds indicated that the participation had had at least some effect on their firm's competitiveness. Of the latter, about 40% rated the impact of ERC participation on corporate competitiveness as moderate to a great deal. The major benefits fell in five categories: (1) access to new ideas, know-how, or technologies; (2) receiving technical assistance; (3) interaction with other firms participating in the ERC; (4) access to ERC equipment and facilities; and (5) hiring ERC students and graduates. In addition, over half of the respondents indicated that, as a result of their ERC involvement, they had influenced their firm's research agenda. 1. Access to New Ideas, Know-how, or Technologies More than 80% of the respondents indicated that access to new ideas know-how or technologies had been a very important reason for participating in the first place, and over 80% of these reported that their firm had in fact obtained access to such things as a result of ERC participation. Further, more than a quarter of those who reported receiving such access reported that their firm had benefited "a great deal" from the ideas, know-how, or technologies. Examples included: "The benefits are considered large compared to the limited costs of participation. Of main importance is knowledge about personnel to contact. Participation has also led to a substantial increase in knowledge in certain areas of importance and familiarized my unit with U.S. research in relevant fields as well as led to contact with other companies." "As a small company (40 employees), we have a difficult time accessing technical information. Through the ERC, we have had access to worldwide information in the fields of our interest." "We get ideas from the ERC which we translate into development programs in our department. We also use the ERC to critique some of our ideas." 2. Receiving Technical Assistance Over half of the firms benefited from receiving technical assistance from an ERC. Examples of the most significant benefits belonging to this category include the following: "At the time we started working with the ERC, we were a very small start-up company with few employees. The collaborative research done with the ERC was pivotal in establishing the feasibility of our product and defining certain technical parameters." "Technical advice on a new product concept has been crucial - a contract to start next week will provide the important information on whether to go forward." "...direct technical collaborations. ERC faculty have developed excellent computer modeling and simulation tools, which have been of great value in providing deeper understanding of device performance limitations and design trade-offs." 3. Interaction with other Firms Participating in the ERC Over 50% reported that their firm benefited from increased interaction with other firms participating in the same ERC. Industrial representatives described a variety of particularly significant benefits that resulted from opportunities to interact with other participating firms, which are often competitors: "The ERC has served as a networking forum in which to discuss issues critical to our industry with competitors and customers." "...contact with U.S. competitors in a non-competitive environment." "As a small company, we have had access to people that were not available to us, such as technical personnel from large multinational corporations in our industry." 4. Access to ERC Equipment and Facilities ERC facilities and equipment also proved useful to 40% of the respondents' firms, with over a quarter finding the facilities and equipment to be of great benefit to their firm. Descriptions of significant benefits from access to ERC equipment and facilities include: "Cross-training of company technicians in ERC facilities has shown improvements in laboratory analysis accuracy and throughput. In one training session, immediate results were observable." "...access to $28 million in instrumentation that can be used for evaluation - in some cases the benefit has caused us to purchase capital equipment." "The opportunity to participate in joint projects at the ERC site. ERC services and workshops are particularly beneficial to our company." 5. Hiring ERC Students and Graduates One especially beneficial outcome was hiring ERC students while still in school or graduates as soon as they received their degree. While less than half of the firms (39.9%) did so, more than half of those who did received "a great deal" of benefit as a result. The benefit received from this outcome was more highly rated than was the benefit from any other outcome of ERC participation. As the following figure shows, there was substantial variation across the centers in the percent of firms that received this benefit: Figure 3-1 Section 4 provides detail on the benefits to firms of hiring ERC graduates, the graduates' capabilities relative to peers without ERC experience, and the impact of ERC and non-ERC graduate school activities on their job performance. While firms frequently initiated interaction with a center in order to receive one or more of the five benefits described above, they often experienced benefits that they had not anticipated. This was true in the case of access to ERC facilities and equipment in that far more firms ultimately found such access valuable than had originally intended to make use of it. Other Benefits As a result of having received these benefits, over half of the firms were able to improve an existing product or process. The following figure illustrates the substantial difference across centers in the extent to which participating firms reported this occurrence: Figure 3-2 Additional benefits related to intellectual property and product/process development were reported by smaller proportions of respondents, but in each case, firms valued them among the highest of all types of benefits received from involvement with ERCs. More firms benefited from these results than had anticipated doing so. Nearly a quarter of corporate representatives indicated that their firm had developed a new product or process as a result of their interaction with an ERC. A quarter of these firms characterized the outcome as being of great benefit to their firm. Similarly, of the nearly one-third of respondents reporting that their firm had patented or copyrighted technology or software developed on the basis of interaction with the ERC, the same proportion indicated that their firm had benefited a great deal from this result. Licensing of ERC-developed software was reported by slightly over a quarter of respondents, with 11.8% of these people reporting that their firm received great benefit from having done so. Factors Affecting Receipt of Benefits 1. Length and Nature of Participation Companies receive benefits in direct relation to the number of years they participate in an ERC and the extent of active involvement with the center: the longer a firm participates and the more direct personal interaction corporate and center personnel have, the more direct benefits the firm will receive and the greater the effect on company competitiveness. While three-quarters of firms participating in an ERC for less than a year experienced at least some benefit in that short time period - and 22.6% had received a great deal of benefit - over 90% of industrial representatives from firms involved with an ERC for eight to 10 years report at least some benefit, with 47.2% reporting moderate benefits and another 36.1% reporting a great deal of benefit. The same basic pattern holds true for competitiveness. Firms with a year or less participation in a center reportedly had experienced little or no effect on the firm's competitiveness, although 12.5% of the respondents indicated that the center had already contributed a great deal to their firm's competitiveness. By eight to ten years of involvement, the industrial representatives reported that more than 80% of their firms had already experienced at least some positive effect and 10.4% reported a great deal of effect on their competitiveness. Overall company benefits from participation in an ERC also are almost directly proportional to the extent of interaction between company employees and the ERC. In particular, active interaction on the part of the industrial representatives and others in their firm with ERC personnel and facilities led to substantially greater benefits to the companies. Active interaction includes such things as participation in research at the ERC site, collaboration with ERC researchers working at the company's site, supervision of a current or former ERC student working at the company's site, use of the ERC's facilities or equipment, co- authoring publications or developing inventions jointly with ERC researchers, or receiving technical advice or consultation from ERC researchers. Nearly a quarter of the representatives who engaged in none of these things reported that their company received no benefit from involvement with the ERC. On the other hand, 52.0% of respondents who were personally involved in at least four of these activities reported that their firm had received a great deal of benefit from participation in the center. When reporting on the involvement of other members of the firm in the center, 38.2% of the respondents reporting that their colleagues had engaged in six to eight different types of involvement indicated that their firm had benefited a great deal from participation. In terms of quantitative ratings, representatives who reported no active personal interaction with the center rated their firm's overall benefit as an average 2.2 on a scale of 1 (little or no benefit) to 4 (great deal of benefit). The rating of benefit rises to 3.3 when either the industrial respondent or others in the firm are involved in at least four different types of personal interaction with faculty, researchers, and students in the center. Attendance at ERC seminars and workshops, as well as reviewing ERC research results or publications were excluded from this analysis because they do not involve active participation in center activities. 2. Corporate and Center Characteristics and Activities A number of factors contribute to the receipt of benefits from interaction with ERCs. The most notable of these relate to the centers themselves. One of the most important is the closeness of the ERC's technical focus to that of the company unit that is involved; this factor was rated as very or extremely important by 58.3% of the industrial representatives. Additional factors identified as very or extremely important include the responsiveness of the ERC faculty and other researchers to the needs of the company (49.2%) and efforts by the ERC to communicate and stay in contact with corporate participants (48.1%). Another group of factors relate to company activities that contribute strongly (very to extremely important) to positive outcomes for the companies from interactions with an ERC. Three such activities emerged: the continuous existence of a "champion" for the ERC within the company unit (43.1%), receptivity of company technical staff to ERC ideas and/or results (41.7%), and management support of the ERC within the company (38.1%). 3. Barriers to Receiving Benefits Corporate representatives reported that there was no particularly strong barrier to their firm's ability to benefit from its involvement with the ERC. On a scale of 1 to 4 from "not a barrier/doesn't apply" to "major barrier", the primary problem was that other company priorities preempted involvement with the ERC. This item was considered a major barrier by 21.1% of the respondents and a moderate barrier by 23.7%. The next most frequently cited "major" barriers pertained to differences in approach to work and values between individual ERCs and companies: (a) different conceptions of time between project initiation and completion (13.9%); (b) ERC research is not sufficiently relevant to the company's needs (13.8%); (c) differences between the ERC and the company in values, mission, or priorities (12.3%); (d) intellectual property issues (12.2%); and (e) different conceptions of the importance of meeting project deadlines (12.2%). An additional 14.7% to 28.9% of the remaining respondents rated these items as moderate barriers to receiving benefits from involvement with the center. On the other hand, over half of the industrial representatives indicated that a number of things did not cause a problem or were not applicable in their relationship with the center: (a) poor communication between the ERC and the firm; (b) internal company "politics"; and (c) geographic distance between the company and the ERC. Approximately a quarter of the other representatives identified these things as minor barriers to receiving benefits from the center. Similarly, "lack of awareness about the ERC within my company" and "insufficient influence on the ERC's research agenda" were viewed as inapplicable or not a barrier by approximately a third and as minor or moderate problems by approximately half of the respondents. The latter is consistent with only 38% indicating that they have moderate to great influence on their center's research agenda. While the level of influence that corporate representatives have on a center's research agenda does not appear to be major hindrance, the proportion of representatives describing their influence as minimal appears to have grown over the years. In an early study of the ERC Program by the U.S. General Accounting Office (GAO, 1988),* 16% of industry representatives reported little or no ability to influence the research agenda. This compares with 31% in the present study. This increase is important to note because the level of influence appears to be related to a firm's receipt of benefits. As the following table illustrates, those describing their influence on the center's research agenda as moderate to great reported greater benefits than did those with little or no influence. _______________ *GAO, 1988. Engineering Research Centers: NSF Program Management and Sponsorship (GAO/RCED 88-177). Washington, DC: Government Accounting Office. _________________________________________________________________ Table 3-2: Overall Benefits to the Company from ERC Participation, by Company Influence on the ERC's Research Agenda Influence on Little Some Moderate Great Mean the ERC's or Amount Deal Rating** Research None Agenda __________________________________________________________________ Little or none 26.9 40.7 25.0 7.4 2.1 Some 6.5 30.6 50.9 12.0 2.7 Moderate Amount 1.0 14.1 52.5 32.3 3.2 Great Deal 0.0 3.2 48.4 48.4 3.5 __________________ * Chi-square analysis shows differences to be significant at .05 level or less **Items were rated on a 4-point-scale, with 1="little or none", 2 = "some", 3="a moderate amount", and 4="a great deal." _____________________________________________________________________ Industrial representatives identified some of the barriers that hinder or prevent their firm from receiving full benefits as a result of participation in the ERC: "We have not yet been actively involved in mentor programs and/or significant collaborative research. We are hoping to be much more involved in ERC Programs in the future. We have increased our hiring to allow support of this objective." "We would like to have collaborative projects, but our declining budgets and misalignment between the ERC's strengths and our short-term needs have made coordination difficult." "The research had better reach fruition of the objectives outlined or we will discontinue our association." "The technology is excellent in theory, but there was not the same energy to see if it works and what it takes to make it work." Firms' Future Expectations and Plans At the time the survey was conducted in 1995, two-thirds of industrial representatives identified their firm as likely or planning to, continue participation in their ERC in 1996, while over half were probably or definitely going to continue doing so in 1997. The proportion definitely not continuing in either year is essentially constant (16%-17%), but the group that is uncertain grows in the out-years. A number of statistical correlation's emerged connecting future plans with benefits firms had already received, impact of center interaction on the firm's competitiveness, length of involvement, and ease of receiving approval within the firm for center membership fees: - Nearly two-thirds of the industrial representatives from firms having received moderate to a great deal of benefits indicated in 1995 that they probably would continue participation in 1996 and were nearly as certain about involvement in 1997. - Over 80% of firms experiencing moderate to a great deal of impact on their firm's competitiveness as a result of benefits received from center involvement were probably or definitely going to continue participation in the following year, compared with 53% of firms experiencing less impact on competitiveness. - On a scale of 1 to 5 (1 = no likelihood of continuing, 5 = definitely will continue), the mean ranking of likelihood of continuing rises according to length of participation in a center from 3.2 for firms with one to two years of involvement to 4.3 for firms with eight to ten years of participation in 1996. Means are similar for likelihood of involvement in 1997. While all of these relationships are statistically significant, overall, firms are statistically less likely to continue for one or two more years than was the case in the 1988 GAO study of the ERC Program. In particular, fewer firms are definitely continuing and more are either uncertain or likely not to do so. Beyond the correlations described above, additional factors contributing to this difference may include recent shifts in patterns of funding and conducting R&D in industry and business cycle changes since 1988 within the various technology sectors of firms participating in ERCs. Secondary Participants' Responses To obtain a broader understanding of the extent and impact of firms' interaction with ERCs, this study included a survey of individuals whom the corporate representatives of participating firms indicated were knowledgeable about or personally involved in the firm's interaction with the ERC and the results in the firm. These secondary individuals were surprisingly similar in many ways to the corporate representatives. Although they were less likely to have a formal role with the center (e.g., as the firm's main point of contact with the center or as a voting member of a center's industrial advisory board), the secondary participants were as actively involved in the centers as were the primary representatives. Ironically, they were less likely to do passive things, e.g., reviewing ERC research results, than were the formal representatives! Overall, secondary participants confirmed findings obtained from primary respondents. In fact, the former tended to be even more positive about center involvement, especially concerning: - influence on the center's research agenda; - benefits received; - impact on the firm's competitiveness; and - likelihood of continuing involvement. Secondary participants were also more apt to identify involvement in joint projects and obtaining licenses to intellectual property developed in the center as being important reasons for participating in ERC than was the firm's representative to the center. Similarly, secondary respondents were statistically more likely than their colleagues to report that their firm benefited from hiring ERC students/graduates and from patenting or copyrighting intellectual property developed by the company as a result of ERC interaction, and that their firm was going to remain involved in the ERC in the following two years. 4. EFFECTIVENESS OF ERC GRADUATES IN INDUSTRY AND OTHER SECTORS One of the three common elements identified for all ERCs under the 1983 NAE guidelines for the ERC Program was the mandate to "contribute to the increased effectiveness of all levels of engineering education." In the first years of the Program, the latter came to be associated with creating a "new breed" of engineer. Drawing from the principle that the ERCs were to act as change agents in academic and industrial culture, this new type of engineer should reflect the features that make ERCs distinctive. That is, he or she should be adept at working in collaborative teams on interdisciplinary topics, approaching problem-solving from an engineering systems perspective, and staying attuned to the needs of industry. The new engineers were also expected to use an integrative approach to their work and have technical breadth as well as depth. Involvement of students in ERC activities was intended to augment graduate and undergraduate education, adding dimensions to the students' capabilities and knowledge that are necessary for successful engineering practice in industry. ERCs do not grant degrees; students remain affiliated with a home department, from which they receive the degree and in which they continue to take many of the courses in the required curriculum. The ERC experience is an enhancement, but (especially for graduate students), it is intended to have a major impact on the perspectives and capabilities of the student. While ERC experiences are intended to prepare individuals for industrial employment, those graduates who work in other sectors are expected also to apply the skills and approaches to engineering problems that the ERC Program aims to instill. Assessments of the extent to which the Program is succeeding in preparing graduates who are effective in industry were obtained from two independent sources: (1) the industrial representatives to ERCs whose firms had hired ERC students or graduates; and (2) the supervisors of engineers who had had ERC experience in graduate school. Both groups compared specific skills, capabilities, and knowledge of ERC graduates with those of peers. The views of industry representatives whose firms had ERC students as temporary employees (21.8% of the firms) and graduates as full-time employees (27.3% of the firms) were obtained as part of the study reported in the preceding Section 3. Supervisor's assessments were obtained in a separate survey process under this Effectiveness study that included surveying of the graduates as well. Four hundred thirty-three ERC graduates working in industry, academia, the Federal government, and other sectors responded to a written survey. At the time that these surveys were conducted, a typical ERC graduate had been working for the same organization for an average of about four years and had had the same supervisor for an average of two to three years. While the primary interest was in the effectiveness of ERC graduates in industry, individuals employed in other sectors and their supervisors were surveyed as well. Table 4-1 provides demographic data on graduate survey respondents. Where findings relating to individuals working outside industry differ from those for individuals employed in industry, those findings are noted. ________________________________________________________________ Table 4-1: Demographics of Graduate Survey Respondents Percent of Graduates Gender Race-Ethnicity African Asian Native Pacific Degree Level Male Fem. Amer. Amer. Hisp. Amer. Islander White Masters Graduates 88 12 1 23 1 0 1 72 Doctoral Graduates 90 10 0 30 2 0 0 66 _________________________________________________________________ Overall Assessment of ERC Graduates As Table 4-2 shows, supervisors of ERC graduate and corporate representatives of firms that had hired ERC graduates or students judged their ERC-trained employees to be better than peers in the same organization on six key performance dimensions; between a third and a half of the same respondents characterized their ERC- trained employees as "much better" overall than their peers. __________________________________________________________________ Table 4-2: Supervisors' and Representatives' Rating of ERC Graduates As Superior to Peers Dimension Supervisors Representatives Overall preparedness 89.4% 80.2% Contribution to technical work 84.8 77.3 Depth of technical understanding 85.0 80.2 Ability to work in interdisciplinary teams 80.3 64.3 Breadth of technical understanding 80.7 74.3 Ability to apply knowledge and use technology 69.9 72.3 SCALE: 1 = "much worse"; 2 = "somewhat worse"; 3 = "about the same"; 4 = "somewhat better"; 5 = "much better" NOTE: Values shown in table are percent responding at level 4 or 5. _________________________________________________________________ On all other performance dimensions - familiarity with engineering systems approach; ability to use knowledge to develop technology; the amount of training the employees needed when hired; and ability to solve problems within time, money, and human restraints - half to two-thirds of supervisors and corporate representatives rated their employees with ERC experience as superior to their peers (4 or higher). There is little overlap between supervisors and industrial representatives, as only 17.5% of the latter had supervised any ERC students or graduates. Corporate representatives whose firms received significant benefit from hiring ERC students and graduates described the results of such hires in a variety of ways, including: "We have had a significant technology transfer through a recent graduate from the ERC. This has given us new and better approaches to R&D (product development) and a new focus on claim substantiations." "Since the formation of the ERC, we have hired four of their graduates. They are top contributors to our R&D and were well prepared to work and contribute in industry. This is the primary benefit of the ERC." "The most significant benefit has been access to students for cooperative education programs and full-time employees. ERC students have excellent applied skills in our company's competencies as a result of the academic and research environment of the ERC. The ERC is the best source of technical talent we have experienced." "As a small company, we have had relationships with graduate students that have been powerful for them and us. Students' projects at our company have educated both the students and our employees. We have been able to develop practice- oriented engineers." Graduates' Employment Characteristics Career Paths A number of differences exist across employment sector with regard to graduates' career paths up to the time of the survey. Nearly all of those working in industry and most of those in academia at the time of the survey - 92% and 86%, respectively - began work in that sector following their final degree. In contrast, less that three-quarters of those employed in federal laboratories or other sectors began work there. Of those moving into these non-industry, non-academic sectors, nearly two-thirds had previously been in industry and the rest came from academia. Overall, ERC graduates had developed stable but varied career paths with the same organization. Graduates employed in industry and academia in 1995 had worked on average around four years with their present employer, while those in federal laboratories and other sectors had been with the same employer for an average of five years. The extent to which they identified the field in which they were working as the same as that of their final degree varied. Not surprisingly, 86% of academics were working in their field of education. However, graduates in non-academic sectors were more likely to have shifted to another field, as 75% of those in industry and 70% of those in federal laboratories or other work settings no longer considered the field of their work to be the same as that of their highest degree. Job Activities ERC graduates working in industry* reported responsibilities in a wide range of activities, with the emphases being process and product engineering, manufacturing, fundamental or basic research, and technical management. Marketing/sales and general management were not prominent among their responsibilities. In their current jobs, they were most likely to use knowledge or techniques learned through their ERC work and to make use of specific ERC research results. They are unlikely, however, to continue working on things that they had been doing at the center. This is not surprising, given the variety of their current responsibilities and the likelihood that their jobs fit in with a broader corporate strategy or agenda that has little relation to ERC activity. In addition, since the graduates had worked an average of four years with their existing industry employer at the time they were surveyed, most had moved on in their careers and were not new degree recipients. ____________________ *The analysis of employment characteristics focuses on graduates working in industry, unless otherwise noted. Graduates working in industry engaged in a variety of activities with and without academic involvement. Approximately half of these graduates reported having both applied for patents and had articles accepted for publication, and over 70% also made presentations at professional meetings that did not include academic participation. On the other hand, more than two-fifths also indicated that they were involved in collaborative work with university researchers and over half have obtained technical advice from academic faculty or research staff; approximately one- third of those involved in such activities were collaborating with ERCs. Finally, approximately a quarter of the ERC graduates had made presentations based on collaborative work with university researchers or were co-authors of publications with academic researchers. The majority of these activities were performed with individuals and groups not associated with an ERC. In certain respects, graduates working in federal laboratories and those in industry were similar. For example, those in federal laboratories were even more likely than their colleagues in industry to collaborate with university researchers and to make presentations at professional meetings about work not related to collaborations with universities; but those in industry generally had a higher level of involvement with ERCs than did those in federal laboratories. Additionally, two-thirds of federal laboratory employees established or participated in cross-disciplinary research and nearly half carried out engineering systems-based research. Those employed in academia identified their primary job activities as conducting their own research and supervising research performed by students and, where applicable, postdoctoral fellows. Less important duties included administration not related to research, industrial liaison, and outside consultation. Faculty had, on average, a medium level of responsibility for teaching; but individuals tended to have either a high or relatively low level of teaching responsibilities. As with those working in federal laboratories, those in academia participated in activities involving industry and the general approach to research espoused by the ERC Program. Over half pursued collaborative research with industry, served as a consultant to industry, and received research support from industry. Most indicated that these activities were encouraged at their institution. Additionally, nearly 80% had established or participated in cross-disciplinary research groups and about 60% had worked on engineering systems-based projects. A majority also indicated that their research was focused to a substantial extent or entirely on problems directly relevant to industry. Graduates' Self-Assessment of Job Performance Using the same list of 16 job capabilities and skills with which supervisors rated their employee(s) and that graduates used to identify the items that are valuable in their present job (see below), the ERC alumni also assessed themselves relative to peers in their own organization. Using a scale in which 1 equaled "much worse than average," 3 represented "about the same," and 5 was "much better than average," graduates ranked themselves highest (mean values of 4.20-4.34) in general capabilities. These include: - ability to grasp quickly the key features of new problems; - contribution to the firm's technical work; - breadth of technical understanding; and - ability to define the steps needed to solve new problems. The group of capabilities immediately below this (mean values of 4.02-4.18) included: - ability to communicate ideas in writing; - creativity and innovativeness; - depth of technical understanding; - integrating and synthesizing information from different fields; - ability to work in interdisciplinary teams; and - ability to communicate ideas verbally. The items in the remaining group were less positively rated (mean values of 3.54-3.90): - understanding the relationship between work and customer needs; - meeting business goals while satisfying technical requirements; - leadership ability; - solving problems within the constraints of time, money, and human resources; - ability to transfer outside technology into the firm; and - networking within the company. The list was modified for graduates working in academia and other sectors. Graduates working in academia rated themselves slightly lower on these skills and capabilities in the final group listed above than did those employed in either industry or federal laboratories and other sectors. The general pattern of strengths and weaknesses across all groups, however, was essentially the same. Overall, ERC alumni in industry gave themselves a mean ranking of 4.32 in job performance compared to their peers. Supervisors assessed the overall performance of their ERC-trained employees slightly higher, at 4.35. In 12 of the individual skills and capabilities as well, the graduates ranked themselves slightly lower than their supervisors did. The biggest differentials occurred with three items on which the graduates rated themselves lowest: networking, transferring outside technology into the firm, and understanding the relationship between work and customer needs. Similarly, ability to work in interdisciplinary teams was rated notably higher by supervisors than by graduates: 4.27 compared with 4.11. Average ratings by ERC graduates working in industry were closest to those of their supervisors. In contrast, ratings by supervisors of those graduates working in federal laboratories and especially in academia were more favorable than those of the graduates themselves. Graduates in industry and their supervisors had slightly differing views of the amount of post-employment training the graduates needed to do their job. Using a five-point scale in which 1 meant "much less training needed than average" to become a net contributor to the company's work and 5 meant "much more training needed" than average, the mean rating by the graduates in industry was 1.96. Their supervisors provided a slightly higher mean ranking of 2.09; although they still placed their ERC- trained employees in the category of needing less training than their peers, supervisors did not see the difference as being quite as large as did the graduates themselves. Graduates' Most Valued Job Skills and Capabilities As Figure 4-1 shows, (see the following page) the sector of a graduate's employment had a considerable influence on the skills and capabilities that were deemed most useful in his or her present job. In the case of each skill or capability, however, there were individuals for whom it was very important. These results reflect variation in both the numbers of those for whom each was valuable and the extent of the value to each. (Similar patterns emerged in Section 3, "ERC-Industry Interaction," in discussion of the range and extent of benefit each firm received from interaction with an ERC.) ERC Experience and Job Performance Impact of ERC and non-ERC Graduate Experience on Job Performance One of the underlying assumptions of the ERC Program has been that significant involvement in ERC activities during graduate study builds upon, or is complementary to, traditional graduate education. To examine the extent to which this has been the case, graduates rated separately the impact of their ERC experiences and non-ERC graduate experiences on their performance in each of the 16 capability or skill areas listed in the preceding section. Using a 1-to-5 scale in which 1 was defined as "very negative" impact and 5 meant "very positive," graduates in the industry described the impact of their ERC experiences as ranging from neutral to somewhat positive, while their ratings for other graduate training experiences were slightly negative to positive. The impact of non-ERC experiences on those working in industry was most noticeable in six capabilities and skills: contribution to the company's technical work; breadth of technical understanding; depth of technical understanding; ability to grasp quickly key features of new problems; ability to define the steps needed to solve new problems; and ability to apply knowledge from different disciplines in an integrated fashion to solve problems. On average, the impact of non-ERC experiences on the graduates' job performance in these six areas was rated by those in industry approximately 0.25 higher than was the impact of ERC experiences. Performance in two other skills - creativity and innovation plus being able to communicate ideas in writing - also was slightly more affected by non-ERC graduate training. Conversely, ERC experiences had a greater impact on performance of the remaining half (8) of the skills and capabilities, all of which were consistent with ERC Program intentions. The four with the greatest difference between ERC and non-ERC experiences were: ability to work in interdisciplinary teams; ability to develop solutions that meet business goals as well as satisfying technical requirements; understanding the relationship between their work and the needs of the company's customers; and skill at networking in the company. The differences between ERC and non- ERC impact were smaller in the remaining four items: ability to communicate ideas verbally; ability to identify and transfer useful technology from outside sources such as universities or national labs; leadership ability; and ability to solve problems within constraints of time, money and human resources. Looking across graduates in all employment sectors, the impact of ERC experiences on these items was strongest among those employed in industry. Graduates also rated the overall impact of both types of experiences on their job performance. Non-ERC experiences were deemed to have had a higher overall impact on performance, compared with the impact of ERC experiences. Finally, looking at the ERC and non-ERC experiences across each of the 16 skills and abilities plus overall rating, the grand mean for non-ERC training was fractionally above that for ERC experiences. Given the greater amount of time students spent in non-ERC experiences while in graduate school, these difference are not surprising. Looking at the relative impact of ERC and non-ERC experiences on graduates across employment sectors, the ERC impact was greatest among those working in academia. An additional factor was the level of involvement of masters students in ERC activities. The finding is deceptively simple: the more extensively masters students engaged in ERC activities, the greater influence their ERC experience had on their subsequent performance in terms of ERC-related capabilities. While this is to be expected, it proved to be a major determinant of the overall influence of several ERCs on their graduates' capabilities.* Centers with similar participation profiles for masters and doctoral students have a much greater overall impact on graduates' performance than those centers with less masters involvement. Given that there have been more masters than doctoral students involved with ERCs over the course of the Program, this finding carries major implications. _______________ *A few ERCs operatein areas with no masters programs at their institution that are directly relevant for connections with the ERC. Thus, this finding applies to those ERCs that already involve masters graduates or could do so. Graduate School Activities Examination of what ERC graduates did while in graduate school sheds additional light on how ERC experiences add value to the rest of their graduate education. Masters and doctoral degree recipients in this study did substantially different things while in graduate school, and with notably different frequencies (see Appendix B). Of the 23 activities in which they might have engaged, only three were identified by more than half of those pursuing a masters degree while participating in an ERC they had done: while studying for that degree they worked on research projects with ERC faculty, attended one or more professional meetings, and took non-ERC cross-disciplinary or interdepartmental course. Note that there is a roughly equal balance between ERC-related and non-ERC activities in which 25%- 49% of the masters students engaged. Those pursuing a doctorate while at an ERC were split relatively evenly between ERC and non- ERC activities throughout the full range of possible activities. Note that the second most frequent activity for both masters and doctoral graduates was attending professional society meetings, and that in many cases respondents from both groups also presented papers at such meetings. While the graduates were not asked how the trips were financed, it is safe to assume that at least some of the graduates received funds from their ERC to attend such meetings. This is particularly likely if the individuals were making presentations about their ERC research projects. Looking across all of their graduate activities, ERC graduates engaged in a surprising amount of ERC-like activities outside of ERCs as well. Table 4-3 provides some comparisons for both masters and doctoral graduates in terms of rates of participation in ERC and non-ERC activities. _______________________________________________________________ Table 4-3: ERC and non-ERC Graduate School Activities of ERC Graduates Percent or Graduates Activity Masters Graduates Doctoral Graduates ERC non-ERC ERC non-ERC Collaborated with Corporate researchers 24 15 37 30 Made presentations to industrial audiences 23 24 50 55 Published papers with industrial co-authors 5 5 21 15 Industrial internships - 18% of doctoral - - 48 52 Cross-disciplinary/ cross-dept. courses 28 50 44 63 Presented talks at seminar series 18 25 45 60 _________________________________________________________________ The following section will examine the extent to which there are in fact differences in the impact of these seemingly similar ERC and non-ERC activities on graduates' subsequent job performance. Statistical Relationships Between Graduate School Experiences and Job Performance None of the results presented thus far dealing with the impact of ERC and non-ERC activities on graduates' job performance has been statistically significant. Thus, the possibility that they might be due to chance cannot be ruled out. However, more complex analyses (stepwise regressions) have identified statistically significant relationships between graduate school experiences and job performance. Looking first at ERC-educated masters degree recipients who went to work in industry immediately upon receipt of the degree, those who took at least one ERC-developed or sponsored course while pursuing that degree rated their communications and leadership abilities somewhat higher than did those who did not take such courses. Among those who received a Ph.D. and took a position in industry, there were several relationships between activities engaged in while pursuing that degree and subsequent job performance. Looking at ERC-linked activities, the following relationships emerged: - Those involved with ERC prototyping projects assessed their performance in a number of technical areas to be superior to that of peers. Specifically, 60% of those previously involved in such projects identified their job performance as "much better than average" in their contribution to their company's technical work. They also rated themselves comparatively highly on their overall ability to carry out their job responsibilities and their breadth of technical understanding. - Doctoral recipients who took ERC courses as doctoral students rated their ability to work in interdisciplinary teams significantly higher than did those who took no such courses. Over 40% of all doctoral recipients who took courses developed or sponsored by an ERC while pursuing a doctorate rated themselves "much better than average" in interdisciplinary team work. Conversely, only 28% of Ph.D. graduates who did not take any ERC courses rated their performance as highly on this dimension. - Ph.D. recipients who had co-authored papers with corporate sponsors of the ERC while in their doctoral program rated themselves relatively higher than did those who did not engage in such co-authorship in their ability to apply knowledge from different disciplines and their ability to transfer useful technology from outside sources. Several relationships between non-ERC activities and job performance also emerged. Specifically: - Those who gave talks in non-ERC seminar series and published papers with non-ERC corporate authors while in doctoral programs rated themselves significantly higher in two areas - their ability to communicate ideas both verbally and in writing and their leadership abilities - than did doctoral graduates who did not engage in these activities. - Ph.D. recipients who presented talks at non-ERC seminar series also rated themselves relatively higher than did those who gave no such talks on (1) a variety of technical capabilities and skills, (2) their ability to apply knowledge from different disciplines, and (3) their ability to identify and transfer useful technology from outside sources. ERC Aspects with Greatest Career Impact First, the ERC graduates who participated in this study looked back over their various experiences with an ERC and made their own lists of the aspects of their ERC involvement that had the greatest impact on their careers. Table 4-4 contains these aspects in order of frequency of identification by the graduates: ___________________________________________________________________ Table 4-4: ERC Features with Most Impact on ERC Graduates' Careers Percent of ERC Feature Respondents Internship in industry, "real-world" experience 31 Specific courses, faculty, advisors 23 Multidisciplinary; exposure to different disciplines 21 Good facilities and equipment 19 Teamwork 16 Research focus 14 Contacts, networking 12 Presentation, oral, or writing skills 7 Mention of a specific substantive area 6 __________________________________________________________________ Graduates' Recommendations for Program Improvement ERC graduates identified three generic improvements that would enhance the value of center involvement. Three-quarters of the ERC graduates now working in industry recommend that students have enhanced industrial involvement. Keeping in mind that less than a quarter of either the masters or doctoral graduates had had any industrial internships while in graduate school and that less than 30% of ERC graduates had been involved in collaborative projects with industry, there is clearly room for increasing the amount of students' active involvement with industry. The second area for improvement of the ERCs' value to graduate students is providing more experiences in making presentations and general training in communications. This is not surprising. Graduates rated themselves less highly in these areas than they did technical capabilities, yet supervisors identified communication and presentation skills as most important after technical competence. Third, ERC graduates identified a large number of specific courses in their home ERC that would benefit from improvements of various kinds. 5. FINDINGS AND CONCLUSIONS ERC-Industry Interaction: More is Better By examining specific impacts of the ERC Program, the two studies provided a critical look at the validity of the assumptions underpinning the Program. The ERC-industry interaction study shows in numerous statistically significant ways that company benefit is directly related to length and level of involvement in a center. Major impacts are generally unlikely to be realized in the first year or two of a company's interaction with a center. From their interaction with ERCs, companies receive a wide range of benefits that are improving their competitiveness and level of operation. Of these benefits, the most important are in the areas of: - Access to new ideas, know-how, or technologies; - Receiving technical assistance; - Interaction with other firms participating in the ERC; - Access to ERC equipment and facilities; and - Hiring ERC students and graduates. Those who hired ERC students and graduates rated this benefit higher than did recipients of any other outcome of ERC participation. In addition, two-thirds of firms experienced gains in competitiveness as a result of ERC participation and the majority of the respondents indicated that their ERC involvement had influenced their firm's research agenda. How much a company benefits is directly related to the number of years it participates in an ERC and the extent of active involvement its employees have with the center: the longer a firm participates and the more direct personal interaction there is between corporate and center personnel, the more direct benefits the firm will have received and the greater the effect on company competitiveness. The longer a firm has had active participation, the more likely it is to continue supporting the center for up to two years more. The key, then, is for centers to make participating firms understand that "sweat equity" is important for the partnership to be fruitful. The more centers can expose the industry representatives and their colleagues to the variety of ways in which they can be actively involved in the center, the more the firms, students, and centers will benefit. With the realization of benefits comes a greater likelihood that a firm will continue the collaboration for years. Sitting on the sidelines for three to four years before deciding to increase activity and get other members of a firm involved delays the receipt of results and threatens the longevity of partnerships. Three factors are strongly related to the realization of positive outcomes for companies from their interactions with an ERC: the existence of a "champion" for the ERC within the company, the receptivity of company technical staff to ERC ideas and/or results, and management support for the ERC partnership within the company. The biggest barrier to firms' participating in and benefiting from ERCs is that various other company priorities preempt firms from taking advantage of ERC involvement. Other major barriers to company participation are inherent in the differences in mission, values, and perception of project goals and timelines between industry and universities. These factors are largely out of the center's control. As a result, there may be a limit to what centers can do to ensure that firms get the most out of their involvement. Impact of ERC Experience on Graduates Regardless of employment sector, masters and doctoral degree recipients with ERC experience take the knowledge, skills, capabilities, and techniques they learned in ERCs with them to their subsequent jobs. Regardless of the nature of their jobs, ERC graduates continue working in ways they learned in ERCs, e.g., in interdisciplinary teams and by engaging in industry- university collaboration. ERC firms employing ERC students and graduates value this benefit of ERC interaction more than any other type of benefit. Supervisors and other industry representatives of firms employing ERC graduates judged the ERC- trained employees to be superior to non-ERC employees on a number of key performance dimensions. With the demand for ERC graduates by companies consistently exceeding the supply, it is evident that ERC graduates are living up to the Program's intention that they be successful in industry. Significant involvement in ERC activities during graduate study builds upon, or is complementary to, traditional graduate education. The skills and abilities in which ERC experience had more impact on performance - e.g., technology transfer, understanding the relationship between work and clients' needs, and ability to work in interdisciplinary teams - are consistent with the way ERC research and related activities are conducted and with the Program's goal of preparing engineers who will be able to work successfully in industry. The dimensions in which non-ERC experiences had a greater impact on performance - general technical competence and ability to contribute to the employer's technical work - are areas in which non-ERC training would have been expected to have more impact, given the longer time that graduate students spend in non-ERC education. Thus, ERC experience added to the breadth of capabilities of those who spent time in an ERC while in graduate school. While the ERC Program was a pioneer in developing this new mode of education, its goals for graduates are no longer unique to the Program. There are now other efforts underway at engineering schools designed to achieve the same goals. The growing prevalence of similar non-ERC educational programs and activities that have ERC-like characteristics may contribute to the relative difficulty in finding statistically significant differences between graduates' perceptions of the value of ERC versus non-ERC activities in their impact on their job performance. At this point, we do not know whether this is due to ERC students seeking ERC-like experiences, e.g., coops, etc. in their non-ERC education or whether it is due to a pervasive change in the engineering education experience for all students. NSF is looking at options for examining whether self-selection or changes in educational opportunities for graduate students may contribute to this situation. Ways to Increase the Impacts of ERCs The core of the ERC "culture" - interdisciplinary teams, research relevant to industry, direct involvement of industry in ERC activities - has produced a wide range of outcomes and benefits. Ultimately, corporate partners and students will benefit most from greater involvement of corporate staff in ERC activities. The adage "More is Better" is the main message of these studies. ERCs that approach their interactions with industry and graduate education from this perspective will go a long way toward ensuring that firms and students benefit from the ERC culture to the greatest extent possible. Firms can maximize the probability that they receive benefits that have demonstrable impact if : (1) their representatives to the ERCs are actively involved in the ERC; (2) other members of the firm also participate actively; and (3) this level of participation begins when the ERC-firm partnership commences. Corporations need more reasons than ever before to justify their ERC membership fees and related costs. Evidence of a firm benefiting from an ERC membership need not take many years. ERCs need to emphasize that sitting on the sidelines and watching what happens for four or more years decreases the probability that the kinds of benefits they need to justify future membership investments. While the ERCs intend to develop long-term relationships with firms and focus on long-term technical challenges, the results show clearly that firms begin to benefit early in their involvement - but only if their level of interaction with the center is considerable. Across masters and doctoral graduates, there is clearly a link between level of involvement of ERC activities that allow exposure to the ERC "culture" and subsequent attribution of the impact of such experience on their performance in areas that ERCs seek to foster. ERCs can increase their impact on these performance areas by broadening masters and doctoral students' participation in those activities and ways of conducting research that are distinctive. Since the establishment of the ERC Program, there has been a major growth in the number of industry- university research centers, many of which are now emphasizing interdisciplinary. Similarly, working in teams has become increasingly important within and outside academia. Students now have more options than just ERCs for such experiences. To make ERC involvement most meaningful and useful to them, they must have experiences that capitalize on the defining features of ERCs, especially their real-world, industry focus. Individual centers that include masters students must involve these students - not just doctoral students -- in the kinds of activities that typify an ERC, that can not be found in other centers or institutes in the engineering school. By doing these things, individual centers can maximize their impact on all levels of students, which translates to maximum impact on job performance. For continuous improvement of the ERC Program, results of these studies and findings relating to maximizing the Program's impact will be disseminated not only to the ERC Program Leader, but also to the individual ERCs at their annual meeting. In addition, the ERC Program Directors will be using this information to work with individual centers to implement changes reflecting the study results. APPENDIX A ENGINEERING RESEARCH CENTERS 1/CENTER FOR ENVIRONMENTALLY BENIGN SEMICONDUCTOR MANUFACTURING (1996) University of Arizona/Stanford University/Massachusetts Institute of Technology 2/ADVANCED COMBUSTION ENGINEERING RESEARCH CENTER (1986) Brigham Young University/University of Utah (in both studies) CENTER FOR NEUROMORPHIC SYSTEMS ENGINEERING (1994) California Institute of Technology 2/THE ENGINEERING DESIGN RESEARCH CENTER (1986) Carnegie Mellon University (in both studies) THE DATA STORAGE SYSTEMS CENTER (1990) Carnegie Mellon University (in industry interaction study only) CENTER FOR OPTOELECTRONIC COMPUTING SYSTEMS (1987) University of Colorado/Colorado State University (in both studies) 2/CENTER FOR TELECOMMUNICATIONS RESEARCH (1985) Columbia University (in both studies) CENTER FOR EMERGING CARDIOVASCULAR TECHNOLOGIES (1987) Duke University/University of North Carolina and other North Carolina Universities (in both studies) CENTER FOR PARTICLE SCIENCE & TECHNOLOGY (1994) University of Florida CENTER FOR LOW COST ELECTRONIC PACKAGING (1994) Georgia Institute of Technology 2/CENTER FOR COMPOUND SEMICONDUCTOR MICROELECTRONICS (1986) University of Illinois, Urbana-Champaign (in both studies) 2/ERC FOR ADVANCED TECHNOLOGY FOR LARGE STRUCTURAL SYSTEMS (1986) Lehigh University (in both studies) 2/INSTITUTE FOR SYSTEMS RESEARCH (1985) University of Maryland/Harvard University (in both studies) CENTER FOR BIOTECHNOLOGY PROCESS ENGINEERING (1985) Massachusetts Institute of Technology (in both studies) 1/CENTER FOR COMPETITIVE PRODUCT DEVELOPMENT (1996) Massachusetts Institute of Technology 1/CENTER FOR RECONFIGURABLE MACHINING SYSTEMS (1996) University of Michigan-Ann Arbor CENTER FOR INTERFACIAL ENGINEERING (1989) University of Minnesota (in both studies) CENTER FOR COMPUTATIONAL FIELD SIMULATION (1990) Mississippi State University (in industry interaction study only) BIOFILM ENGINEERING RESEARCH CENTER (1990) Montana State University (in industry interaction study only) CENTER FOR ADVANCED ELECTRONIC MATERIALS PROCESSING (1989) North Carolina State University (in both studies) 2/CENTER FOR NET SHAPE MANUFACTURING (1986) Ohio State University (in both studies) CENTER FOR COLLABORATIVE MANUFACTURING (1985) Purdue University (in both studies) 1/INTEGRATED MEDIA SYSTEMS CENTER (1996) University of Southern California OFFSHORE TECHNOLOGY RESEARCH CENTER (1989) Texas A&M University (in industry interaction study only) 1/UNIVERSITY OF WASHINGTON ENGINEERED BIOMATERIALS (1996) University of Washington CENTER FOR PLASMA-AIDED MANUFACTURING (1989) University of Wisconsin, Madison (in both studies) ________________________ 1/The ERC awardee classes of 1994 and 1996 did not participate in the surveys for these studies. 2/Will have graduated from the ERC Program and become self- sufficient during FY 1996 or 1997. APPENDIX B PARTICIPATION OF ERC ALUMNI IN GRADUATE TRAINING ACTIVITIES Percent of Graduates by Sector of Employment Activities as a Master's Student All Industry Other Academic Alumni Worked on research projects with ERC faculty 54.9 61.2 46.4 40.8 Attended one or more professional society meetings 51.0 55.6 44.6 40.8 Took other cross-disciplinary or interdepartmental courses 50.1 54.7 42.9 40.8 Based thesis on ERC-sponsored research project 39.3 43.1 30.4 33.8 Published papers in refereed journal 35.9 39.2 35.7 25.4 Took any courses developed or sponsored by an ERC 27.9 34.1 19.6 14.1 Attended more than half of an ERC seminar series 26.7 28.9 25.0 21.1 Made presentation at professional society meetings 25.6 27.2 19.6 25.4 Presented talks as part of non-ERC seminar series 24.8 25.9 25.0 21.1 Collaborated with researchers from ERC corporate sponsors 24.2 28.9 17.9 14.1 Presented research to non-ERC industrial audience 24.2 27.6 19.6 16.9 Presented research to ERC corporate sponsor 22.6 24.6 17.9 19.7 Published papers based on an ERC research project 21.4 25.0 16.1 14.1 Presented talks as part of an ERC seminar series 18.4 18.5 16.1 19.7 Worked on ERC-sponsored prototyping project 17.3 18.1 14.3 16.9 Collaborated with non-ERC researchers 15.0 16.4 14.3 11.3 Worked on research using ERC-based test bed 14.8 16.4 17.9 7.0 Published papers with non-ERC corporate co-authors 5.0 6.0 3.6 2.8 Published papers with co-authors from ERC sponsor 4.7 5.6 3.6 2.8 Applied for patent 3.1 3.4 4.2 Applied for patent based on ERC research project 1.1 1.7 Applied for patent with non-ERC sponsor co-applicants 0.8 1.3 Applied for patent with ERC sponsor co-applicants 0.6 0.9 APPENDIX B, Cont. PARTICIPATION OF ERC ALUMNI IN GRADUATE TRAINING ACTIVITIES Percent of Graduates by Sector of Employment Activities as a doctoral student All Industry Other Academic Alumni Published papers in refereed journal 88.1 91.0 79.5 86.3 Attended one or more professional society meetings 86.0 88.6 79.5 83.6 Worked on research projects with ERC faculty 85.3 87.3 79.5 83.6 Made presentation at professional society meetings 78.4 82.5 61.5 78.1 Published papers based on an ERC research project 63.7 67.5 48.7 63.0 Took other cross-disciplinary or interdepartmental courses 62.9 63.9 64.1 60.3 Based thesis on ERC-sponsored research project 60.8 63.9 48.7 60.3 Presented talks as part of non-ERC seminar series 59.4 62.0 41.0 63.0 Presented research to non-ERC industrial audience 55.4 62.0 53.8 41.1 Presented research to ERC corporate sponsor 50.4 55.4 43.6 42.5 Attended more than half of an ERC seminar series 48.9 51.2 43.6 46.6 Presented talks as part of an ERC seminar series 44.6 45.2 41.0 45.2 Took any courses developed or sponsored by an ERC 43.9 54.2 23.1 31.5 Collaborated with researchers from ERC corporate sponsors 36.3 42.2 30.8 26.0 Collaborated with non-ERC researchers 30.2 30.7 30.8 28.8 Worked on ERC-sponsored prototyping project 21.2 22.9 15.4 20.5 Published papers with co-authors from ERC sponsor 20.9 24.7 10.3 17.8 Worked on research using ERC-based test bed 20.1 21.1 20.5 17.8 Published papers with non-ERC corporate co-authors 14.7 18.1 15.4 6.8 Applied for patent 10.8 11.4 10.3 9.6 Applied for patent based on ERC research project 5.0 4.8 5.1 5.5 Applied for patent with ERC sponsor co-applicants 1.4 1.8 2.6 Applied for patent with non-ERC sponsor co-applicants 0.7 1.2 NSF 98-40