Skip to contents

National Science Foundation Logo

Search KDI Site

Banner: Multi-Disciplinary Research at NSF: Accomplishments of the KDI Initiative
 KDI Home    Contact Us   

Image: People Button

Image: Ideas Button

Image: Tools Button

 About KDI

 Behind the Scenes

 Taking Stock

 Links and Resources


Taking Stock of the KDI - Science of Evaluation

Image of scientists working on computerResearch, like any other tax-supported activity, must be accountable. This auditing function is not merely a matter of counting publications or patents. Metrics have to be developed that match the real goals of the research program. The KDI goals were to develop new ideas, innovative tools, and a science and engineering workforce that could collaborate in new, interdisciplinary areas. Each project also had its own specific goals, such as modeling the earth in three dimensions to better predict oil reserves. More generally, the National Science Foundation provides an interface between basic, civilian non-medical sciences and the public. Hence, no single metric would be sufficient to evaluate KDI, and any single metric would be misleading.

NSF sponsored a systematic evaluation of the KDI, not just to audit the program but also to serve as a diagnostic tool. What worked best, and what should be managed differently? To begin, researchers from all 71 KDI projects were invited to attend an NSF-sponsored workshop to discuss their experiences. The variety of positive outcomes that workshop participants described fell into four categories, as illustrated below.

New Ideas

New Tools

Student Training

Project Outreach

  • Museum community improved access to software
  • New collaborations with researchers at CONABIO Mexico
  • Partnership with Sun Microsystems to obtain computer system
  • Work with lawyers at the Courtroom 21 project
  • Collaboration with IBM Watson Research Center
  • Formation of alliance and development of transfer technology
  • Meeting of researchers via seminar series
  • Partnership from Pfizer as project spinoff
  • Strengthening of relationships with government community
  • Deployment of project software in the IT industry
  • Supervision of talented high school students
  • Formation of new community around project results
  • Formation of close ties with IBM and Intel

As these lists suggest, the program as a whole met its goals. What predicted success of the individual project? Jonathon Cummings, on the faculty at MIT and a former KDI postdoc, conducted a systematic survey of grantees to find out. He found that dispersion rather than multi-disciplinarity was the most problematic aspect of KDI projects. Projects with PIs in more universities were significantly less well coordinated and reported fewer positive outcomes. Coordination mechanisms that brought researchers together (such as holding a project-related conference or workshop) appeared to reduce the negative impact of dispersion.

One major difficulty still stands in the way of realistic accountability. Much innovative research takes a long time to develop, and its consequences may not be known for years. Evaluations need to be used with great care to ensure that the scientists are busy doing long-range science and taking risks, rather than working for short-term reporting goals.

Back to Top of Page

People | Ideas | Tools
About KDI | Behind the Scenes | Taking Stock | Links and Resources

KDI Home | Contact Us | Site Map
NSF Home | CISE Home | Privacy Statement | Policies | Accessibility

National Science Foundation: Celebrating 50 Years Logo The National Science Foundation
4201 Wilson Boulevard, Arlington, Virginia 22230, USA
Tel: 703-292-5111, FIRS: 800-877-8339 | TDD: 703-292-5090