EFFECTIVE PROCESSES TO GIVE ENGINEERING EDUCATORS EASY ACCESS TO QUALITY-REVIEWED ELECTRONIC COURSEWARE
|
Chair:
|
Alice M. Agogino
Associate Dean of Engineering University of California at Berkeley Director, Synthesis Coalition
Pamela A. Eibeck
Brandon Muramatsu
Judith L. Stern
|
This workshop focused on the desirable characteristics of an electronic, multimedia database of engineering courseware to serve the needs of engineering educators for easily-accessible, quality-reviewed material of this nature. Much of the discussion centered around an existing system that appears most nearly to approach this goal at present -- the National Engineering Education Delivery System (NEEDS) database, developed in prototype form through the Synthesis Coalition. However, NSF expects that future databases of this nature will go well beyond the current NEEDS structure and probably become a part of the Digital Library Initiative (DLI) now under development with support from NSF, NASA, and DARPA.
THE STATE OF THE ART
Workshop chair Alice Agogino led off by describing the two main themes of the workshop: first, what means are available to access and disseminate engineering education courseware; and second, how can quality evaluation and peer review of engineering education instructional software be accomplished?
The purpose of the workshop was to explore the desired features of a centralized, searchable, online repository of engineering education instructional software, and in that light to examine the utility of NEEDS as an example of such a clearinghouse while obtaining feedback from the workshop participants on the specific features of NEEDS and the quality-evaluation procedures currently being employed or envisioned for it.
Brandon Muramatsu presented an overview of the current state of access to engineering education instructional software, focusing in particular on the emergence of the World Wide Web (WWW) as a publishing and information source. With the exponential growth in the number of users of the Internet and the WWW, this is the most promising medium. (More than 37 million people in the United States and Canada had access to the Web in 1996, and the number is growing daily.) Access options include the use of search engines and/or directly accessing the various Web pages of specific colleges of engineering, faculty members, and even individual courses.
However, this approach is too random and time-consuming. (For example, searching the keywords "engineering AND multimedia" produces more than 100,000 "hits.") Far preferable would be online access to a centralized, searchable database of such software. There is a need for such a clearinghouse in engineering education. To gain objective input on that need, Muramatsu passed out a questionnaire, attached to this summary as Attachment A. Participants were asked to fill out the questionnaire and hand it in.
Mr. Muramatsu described NEEDS (located at http://www.needs.org) as a "one-stop shop" based on the library metaphor, containing: (1) bibliographic records with downloadable courseware, and (2) multimedia elements including movies, images, and text. Searches can be conducted over both types of entry. NEEDS incorporates a multilevel courseware evaluation system with peer review and also a national competition called the Premier Award (sponsored by John Wiley and Sons). NEEDS allows a user to search, browse, display, and (in contrast to an online catalog system) download courseware. NEEDS can be used by anyone with a standard Web browser and Internet-connected computer. Courseware downloadable through NEEDS ranges from complete author packaged courseware to "samplers" of large courseware (i.e., courseware that usually comes on a full CD) to "ticklers" of commercially available courseware. In addition, multimedia elements -- in some cases the individual movies, images, and text that are used in a specific courseware package -- are available for download.
NEEDS is a prototype model for a possible larger system. While the NEEDS database originally was developed to support dissemination of Synthesis-developed materials, it is expanding to include courseware developed by the other NSF-funded Engineering Education Coalitions.
QUALITY REVIEW AND CRITERIA
Quality review of instructional software ensures that a clearinghouse becomes a "living system." For example, the quality review of NEEDS courseware has two purposes. The first is to establish the credibility of NEEDS as a source of quality educational material. The second purpose is to gain enhanced recognition for the scholarly and creative effort of courseware developers. The quality review of courseware is divided into a peer review system based on the journal model and the Premier Award, a national award competition. The peer review is a gestalt review that considers the following major criteria:
The Premier Award is a competitive, annual award recently developed to recognize the development and use of outstanding engineering education instructional courseware. (The first Premier awardees will be announced in late 1997.) The term "outstanding courseware" recognizes not just the courseware itself but also the pedagogy, instructional objectives, and use in an instructional setting as defined by the author. The Premier Award involves a rigorous examination of courseware that explores student learning in depth. The three criteria used in Premier Award judging are:
To obtain input from the group on these criteria, the chair handed out a questionnaire that elicited their responses regarding the relative importance of aspects of the content, software design, and instructional design. This handout is given as Attachment B. Group discussion of these criteria led to a number of suggestions for ways to improve the criteria and the database.
It was agreed that the database is only a skeleton. It might contain all the engineering courseware in the world, but without strong quality review the utility of the repository remains limited.
The consensus was that there should be more focus on content -- i.e., include a question such as "Is the engineering technical content comprehensive and rigorous within the context of the defined objectives of the courseware?" The presenters agreed; they explained that the criteria for the Premier Award are weighted more heavily toward instructional design than content because the award is intended to encourage more "up-front" use of instructional design.
With regard to Software Design criteria, suggested additions include:
The criteria form itself (Attachment B) should specify up-front that, while the ultimate purpose of the criteria is to identify an award-winner, judges should nevertheless apply the questions from a functional perspective, as if the courseware were "for use."
The form should define "courseware" up-front. A suggested definition is "a relatively complex package of material designed to convey knowledge content to the learner in an area of the curriculum." It was suggested that the minimum size of an item that could be classed as courseware is a "unit" covering at least two days of instruction.
A wide range of learning materials should be available on the database, including tutorials, demonstrations, modules, and both large-scale and small items. NEEDS operators may eventually have to categorize the various types of courseware and apply different criteria depending on the size and purpose of the material.
The wording of criteria should be unambiguous. For example, in question 3.1 regarding opportunities for student input (under Instructional Design, "Interactivity"), the form should specify whether this refers to the interactivity of the software itself or interactivity with the pedagogical content.
The need for easily recognizable coding of reviews. One suggestion was a graphic of 1 to 4 "thumbs up," like a movie or restaurant review. Such a coding should not be too simplistic; it would have to be done from several standpoints (e.g., ease of use, learning potential, "integratability," etc.) Another suggestion to improve users' ability to quickly assess the quality of courseware is to provide space for viewable user feedback -- i.e., a forum, kiosk, or bulletin board. One idea is to develop a users' group, in addition to the standards group which already exists, to provide formal input directly to NEEDS developers.
It is difficult to find qualified reviewers of courseware. This field is not yet at a point where reviewing such materials contributes toward promotion and tenure. One likely venue for making progress here is the Journal of Engineering Education. It was suggested that the reviews of top-rated materials for the Premier Award could be published in JEE, either as a "signed book review" or as an edited composite summary of the reviews on a given item of courseware. More broadly, it would be very useful if educational software/instructional technology could develop as a new subcomponent of the traditional engineering professional journals. It was noted that there are already a few fully electronic journals; these would seem to be likely candidates.
NEEDS DATABASE FEATURES
Based on the overview of NEEDS features provided by Muramatsu, using an online demonstration, a number of issues and suggestions were discussed regarding aspects of the database and the system itself.
The issue of keeping a database such as NEEDS up to date over time is problematic. Engineering content changes, platform obsolescence, the emergence of new platforms, and the addition of new courseware all pose problems for NEEDS users. Allowing users to access the database and find the materials they are searching for and that are relevant is an on-going information retrieval problem. The NEEDS operators plan on continuously upgrading the search tools and adding filters to help users solve this problem. It was suggested that they might want to add a date window (filter) for archiving courseware.
There is variability across engineering fields in terms of whether users most often look for new, exploratory courseware versus helpful incremental improvements to traditional courses. Perhaps this variability should be reflected more explicitly in the choice of materials for NEEDS.
Based partly on the questionnaires that participants had filled out earlier, they were asked to discuss and identify features of an instructional software database that would be necessary or useful. Those identified include:
Most of the features identified already are present in NEEDS. Those that are not are identified with an asterisk.
Control of intellectual property rights in NEEDS is an issue that has not yet been fully resolved. Authors of included courseware must sign an affidavit stating that they have not violated any copyright. However, NEEDS operators do not police the use of the freeware in terms of users modifying it and then selling it. They will inform the author if they happen to become aware of such unauthorized activity. Firmer policies are needed. It was suggested that a standard intellectual property statement be included on the package, stating that it is copyrighted and can be used but not commercialized.
Several of the participants expressed a willingness to log on to NEEDS, explore the database, and provide feedback and/or be part of the peer review process. The general consensus was that NEEDS is a very useful concept to pursue as one approach to the development of electronic means for dissemination of engineering educational innovations.
|
|