Undergraduate Curriculum Development Testbed

Fall Term 1996 Evaluation Team Report

Grant Timeline: First Semester

Report Date: May 1, 1997

Introduction

The Undergraduate Curriculum Development Testbed (UCDT)was created to develop interdisciplinary course sequences in global change, define systematic barriers to a major expansion of interdisciplinary instruction and interdisciplinary curricula, articulate means of eliminating those barriers, and develop templates and modules for interdisciplinary course development that can be exported to other academic units. This report will primarily focus on an evaluation of Global Change I--the first course in the University-wide introductory sequence in global change--and the activities of the UCDT curriculum development group.

The evaluation team (hence forth known as the Team) utilized a formative evaluation approach to investigate UCDT. It is our intention to use formative evaluation to provide feedback to the curriculum development group (hence forth known as the development group) to help with them with planning and decision making. To provide helpful information to the development group, the Team measured student performance and evaluated the educational technology. After reviewing the background of the project, this report outlines Global Change I objectives, discusses the program activities and outcomes, and articulates several recommendations.

Background

The University of Michigan is a Research I institution that provides educational opportunities to approximately 24,000 undergraduate and 12,000 graduate students. The University of Michigan is a highly decentralized educational environment that is comprised of 17 separate schools and colleges.

In early 1993, an interdisciplinary group of faculty came together to discuss the need to develop a sequence of undergraduate courses which were focused on investigating environmental changes on a global scale. In their estimation, all undergraduate students at the University of Michigan should learn about the role humans play in the global change process, so they can take an intelligent, active role in the worldwide debate about the environmental problems facing the international community.

To address this need, these faculty members created the University of Michigan Global Change project. These faculty felt that in order to provide undergraduate students with the skills and knowledge they would need to comprehend and be intelligently conversant in the international discussion of the impact of worldwide environmental change, the faculty would need to construct a new, truly interdisciplinary course structure. During 1993-1994, two new courses were constructed and approved for the general curriculum. These courses were named Global Change I and Global Change II.

In December of 1995, an interdisciplinary group of professors from six schools and eight departments at the University of Michigan took steps to enhance the Global Change curriculum. These professors created UCDT to enhance the Global Change sequence of courses and to accomplish other institutional goals. UCDT received grants from both the National Science Foundation (NSF) and the University of Michigan. These grants provide funding for a period of two year beginning on July 1, 1996.

Program Objectives

A fundamental goal of the UCDT project is to establish improved methodologies for the infusion of interdisciplinary courses and materials into the undergraduate programs at the University of Michigan. Additional objectives of the UCDT are

to: 1) further enhance existing interdisciplinary courses in global change and develop additional courses to provide a university-wide introductory sequence in global change; 2) develop viable templates for implementation in other university settings; 3) promote more widespread deployment of interdisciplinary courses both on and off campus; and, 4) assess the viability of new teaching credit schemes for interdisciplinary undergraduate teaching.

The primary focus of this report is to evaluate the course related activities. The development group created plans and activities to improve the existing introductory course sequence, Introduction to Global Change, Parts I and II, by accelerating the development of computerized, multimedia laboratory modules, Internet resources, and lecture materials. In addition, plans were made to build a new NSF-sponsored "Global Change Laboratory" to provide well-structured opportunities for hands-on experimental and data analysis experiences for undergraduates.

Program Activities

The development group started to work on enhancing Global Change I last summer. During the summer it was decided that the pre-release version of STELLA, a software package which allows students to graphically build and control dynamic models, would play a large role in the laboratory activities. The development group hoped that STELLA would serve as a useful and flexible introduction to how computers may enable global change investigators to model real-world problems and situations. Currently, all predictions which scientists use for estimating the impact of environmental change on the Earth's future are based on dynamic models, like STELLA.

The development group made plans to create lab assignments during the course of the semester. These assignments were intended to help students understand the complex connections and dynamical relationships between environmental components. Once these laboratory exercises were developed, they are posted to the Global Change web page. Course lectures are also posted to the web page.

Text, graphics, pictures, and animation were packaged on the web to provide students with a stimulating visual learning environment. To add depth to the course structure, a number of informational links were researched and connected to the lectures notes to provide students with opportunities for additional information.

New Global Change I Course

The objective of developing additional course structures for a university-wide global change sequence began in December 1996 with an organizational meeting. Conceptual frameworks for a Global Change III course and a possible capstone course were discussed. The Global Change III curriculum development group designed the course to explore the impact of humans on the environment. Special attention was to be given to the sociopolitical and economic consequences of human population growth and consumption patterns. Currently, the evaluation team estimates that the first Global Change III course will be offered in the Winter Term of 1998.

Global Change Laboratory

The redesigned Global Change Laboratory (GCL), sponsored by NSF, was scheduled to open in April 1997. The opening of GCL has been delayed due to an asbestos clean-up problem at the facility. When the lab is opened, students will have the opportunity to conduct laboratory experiments utilizing state-of-the-art remote sensing equipment.

Project Faculty

* Principal Investigator for the Undergraduate Curriculum Development Testbed is Professor Timothy L. Killeen, Director of the Space Physics Research Laboratory and Professor of Atmospheric and Space Sciences at the University of Michigan.

* Dr. J. David Allan is a professor from the School of Natural Resources and Environment at the University of Michigan. Dr. Allan is responsible for the ecosystem dynamics aspects of the UCDT's interdisciplinary curriculum.

* Dr. James Teeri is a professor from the Department of Biology and Director of the Project for the Interdisciplinary Study of Global Change at the University of Michigan.

* Dr. George W. Kling is an assistant professor and assistant research scientist in the Department of Biology at the University of Michigan. Dr. Kling coordinates the biogeochemical cycles components of the UCDT's interdisciplinary curriculum.

* Dr. Eric L. Dey is an assistant professor and a research scientist at the Center for the Study of Higher and Postsecondary Education (CSHPE) in the School of Education at the University of Michigan. Dr. Dey serves is responsible for the evaluation and assessment component of the UCDT project.

Project Staff

Graduate Assistants who worked on the UCDT project during the fall of 1996:

* Bridget Fahey was the graduate student instructor for the Global Change lab. Bridget is a master's degree student in the School of Natural Resources and Environment at the University of Michigan.

* Sean Cash was the research assistant who manages the Global Change web site. Sean is a master's degree student in the School of Natural Resources and Environment and the Department of Economics at the University of Michigan.

* Geeta Jain was the graduate student instructor who developed the lab assignments. Geeta is a master's degree student in the School of Information at the University of Michigan.

* Ravi was the graphic artist responsible for creating images to be included in the lectures, laboratories, and on the Web-page.

* Joseph M. Fenty assisted the evaluation team develop evaluation instruments and write evaluation reports. Joseph is a doctoral student at the Center for the Study of Higher and Postsecondary Education (CSHPE).

* T. Gregory Barrett was responsible for UCDT project coordination and development. Greg is a doctoral student at the Center for the Study of Higher and Postsecondary Education (CSHPE).

Evaluation of Global Change I

The overall objectives of the Global Change curriculum are to educate students about global change issues and to collect information about interdisciplinary teaching. It is the intention of the curriculum developers to posit a model of interdisciplinary research and teaching which will contribute to undergraduate curricula reform. The extent to which the Global Change project meets it objectives must be determined through a program of rigorous assessment and evaluation. Toward this end, the evaluation team has created a multifaceted evaluation program to assess the educational outcomes of the Global Change curriculum. Global Change I was the first course to be assessed. The course objectives methods, results, and conclusions are summarized in the following sections.

Objectives of Global Change I

The course objectives for Global Change included the following:

1. To improve understanding of the interdisciplinary scientific underpinnings for the study of Global Change.

2. To study the evolution of the physical world in order to better appreciate the temporal and spatial scales of change that have occurred in the past as well as those that might occur in the future.

3. To understand why Global Change studies require a system perspective in which many interacting components must be described.

4. To become better equipped to contribute to the important debates concerning global resource management, environment, environmental impact, and societal adaption strategies.

5. To learn how to use the vast resources of the web to find and use environmental information.

6. To learn how to develop simple dynamical models of earth system processes and to understand the importance of physical models of complex systems as well as their limitations.

7. To learn about the inadequacies in the data and knowledge regarding Global Change and the emerging strategies to improve our state of knowledge.

These objectives are a good beginning, but they needed to be sharpened to bring clarity to the curriculum development and evaluation process. The language used by the objectives, and the lack of operational definitions, mitigated evaluation activities. Efforts to clarify these objectives by the Team were minimally successful during the fall semester due to the need to quickly develop the Global Change I curriculum.

Global Change I began without many of the labs and lectures fully prepared. These curricular components fell into place as the semester progressed. Unfortunately, the lack of time to prepare the lab assignments and schedule guest lecturers created a substantial amount of pressure on the curriculum development group to quickly devise a curriculum. Often the pedagogy, lecture and lab topics, and instructional assignments emerged weeks before the actual instructional event. The development group had limited opportunity to come to consensus and to integrate the course objectives into the curricular experience for students. Nevertheless, as the semester progress the development group become more facile at creating and executing the curriculum.

The evaluation team developed assessment instruments and analyzed data to determine the impact of Global Change I on students. The Team used the Early Student Feedback assessment instrument (designed by the University of Michigan's Center for Research on Learning and Teaching), midterm and final assessment surveys, and classroom assessment techniques to collect information about student attitudes, experiences, and perceptions. The Team did not create tests to measure intellectual or cognitive development. The development group developed and administered tests to measure student learning.

CRLT's Early Student Feedback

The Early Student Feedback assessment instrument was designed to help faculty get student feedback about their teaching early in the term. The technique allowed the use of small group discussions to gather information directly from students in a format that permits interaction and clarification. Students were asked to list the major strengths of the course and the changes which would improve the quality of the course.

The Early Student Feedback instrument was administered on Wednesday, October 16, 1996 (approximately six weeks into the term). In small groups, students were asked to reach consensus on the following questions and directions:

1. List the Major Strengths of the Course. What is helping you learn?

2. List changes that could be made in the course to assist you in learning. Please explain how suggested changes could be made.

Early Student Feedback Results

Students indicated several strengths of the course. All of the student groups communicated that the handouts were very helpful. As one student group put it, the hand outs were "detailed, interesting, and easy to understand." The student groups also reported that the visual aids helped them learn the material. One group reported this, "Visual learning not only complements the learning, but makes it more interesting too."

Overall the student groups were pleased with the instructors and the instructional process. One student group reported that, "he [lecturer] is very friendly and knowledgeable. He is soft spoken, not intimidating or boring, and conversational." Many of the student groups also reported that the instructors made effective use of the instructional materials.

After addressing the area of strengths, the student groups were asked to talk about changes that could make the course better. The major concerns expressed by students can be grouped into three categories: labs, examinations, STELLA, and the Global Change web site.

Several structural issues affected the lab experience for students. First, enrollment of one of the lab sections proved to be too large to allow for smooth use. Crowded conditions in the second lab reduced the time students could individually use the computer as well as the time the GSI could spend with individual students. Second, a scheduling error by the Registrar's Office lead some students to believe that the lab would be one hour instead of the required two hours. Due to this problem, a sizable number of students had to leave midway through the lab to attend other classes. These students were required to make-up the lost lab time on their own time. Third, the lab assignments were created as the semester progressed. Shipment of the pre-release version of STELLA was delayed. This situation effected the ability of the instructors to prepare lab assignments. On several occasions the lab assignments reach the students with glitches that the GSI had to fix as the lab was being taught. Fourth, the labs depended on a complex program called STELLA --a dynamical modeling program--that was not available in final form at the start of the term. As a result, the lab activities were designed around the pre-release version of the software. This version of STELLA had some problems such as the inability to save work, problems printing, and occasional system crashes.

Many students complained that the labs and lectures were not fully synchronized. One representative comment called for "Smaller [lab] with more individual help. [Labs that] correspond with the lecture. [A lab instructor who is] more sympathetic to those not computer literate or divide section according to computer literacy. Integrate them [the computer novices] into class."

Student groups made several important comments regarding STELLA. Many of the student groups did not understand the connection between these instructional tools and the lecture. One student group said this, "STELLA and the general format of the labs need to be altered. We don't feel like we learn anything in lab, i.e., we copy the numbers off the paper (plug into the computer), but don't really know what they mean or how they relate." Another student group declared that STELLA is just "busy work" and that "dynamic modeling is a waste of time."

Early Student Feedback Summary

Students indicated that they were pleased with the handouts, visual aids, and lecturers. When they were asked what changes would improve learning, students suggested action be taken to improve the lab scheduling process and that the overcrowded conditions in the lab be alleviated. Students also requested that the lab assignments be relevant to the lecture topics as well as better developed. Furthermore, students indicated problems were associated with the dynamical modeling program called STELLA be remedied.

Technical problems with STELLA as well as accompanying lab assignments probably reduced the potential of students "to learn how to develop dynamical models of Earth system processes; to understand the importance of physical models of complex systems as well as their limitations (course objective number 6)." Many of the problems with STELLA have already been fixed by the software developer. We anticipate the next set of STELLA labs will achieve their full learning potential.

Midterm Assessment

The main purpose of the midterm assessment (see the appendix for the midterm instrument) was to collect information from students about their expectations, plans for the future, perceptions of the lecture notes, and experiences with the lab and lecture. In collaboration with the curriculum development group, we created a midterm course assessment instrument. Many of the questions that appeared on the assessment instrument were taken from the question bank developed by the Office of Evaluations & Examinations. The results indicated that students found the lecture notes useful and the labs less so. Students also reported they learned a good deal of factual material and their interest and enthusiasm in the subject matter grew.

Midterm Assessment Results

The percentage of men (53.4%) and women (46.6%) taking the assessment instrument was approximately the same (see the Appendix for a copy of the midterm questions). The majority of the students are sophomores (41.4%). Together sophomores and juniors accounted for 75.9% of the students.

During the course of the semester, the curriculum development group made a substantial and conscientious effort to post the materials presented in the lecture to the class web site. To gauge the usefulness of the lecture notes, the midterm contained several questions which addressed this issue. Students were asked about the level of detail in the lecture notes, the possibility of substituting the lecture notes for attending class, and the usefulness of the notes. More than 50% of the students reported that the lectures notes were especially detailed relative to other U of M courses, nevertheless, they were not a substitute for going to class. More than eight of ten students reported that the lecture notes were a valuable part of the course.

Students were asked several questions to investigate their lab experiences. Less than half of the students agreed that the lab assignments seemed carefully chosen. Concomitantly, the large percentage (31.0%) of students who said they were neutral on this question seemed to indicate that many have not quite made up their mind about the usefulness of the lab assignments. Surprisingly, 58.6% of the students disagreed with the statement that the laboratory assignment made an important contribution to their understanding of the topics discussed during the lecture. Many of the lab assignments depended on STELLA--a dynamical modeling program. Students are given assignments that require them to model complex environmental and biological phenomena. Students were asked to indicate the degree to which they agreed with the following statement, "I feel confident in my ability to use the program STELLA to construct models." Less than half of the students agreed with this statement; and three of ten students actually disagreed with this statement. Students were also asked to respond to the following statement, "The program STELLA has helped me to understand how many interacting components create global phenomena." While nearly 60% agreed with this statement, approximately 30% did not.

Students were asked about the degree to which they were learning in this course. About 9 of 10 students indicated they learned a good deal of factual material as well as gained a good understanding of the concepts/principles in the field of global change. More than 7 of 10 students agreed that the course had both improved their ability to participate in debates about global change and encouraged them to think critically about global change issues. Furthermore, approximately 7 of 10 students reported that the course had deepened their interest in the subject matter and that they were enthusiastic about the course material.

The Global Change (GC) development group has made a substantial investment in a web site. The web is the focal point for many Global Change curriculum activities. The GC web site contains the lecture notes, lab assignments, course outline, and links to other web sites. Students were asked about the degree to which the GC web site was well organized as well as whether it contributed to learning. Nearly, 95% of the students reported that the GC web site is organized so that it is easy to locate information. Furthermore, 6 of 10 students indicated that the GC web site made a significant contribution to their learning. Nearly all of the students indicated that they enrolled in Global Change I because it fulfilled their need for natural science credit. As one student put it, "I wanted an interesting science course to complete my natural science requirement." Several students reported that the course had been recommended by a colleague. Three students said the course had some connection to their career interest.

Students were asked what they hoped to learn from this course and if these expectations are being met. Nearly one-fourth indicated that they had no expectations. Approximately half of the students said their expectations were being met. One in five students reported that the course has thus far been different from what they expected.

Classroom Assessment Techniques.

The evaluation team introduced classroom assessment to the faculty and GSIs. These techniques, pioneered by Angelo and Cross (1986), are designed to (a) assess course-related knowledge and skills, (b) assess learner attitudes, values, and self-awareness, and (c) assess learner reactions to instruction. Three techniques were considered due to their low time and energy requirements: the One-Minute Paper, Muddiest Point, and One-Sentence Summary . Only the One-Minute Paper was used consistently throughout this course to assess the results of the lab.

To use the One-Minute Paper, an instructor stops class two or three minutes early and asks students to respond briefly to some variation on the following two questions: "What was the most important thing you learned during this class? and "What important question remains unanswered?"

One-Minute Paper Results

The results from this technique collect information on a large variety of topics. When the instructional process in the labs progressed smoothly, student responses like "I am finally beginning to understand STELLA" were not unusual. Unfortunately, more often than not, things did not go well in the lab. The frustration with STELLA was consistently expressed in the One-Minute Papers. For example, students returning from a field trip to the museum appeared to appreciate the break from using STELLA. One student said this, "I learned first hand morphological differences. I enjoyed the lecture beforehand which prepared us also. This lab broke up the monotony of STELLA." Another chimed in with " . . . I just wanted to say that I think that in today's lab [discussion of lecture] I learned more than in all our STELLA labs combined. It was very interesting and we should definitely do more labs like this."

Final Assessment Instrument

The final instrument was designed to parallel the midterm assessment instrument to measure the impact of the instructional experience. Additional questions were added to the final assessment instrument to gauge the effects of team teaching, and to determine interest in taking another Global Change course (see the Appendix for a copy of the final assessment instrument). Students were also asked to share their perceptions of the instructor's effectiveness, lab assignments, and educational technology.

Final Assessment Results

The students that participated in both the midterm and final assessment have similar demographic characteristics. Given the size and characteristics of the sample we can safely assume that the respondents for both assessment instruments are essentially the same.

Slightly more females participated in the survey than males. More than 70% of the students were sophomores and juniors, 10% were first-year students and the remainder were seniors. Students were asked if they were interested in taking another global course, more than 8 of 10 students indicated than they would consider it.

Students were asked several questions about their laboratory experience. About 4 of 10 students felt the lab was excellent, but one-third disagreed. Opinions about the lab instructor improved over the course of the semester. Several student groups which participated in the Early Student Feedback indicated that the lab instructor had difficulty answering questions and explaining concepts. Some of these difficulties probably stem from technology problems as well as poorly constructed lab assignments. At the end of the term, 7 of 10 students reported the lab instructor was excellent. Nearly half the students indicated that they learned a great deal from the lab, but 19.3% disagreed.

Comparison of Midterm and Final Assessment Results

The evaluation team conducted an analysis of the midterm and final assessment results. Figure 1 (See Comparison Table) reports the percentage of the respondents who responded "agree" (combination of students who responded strongly agree and

agree ), "neutral", and "disagree" (combination of students who responded strongly disagree and disagreed) for both the midterm and final assessment survey items. In all 17 questions appeared on both instruments.

By the end of the term, slightly more students felt that the lab assignments helped them understand the lecture topics (question 11 - final survey). Nevertheless, the percentage of students who held this position was only 26.3%. Although the percentage of students that reported that the lab assignments did not help them understand the lecture declined by 18.3%, the plurality of students taking the final survey (40.3%) still maintained that the lab did not make an important contribution. The data suggest that the nexus between the lab and lecture is tenuous, at best. The optimal linkage between the lab and lecture would create overlapping learning experiences which would, in effect, scaffold learning. The degree to which learning is scaffolded should depend on the learning challenge. Concepts that are very important and complex require more scaffolding or more planned learning experiences than those that are less complex. Other important factors which could have contributed to the estranged relationship between the lecture and lab include: software program (STELLA) failures, scheduling difficulties (large number of students had to leave midway through the lab session), and poor constructed lab assignments.

Student opinions about the usefulness of the web-based lab notes became more positive by the end of the term (question 12 - final survey). Nevertheless, those taking this position totaled less than 50%. The large percentage of students who chose to be neutral (45.6%) indicates that the lab notes should be reevaluated. It seems safe to assume that putting the notes on the web did not diminish their usefulness, but rather their content seems to be the problem.

The percentage of students (question 14 - final survey) indicating that the web made a significant contribution to their learning experienced a large increase (16.6%). By the final, nearly 8 in 10 students agreed with this position. Ostensibly, web-based instruction was well received by the students. The course objectives indicate that students should learn how to use the vast resources of the web to find and use environmental information. The development group has not articulated specific web skills which student should acquire as an instructional outcome. Therefore, it is difficult to say with certainty if this course objective has been achieved. Nevertheless, the large number of students reporting that the web contributed to their learning is encouraging. Ostensibly, the investment in the web has yielded dividends.

The number of students who said that STELLA had helped them understand global phenomena nearly doubled (question 17 - final survey). Furthermore, the number of students saying STELLA had not helped them declined by 31.8%. Although this is good news, it is important to note that those agreeing with this position totaled slightly more that 50%. The rest of the students were either neutral (21.4%) or they disagreed (26.8%). These last sets of statistics indicate that a large number of students did not find STELLA to be a very useful tool for understanding global change. During the fall semester there were numerous problems with STELLA. The problems took the form of system crashes, poorly written lab assignments, and the inability to save one's work. The rise in the favorable rating for STELLA is probably associated with the use of a newer version of the program. This new version fixed many of the technical problems experienced earlier in the semester.

By the end of the term, about 9 of 10 student reported they had deepened their interest in global change topics (question 20 - final survey). This represents a 15.3% increase in the percentage of students reporting this opinion on the midterm survey. Many of the effects of this course lie beyond the scope of this term. Indeed, it is the intention of the curriculum designers that students continue to be interested in global change issues through their life. High levels of interest in global change sets the stage for continued activity in this area, and consequently, the realization of this goal.

As the term progressed, students became more confident in their ability to debate global change and environmental issues (questions 22 & 23 - final survey). Nearly 8 of 10 students took this position by the final, this represents an 11.8% increase since the midterm. Consistent with this increase, there was a 10.2% increase in the percentage of students that reported the course encouraged them to think critically about global change issues.

Apparently, as the semester progressed students found opportunities to actively participate in class discussions (question 26 - final survey). The research about learning tells us that as active participation increases, the likelihood that students will learn and remember more also increases. Although it is encouraging that 14.6% more students reported that they actively participated in class discussions, the total percentage is still less than 50%. This means that half of the class are probably missing out on important learning opportunities. Additional opportunities for learning need to be planned so that future semesters the potential for active learning will increase.

Several questions appeared on the final assessment survey which were unique. For example, the final survey contained two questions about team teaching. Students were asked what effect did team teaching have on their ability to learn.

Only one-quarter of the students (question 24 - final survey) agree that team teaching contributed to their understanding of the global change concepts and principles. Nearly half of the students were neutral, and 28% actually disagreed. When students were asked if the transition from one instructor to the next interfered with their ability to learn, nearly 4 in 10 disagreed. About one-third of the students indicated it did interfere and 30% were neutral. The quality of instruction and commitment to the course varied among instructors.

Although each professor who taught in the course was an expert in their field, they varied in their ability to engage students in learning as well as integrate their presentations with previous and future lectures. Professors who had brief teaching stints could not successfully invest the time and effort into course planning and coordination to create a seamless and cogent learning experience.

Final Assessment - Qualitative Data

UCDT provided curriculum developers with the unique opportunity to field test new educational software and teaching techniques. To measure the effectiveness of these techniques, the evaluation team collected information from students about their educational experiences. Students were asked a series of open-ended questions about faculty performance and course improvements. Different themes emerged from the data, but student comments focused attention on both the lab and lecture. These comments should be carefully considered before Global Change I is taught again. Students made several comments about the lecture. Many of the comments were similar to this one: "Give the lecture portion a more concrete format. I enjoyed having different professors teach the different lectures, however, I felt that there was little guidelines for the overall class structure."

Another student said this, "the professors' lectures don't blend into each other and they don't seem to know what the last [person] has said." It is possible to infer from these comments that students perceive a lack of coordination among faculty. This lack of coordination may have attenuated the potential for learning.

Although there were five instructors for Global Change I, only two regularly attended the curriculum development meetings. The lack of effective coordination between instructors resulted in topics unintentionally being repeated several times throughout the semester. The instructors periodically address how the topics discussed during lecture interrelate.

Students also were concerned that instructional materials were being presented without having an opportunity to read them beforehand. One student suggested: "Present the class reading material before the lecture[r] teaches it. . ." Another said:

I feel that we got screwed as far as lecture notes go. If I would have taken this class later, much more effort should [sic] have been put into getting the lecture on the web before that actual lecture. I have been told many times that the future classes don't have to worry about that because the lectures are on the web now, but what do I care about the future classes?

The Global Change web site team had difficulty obtaining lecture notes and instructional materials from some instructors prior to their presentation. Thus, some of the lectures were taught without students being primed for learning. These problems are not surprising when one considers that the curriculum was evolving during the course of the semester. As the curriculum matures, these problems should diminish.

Many students commented about technical problems with STELLA as well as the lack of apparent connection between the labs and lectures. The following comments highlight the difficulties:

Student 1: Work the bugs out of the computer program before actually expecting students to use it! At times it was VERY [student's emphasis] frustrating and seemed as thought [sic] the lab assignment hadn't even been tested before giving it out. It may also be a good idea to explain the object of the lab and why exactly students are doing the model. It seemed like we were just reading the handout and filling the model in exactly the way it was on the sheet--but from myself and those that I spoke with we didn't really know why we were doing it.

Student 2: The Labs need to be overhauled. They just required us to copy information off a piece of paper into the computer and hit enter. No thinking involved or problem solving. They were a frustrating waste of time, and this was complicated by computer problems.

Student 3: . . . Make the lab assignments related more to the lectures. For example, make the assigned questions refer back to materials covered in lectures and or handouts. Also a more thorough explanation of the lab and its contents need to be done before you throw the students in front of the computer and tell them to get started. For example, the labs [assignments] were handed out as you walk into the lab and then the T.A. would be briefly going over the computer procedures before we even know what the lab is all about. I would understand the lab more if I didn't miss her explanation of the computer procedures because I am reading the lab to figure out what the lab is about.

The linkage between the lab assignments and the lectures were never clearly articulated. Ideally, the lab assignments and lectures should work as partners to create reinforcing learning experiences. When they are on separate tracks, opportunities for learning are missed and students experience frustration. The lab assignments should be revamped to engage students in learning tasks which challenge them to grow cognitively and affectively. The current crop of lab assignments lack the creativity and intellectual rigor of the other course components. The frustration experienced by the students are understandable given these problems.

Chronology of Curriculum Reform Events

Three of the major objectives of the UCDT project are to identify and clearly define the major obstacles to interdisciplinary curriculum reform, to design innovative means of overcoming these obstacles, and to create universally applicable templates for implementing interdisciplinary course structures at other universities. The UCDT Faculty Advisory Committee has engaged in a number of activities to make progress toward achieving these goals.

Beginning in mid-October, representatives of the UCDT Faculty Advisory Committee began preliminary discussions with Assistant to the President, Gary Krenz, regarding possible strategies for developing a minor in Global Change. On November 5, UCDT Principal Investigator Tim Killeen met with Gary Krenz and Associate Provost, Susan Lipschutz to get a sense of the level of support for additional interdisciplinary curriculum development from the academic administration. Susan Lipschutz agreed to support the development of the minor, and along with the others at the meeting, Susan endorsed the next step toward developing a Global Change concentration.

Tim Killeen, Gary Krenz, Jim Teeri and Eric Dey were invited by University of Michigan, Dearborn Principal Investigator Charlotte Otto, to attend the UM Dearborn s project advisory committee meeting on November 22. The idea was for the two groups to share progress reports and try to get new ideas which could lead to interdisciplinary curricular reform.

Susan Lipschutz, Gary Krenz and Tim Killeen invited the Deans from each of the schools and colleges involved in the project (Engineering; Literature, Science and the Arts; Natural Resources and Environment; and Education) to a meeting to consider Global Change I & II and to discuss the possibility of further curriculum development. At Susan Lipschutz's suggestion, a brief list of what would be required from each of the respective schools and colleges if a Global Change Minor was to be implemented was created and sent along with an invitation to attend a meeting on this topic on November 25, 1996 in the Fleming Administration Building.

The meeting with the Deans was held on November 25th. Gary Krenz and Susan Lipschutz represented central administration. Sandra Gregerman, program director for undergraduate education, and Lincoln Faller, associate dean, represented Literature Science & the Arts, Harrison Morton, associate dean, represented the School of Natural Resources and Environment; J. Wayne Jones, associate dean, represented the College of Engineering; and Karen Wixson, associate dean, represented the School of Education. Tim Killeen, David Allan, Jim Walker and Eric Dey represented the UCDT Faculty Advisory Committee.

The deans and administrators had a number of questions and comments for the Global Change representatives. They asked whether the UCDT team had done any tracking of previous participants in the courses to see what they are doing now and whether they have modified their behavior as the result of their exposure to Global Change. They asked whether the advocates of a Global Change concentration were wedded to the concept of having a three year program and if so how we proposed to accommodate transfer students at the Junior level. The deans also asked how the courses will fit into the existing distribution requirements. They asked what the Global Change representatives what the minimum number of students that the concentration would need to become self supporting. And, the deans asked whether the Global Change III course would be a science course or whether it would encompass the humanities as well.

All of the associate deans who were present individually expressed their support for the further development of interdisciplinary Global Change courses with the ultimate goal being the establishment a concentration in Global Change. The proviso was that their Deans also check-off on the concept and that it be approved at each step of its development. Overall, the deans who attended the meeting were positive regarding the present program and the plans for expansion.

Following the meeting, Karen Wixson from the School of Education asked Tim Killeen to set-up a meeting with Ron Marx, a program chair in the School of Education who does work in the K-12 area to see whether there might be applications for the Web-based curricula in high school.

Lincoln Faller informed Tim Killeen that if we wanted to offer the Global Change III course in the Fall of 1997, we would have to present it and have it approved by the LS&A curriculum committee in January. We immediately called for the convening of a small group of the UCDT Faculty Advisory committee including Tim Killeen, Jim Teeri, David Allan, Walker, Steve Brechin and Lisa Curran to begin developing the curriculum for the Global Change III course. This group met just before the end of the term with Steve Brechin agreeing to take the lead in the new course development.

Issues Impacting Curricular Reform

The UCDT Faculty Advisory Committee has identified a few areas which must be addressed in order to advance curricular reform at the University of Michigan. Michigan does not utilize a system of majors and minors. The University designates concentrations of study, which are the equivalent of a major at other institutions. Each school or college within the University establishes its own criteria and requirements for achieving the concentration designation so there is not one universally accepted form established for achieving a concentration. Another area of concern is the individual course approval process. There are two obstacles to be overcome in this area. The biggest obstacle consists of Michigan s not having a University-wide process for curriculum approval. The most efficient vehicle to securing the "U-status" for a course is to gain approval from the curriculum committee of the College of Literature, Science and the Arts (LS&A). Once LS&A s approval has been secured, even if all of the curriculum committees for the schools or colleges in which we would hope to offer the course have not approved the curriculum, the course can still be included in the University s bulletins, catalogues, and course schedule books. This means we can offer the course to the University community.

The other substantial obstacle to curriculum approval is time. In order to teach a course through LS&A in the Fall term of an academic year, the course must be approved by LS&A s curriculum committee at the latest in January of the preceding academic year. This means there is a nine month lead time after a course is approved by LS&A s curriculum committee and the actual course offering. Should the curriculum committee decide the proposed course curriculum needs revision this process takes even longer. Perhaps not all schools and colleges in the University of Michigan have such a lengthy lead time for course approval, but since one of the main reasons for this lead time is printing requirements it seems probable that time is an issue for gaining curriculum approval across the whole University.

The final and perhaps greatest hurdle to be overcome by the UCDT Faculty Advisory Committee is the unresolved issues with the implementation of Value Centered Management (VCM) at the University. There are hints that the cost/benefit structure established by the VCM concept may provide a stumbling block to the development of truly interdisciplinary curriculum development. It may cause departments to become more possessive of their faculty and the credit they receive for teaching full time equivalent hours. Additionally, the already established programs in Environmental Studies in the College of Literature, Science and the Arts and in the School of Natural Resources and Environment, may feel that the UCDT s new Global Change curriculum is simply creating an unneeded and competing duplication of the courses they are already offering.

Evaluation Recommendations

Due to the experimental nature of the curriculum, it is reasonable to assume that the project would experience some growing pains. This is a period of building and experimentation for the Global Change program. In this spirit we offer the following recommendations.

Recommendations:

- Sophomores and juniors make about 70% of the class population. The educational needs of this population may differ from those of seniors and freshmen. Where appropriate, students should be grouped and assigned educational assignments which meet their developmental and instructional needs.

- Serious technical problems with STELLA as well as poorly written lab assignments probably reduced the ability of students "to learn how to develop dynamical models of Earth system processes; to understand the importance of physical models of complex systems as well as their limitations (course objective number 6)." STELLA should be re-evaluated as an instructional tool, and the lab assignments should be revamped to improve their linkages with the lecture and to strengthen the educational value of their assignments.

-The lecture should serve as the compass for lab development. Labs should both reinforce the important and complex concepts presented during the lecture; and develop cognitive and technological skills which are needed to become a competent global change investigator. Lab assignments should be intellectually challenging and engaging. They should also focus students' mental facilities over several sessions to take advantage of anchored instructional teaching techniques and principles (The Cognition and Technology Group at Vanderbilt, 1990). If the lab and lecture are properly integrated they can increase the likelihood of a cogent educational experience.

- The lecture and lab should embrace a cogent pedagogical paradigm. The curriculum development group should consider using structured case studies, knowledge building communities (Scardamalia and Bereiter, 1994), or other instructional methods to help organize learning activities. Pedagogy that actively engages students in learning will lead to improvements in academic performance and motivation.

- The data indicated that the current team teaching arrangement has not achieved its potential. Students are experiencing transition problems associated with the rotation of instructors during the semester. Each instructor brings a different pedagogy which students must quickly interpret to get value from the educational experiences. Not only do teaching styles vary substantially, but so does commitment to the Global Change course. Itinerant professors who teach a few sessions do not spend enough time developing the Global Change curricula. Less time is spent on developing visual displays (e.g., overheads), building relationships with students, conducting instructional exercises, and conducting student evaluations. In addition, visiting professors invest little time contextualizing their presentations. That is to say, their lecture topics are not centered with respect to past and future topics. Instructional events which are centered are more likely to promote learning.

Team teaching can be a powerful and cogent pedagogy. Unfortunately, it requires a substantial investment in time and energy to coordinate and integrate the instructional message. As the number of instructors increases, so does the complexity of the coordination and integration effort. We therefore recommend selecting a small number of instructors who are committed to meeting frequently to refine and to sustain this and other Global Change courses.

- The job of evaluating the curriculum was complicated by vague course objectives. The Team experienced problems getting the curriculum development group to create tangible learning outcomes due to the evolving nature of the curriculum. Curriculum priories included scheduling instructors, planning and writing lab assignments, web site development, and demonstration projects to inform the academic community of UCDT activities. Before Global Change I is taught again faculty should articulate student outcomes, methods and timetables, and reporting and deliberation procedures.

APPENDIX

Final Assessment Questionnaire

UCDT- Global Change I -- Fall 1996

1. Gender

2. Class standing

3. Major

4. Would you be interested in taking another course on global change issues in the future?

5. Do you intend to take Global Change II next semester?

-----------------------------------------------------------------Note: For each of the questions below, please choose from the following: Strongly Agree, Agree, Disagree, or Strongly Disagree.

-----------------------------------------------------------------6. Overall, the lab for this course was excellent.

7. Overall, the lab instructor was an excellent teacher.

8. I learned a great deal from the labs.

9. Directions for laboratory assignments are clear and specific. 10. Laboratory assignments seem carefully chosen.

11. Laboratory assignments made an important contribution to my understanding of the topics discussed in lecture.

12. Laboratory notes found at the Global Change web site help me understand the lab assignments.

-----------------------------------------------------------------13. The Global Change web site is organized so I can easily locate the information I need.

14. Using the web has made a significant contribution to my learning. 15. The links from the Global Change web site to other internet web sites has provided me with helpful information.

-----------------------------------------------------------------16. I feel confident in my ability to use the program STELLA to construct models.

17. The program STELLA has helped me to understand how many interacting components create global phenomena.

-----------------------------------------------------------------18. I have learned a good deal of factual material in this course.

19. I have gained a good understanding of concepts/principles in this field.

20. I have deepened my interest in the subject matter of this course.

21. I am enthusiastic about the course material.

22. The knowledge I have learned through this course has improved my ability to participate in debates about Global Change and the environment.

23. The course has encouraged me to think critically about Global Change.

24. Having several instructors give the lecture contributed to my understanding of the concepts/principles related to Global Change.

25. The transition from one instructor to the next interfered with my ability to learn.

26. I actively participated in class discussions.

27. I utilized all the learning opportunities provided in this course. -----------------------------------------------------------------This course used several instructors to teach the lectures in Global Change I. Reflecting now upon your experiences in this course, what suggestions would you offer to the following professors in order to help improve this course in the future? (Your identity will not be revealed when these comments are tabulated).

28. Comments for Prof. Tim Killeen:

29. Comments for Prof. David Allan:

30. Comments for Prof. James Teeri:

31. Comments for Prof. George Kling:

-----------------------------------------------------------------32. How would you improve this course?