UCDT

Evaluation Methods

Surveys

Surveys were delivered via the web to collect data about students' experiences with the lab, lecture, and Global Change web site. The web-based method of delivering the baseline, midterm, and final assessment instruments functioned as an efficient means of collect information about students. Because the data were already in electronic form, it was relatively easy to create ASCII files which could be brought into a word processor, spreadsheet or statistical program. The quantitative data was analyzed using SPSS and posted to the web.

Having the results on the web, allowed the evaluation team to present assessment findings to the curriculum development group during the weekly staff meeting. The curriculum development group used the assessment data and analysis on the web to present the findings to various groups on and off campus. During these presentations, members of the curriculum development group discussed the Global Change Project, UCDT, and the evaluation results using the web.

Baseline Instrument. The Baseline Instrument was designed to collect information about student demographics, the source of information students used to find out about the course, scientific preparation (i.e., science related course that the student have completed to date), expected benefits to be derived from taking the course, and career aspirations. The questions items were open-ended to allow students to respond in their own words.

Midterm Assessment. The midterm assessment instrument was developed to collect information about students' experience with the laboratory, lecture, and Global Change web page. Students were also asked to share their perception of the course instructors. This instrument employed closed-ended and open-ended questions. The questions that appeared on the midterm assessment instruction were approved by the curriculum development group. (A sample of the midterm assessment items and actual results for Global Change II can be found in the Appendix C.)

Final Assessment. The final assessment instrument was designed to parallel the midterm assessment instrument to measure the impact of the instructional experience. Additional questions were added to the final assessment instrument to gauge the effects of team teaching and to determine the level of interest in taking another Global Change course. Students were also asked to share their perceptions of the instructor's effectiveness, lab assignments, and educational technology. (A sample of the final assessment items and actual results for Global Change II can be found in the Appendix C.)

Interviews

The evaluation team conducted interviews with the faculty and staff to understand their experiences with the UCDT program. The interviews with faculty focused on their prior experiences with interdisciplinary teaching, perceptions of the climate for interdisciplinarity at the University, and improvements which would encourage interdisciplinary teaching. The graduate student instructors were interviewed to help the evaluation team to understand the structure and substance of the laboratory assignments. Interviews were also conducted with administrators to understand the interdisciplinary climate at the U of M and the level of institutional commitment to sustain interdisciplinary programs.

Faculty Interviews. An important goal of the UCDT is to help the university community better understand institutional and systemic barriers to a major expansion of interdisciplinary instruction and interdisciplinary curricula. Understanding the characteristics of the academic niche occupied by the Global Change curricula will help us understand the degree to which interdisciplinary programs can be integrated into the current university's academic structure. The quality and scope any academic niche is determined by a variety of factors such as available resources, administrative support, course offerings, and faculty and student participation. To define the academic niche occupied by the Global Change curricula, the evaluation team interviewed the faculty about their interdisciplinary experiences and perceptions.

During the fall 1997 semester three professors who were involved in teaching Global Change I were interviewed using a semi-structured interview format. These professors, all tenured and from different disciplines, were asked about their perceptions of previous and current interdisciplinary experiences; the influence of the institutional, disciplinary, and departmental context on interdisciplinary activity; the transferability of the UCDT experience to other programs at the university; and the resources and/or structural changes needed to support interdisciplinarity, in general, at the University of Michigan.

Observations

A member of the evaluation team attended most of the lectures and some of the labs during a substantial portion of the semester. During these observations the team members recorded notes about the instructional activities and student behavior. From these observations the team constructed instruments to collect data about student experiences. In addition, the team used the observations to present the curriculum development group with ideas for enhancing the opportunities for learning and/or mitigating pedagogical problems.

Focus Group

The evaluation team conducted a student focus group during the 1997 winter semester to gather the collective experiences of students. Student who were taking the Global Change II course and had completed Global Change I were asked to participate. The evaluation team encouraged students to participate by offering pizza and soft drinks. This format allowed students to partake in discussions about the lab, lecture, and the web experience.

Early Student Feedback. During the 1996 fall semester the evaluation team asked the Center for Research on Learning and Teaching to conduct the Early Student Feedback. The Early Student Feedback assessment instrument was designed to help faculty get student feedback about their teaching early in the term. The technique allowed the use of small group discussions to gather information directly from students in a format that permits interaction and clarification. Students were asked to list the major strengths of the course and the changes which would improve the quality of the course.

The Early Student Feedback instrument was administered on Wednesday, October 16, 1996 (approximately six weeks into the term). In small groups, students were asked to reach consensus on the following questions and directions:

1. List the major strengths of the course. What is helping you learn?

2. List changes that could be made in the course to assist you in learning. Please explain how suggested changes could be made.

The data collected from this instrument were compiled, summarized, and presented to the curriculum development team.

Classroom Assessment Techniques

The evaluation team introduced classroom assessment to the faculty and Graduate Student Instructors. These techniques, pioneered by Angelo (1993), are designed to (a) assess course-related knowledge and skills, (b) assess learner attitudes, values, and self-awareness, and (c) assess learner reactions to instruction. Three techniques were considered due to their low time and energy requirements: the One-Minute Paper, Muddiest Point, and One-Sentence Summary . Only the One-Minute Paper was used consistently throughout this course to assess the results of the lab.

To use the One-Minute Paper, an instructor stops class two or three minutes early and asks students to respond briefly to some variation on the following two questions: "What was the most important thing you learned during this class? and "What important question remains unanswered?"

The data collected from this instrument were presented during the weekly curriculum development meetings and posted to the web were students could read the material. On several occasions data from this instrument was used to adjust the pedagogy.