Undergraduate Curriculum Development Testbed 

Evaluation Team Report

Team Members: Dr. Eric Dey, Joseph M. Fenty, and Greg Barrett

Winter 1997 Report

June 8, 1997

The Undergraduate Curriculum Development Testbed (UCDT) is multi-year initiative designed to field a new instructional platform which will help faculty and administrators at the University of Michigan to define systemic barriers to interdiscipl inary teaching and to plan for ways to eliminate these barriers. One of the primary activities of UCDT is to develop a constellation of interdisciplinary courses which will introduce students to the science of global change. Currently, only Global Chang e I and II are available to students during alternate semesters. During the Winter 1997, semester the evaluation team created a plan to assess the impact of Global Change II on the students.

The evaluation team examined the extent to which students achieved non-cognitive outcomes outlined by the UCDT curriculum development group. The evaluation team used a pretest, midterm, and final assessment instrument to collect data about student s. In addition, the team also conducted a focus group to collect information about the shared experiences of students. The results from the assessment instruments were used by the team to identify curricular strengths and weaknesses.

The overall objective of the Global Change curriculum is to educate students about global change issues. It is the intention of the curriculum developers to posit a model of interdisciplinary teaching which will contribute to undergraduate cur ricula reform. The extent to which the Global Change curriculum meets its objectives must be determined through a rigorous program of assessment. Toward this end, the evaluation team has created a multifaceted evaluation program to assess the educationa l outcomes of the Global Change II curriculum.

Unlike Global Change I, the curriculum development group has not formally articulated any student objectives for Global Change II. The lack of student objectives for Global Change II has made evaluation of this course difficult. Student objective s are like landmarks that tell the traveler where they have been and where they are going. Without student objectives it difficult to determine if student outcomes are intentional or serendipitous.

The midterm assessment instrument was developed to collect information about students' experience with the laboratory, lecture, and Global Change web page. Students were also asked to share their perceptions of the course instructors. The que stions that appeared on the midterm assessment instrument were approved by the curriculum development group. The midterm instrument questions and the results are summarized by Table A1 in the Appendix.

The laboratory assignments and software for Global Change II were revamped during the winter semester. The curriculum development team introduced new instructional activities designed to give students a synergistic blend of computer assignment s and discussions.

The midterm assessment data indicated that more than 8 in 10 students reported that the lab assignments were carefully chosen and intellectually challenging. More than half the students indicated the lab assignments made an important contribution to their understanding of the topics discussed in the lecture. Furthermore, over 80% of the students credited the lab discussions with making a contribution to their learning. Given these positive self-assessments, it is not surprising that the laborato ry instructor shares some of the credit. When asked if the lab instructor was an excellent instructor, 94% agreed.

The curriculum development group created laboratory assignments using the ArcView 3.0 GIS. ArcView is a powerful computer mapping system designed by Environment Systems Research Institute, Inc. This geographic information system was created to he lp the user to analyze data in a spatial context. GIS technology integrates common database operations such as query and statistical analysis with unique visualization and geographic analysis. ArcView is most often used as a tool by GIS specialist to an alyze street networks (traffic planning and maintenance), natural resources (natural resource management, habitat assessment), land parceling (zoning), facilities management (utility planning and maintenance). ArcView’s powerful visual and analytical cap abilities were used by the curriculum development group as a pedagogical tool.

The midterm assessment instrument included several questions about students’ experiences with ArcView. When students were asked if ArcView helped them understand global change concepts and principles, nearly 7 in 10 agreed. Approximately, 9 in 10 reported that ArcView helped them understand trends and relationships among variables. The results revealed that students enjoyed working with the data sets that contained information about various countries throughout the world. These data were used t o illustrate relationships between certain key variables which affected the natural environment and human behavior.

Unfortunately, less than half of the students indicated that they felt confident in their ability to use ArcView. This may be explained by the few number of ArcView Assessments they experienced at the time of the midterm.

Students were asked to recount their experiences in the lecture. The midterm assessment instrument captured information about the transition issues related to the team teaching format, the ability to critically think about and debate global ch ange issues, and the connectedness of the lecture topics. More than 9 in 10 students reported that they learned a lot of factual material, and improved their ability to debate as well as think critically about global change issues.

Nearly one quarter of the students agreed that having several instructors interfered with their ability to learn and made it difficult to identify what information was important. One-half of the students reported that when a visiting professor gav e a lecture they had difficulty identifying what information was important. On the other hand, 52% of the students claimed that having several instructors give the lecture contributed to their understanding of global change concepts and principles. Oste nsibly, several students are having problems with the team teaching format. It may be worthwhile to closely examine this approach to ease the transition between instructors.

Approximately 9 in 10 students reported that the Global Change web site was organized so they can easily locate the information they need. Nearly three-quarters of the students indicated that the web had made a significant contribution to thei r learning. The students were so impressed with the Global Change web site that 86% suggested that other classes should have a similar web site. Most students use the Global Change web site to read the lecture notes once or twice a week.

The data from these series of questions indicate that students have been personally affected by the Global Change II course. More than 9 in 10 students indicated that they have deepened their interest in the course, and 8 in 10 reported they a re enthusiastic about the course material. Three-fourths of the students said they felt empowered to act.

The final assessment data was similar in many ways to the midterm assignment. The final assessment data revealed that more than 85% of the students reported that the lab assignments were carefully chosen and intellectually challenging. More t han 70% of the students indicated that ArcView helped them understand global change concepts and principles. Students' experience with the lecture was decidedly mixed.

Over 90% of the students reported that they learned a good deal of factual material, improved their ability to participate in debates about global change, and thought critically about global change. Students were somewhat ambiguous about the team- teaching format. Slightly less than half of the students reported that having several instructors contributed to their understanding of Global Change concepts and principles. When students were asked if visiting instructors integrated concepts and theme s presented earlier in the course into their presentation, 60% agreed. Although 44.5% indicated that the transition from one instructor to the next did not interfered with their ability to learn, 28.9 % of the students reported this teaching format inte rfered with their learning.

The most surprising results of the study can be found in the final assessment instrument. When students were asked if it was difficult to understand how the topics covered in the lecture fit together, 86.7% agreed. These results are suggestive a bout the structure of the Global Change II curriculum. While this course may make sense for the professors and graduate student instructors, students are having difficulty making sense of their educational experience.

Comparison of the Midterm and Final Assessment Results

The final and midterm assessment instruments were designed to mirror each other. Many of the questions that appeared on the midterm were also used on the final. The evaluation team used these instruments to measure the same domains during the two time periods. Changes in attitude detected by comparing the final and midterm may be attributable to the instructional events in the Global Change II course. Although we can not be 100% certain that the changes are wholly attributable to the course, we believe it is reasonable to assume this is true.

In many ways the results of the midterm and final assessment are similar. In general, the response patterns for most of the individual question items differ by only a few percentage points. Nevertheless, there are some exceptions:

- By the final, the percentage of students reporting that the laboratory assignments helped them understand the lecture increased by 10.7%. The lab assignments and lecture presentations appear to have a positive synergistic relationship with the lectu re.

- Over the course of the semester students became more confident in their ability to use ArcView. By the final, 62.2% of the students expressed this position, this percentage represents a 14.2% increase. The lab developers are apparently achieving g ood results with this important computer program. ArcView is a powerful tool for understanding patterns of data anchored to geographical objects. Confidence and acumen with ArcView indicates that students are in a good position to use this tool to explo it learning opportunities.

- Over time fewer students indicated that they had difficulty identifying what information was important

when a visiting instructor taught the course. Although nearly half the students had this problem, the percentage of students who reported this experience declined. These results indicate that the curriculum development group needs to structure instru ctional events to help students quickly acclimate to the different pedagogical environments created by the instructors.

- Students’ attitude toward the web improved over time. The percentage of students reporting that the web made a significant contribution to their learning increased 10.8% to a total of 84.8% by the final. This level of approval indicates that the we b is being deployed as an effective learning tool by the curriculum development group.

- Even though the number of students indicating that the Global Change links provided helpful information increased by 8.6%, overall support for this position did not exceed one-third of the student respondents. These figures seem to indicate that t he curriculum development group needs to demonstrate to students how these links can be used to learn Global Change concepts and principles. The use of web-based links can be a powerful pedagogical tool to encourage learning. Thus externally maintained links are probably the best source of current data about the environment and human activity. Links should be incorporated into the Global Change web page which have instructional value. The percentage of students that indicated that they never checked o ut the links at other sites rose 16% to a total of 50%. This suggest that as time progressed fewer students explored the Global Change web links. The criteria by which the web links are selected should be examined to ensure they are of high quality and an integral part of instructional events.

-Although the percentage of students who indicated that they have deepened their interest in the field of Global Change declined slightly (-4.7%), overall interest remained very high. By the final, nearly 9 in 10 students indicated that they were more interested in global change subjects. Concomitantly, enthusiasm for the course material also became stronger during the course of the study.

-Research about student learning tells us that cooperative learning can be a powerful pedagogic tool to achieve both cognitive and affective outcomes. Student who become responsible for their own learning and teaching others tend to achieve well on pe rformance indicators. Thus it is encouraging to note that the percentage of students reporting that they made an important contribution to the learning of others in the course increased by 15.3%. Nevertheless, only 4 of 10 students agreed with this posi tion by the time of the final assessment. These results seem to indicate that few students either avail themselves of the opportunity to help other students or the course format does not make this kind of cooperation possible. It would behoove the curri culum development group to increase the opportunities for cooperative learning. Investments made in this pedagogy should yield student performance dividends in the future.

Recommendations

1. Develop student goals and objectives for Global Change II that articulate the intellectual and cognitive outcomes students should acquire by the completion of the semester. These goals and objectives should drive curriculum development and assessme nt.

2. The curriculum development group should begin to articulate how the Global Change curricula may produce "generalizable tools and templates for interdisciplinary courses." According to the NSF grant proposal these tools and templates are s uppose to promote development of interdisciplinary courses on- and off-campus. At this point it is unclear how these goals are going to be achieved. It is important at this juncture for the curriculum development group to have a sustained discussion abo ut the comprehensive goals of UCDT. These discussion should culminate in plans and timetables to achieve these goals.

3. Efforts to improve the web should continue. The web designers should seek to make the Global Change web page more interactive. Research about student learning suggest that the web can be used to achieve cognitive gains when students use the web to engage in sustained discourse about academic issues. To achieve these ends, the web designers may consider the following suggestions: