Paper presented at AERA 2006, San Francisco
(submitted to Educational Philosophy and Theory)
Complex Systems and Educational Change:
Towards a New Research Agenda
Jay L. Lemke, University of Michigan
Nora Sabelli, SRI International
How might we usefully apply concepts and procedures derived from the study of other complex dynamical systems to analyzing systemic change in education?
This seemed a natural question to the diverse group of about 40 leading natural science and education researchers who met first in 1999 at MIT’s Endicott House under the sponsorship of the National Science Foundation and the New England Complex Systems Institute (NECSI, 1999) to consider the future role of complex systems science in the K-16 curriculum. It was clear that whatever recommendations for curriculum development might eventually emerge, the curriculum change process itself would pose challenges to making this important new area of the sciences accessible to large numbers of students. The Endicott conference participants were hopeful that complex systems theory could offer insights into the processes of curriculum change and, more generally, of systemic reform in education. A working group was formed to examine this question, and it produced an initial report (Lemke et al., 1999).
A key recommendation from the Endicott House conference was to bring these issues to wider attention in the education community, and at a smaller second meeting sponsored by NECSI, plans were developed for a symposium at the American Educational Research Association annual meeting in 2002 (Jacobson et al., 2001). At the same time a number of the participants also convened in 2001at the Balcones Springs conference sponsored by NSF and the University of Texas to discuss issues of urban systemic reform with leaders of four major NSF-sponsored projects and about 16 other experts in the science and mathematics education community (Confrey et al., 2001). This article describes a number of the important concepts and research issues regarding the application of complex systems approaches to education that have developed out of this continuing discussion.
New conceptual approaches to the study of complex systems have been developed in the last two decades by mathematicians, physicists, chemists, biologists, and computer scientists (cf. Bar-Yam, 1997) . They are being applied and extended by economists, psychologists, organizational scientists, and researchers in many disciplines whose insights are being scaffolded not only by new quantitative techniques, but by new qualitative conceptions of phenomena common to many different complex systems. Concepts such as multi-scale hierarchical organization, emergent patterning, agent-based modeling, dynamical attractors and repellors, information flows and constraints, system-environment interaction, developmental trajectories, selectional ratchets, fitness landscapes, interaction across timescales, and varieties of self-organization are becoming key tools for qualitative reasoning about complex socio-natural systems as well as for quantitative modeling and simulation.
Can the new tools of complex system analysis help us understand the potential impact on the educational system of new technologies and help us predict the paths that different efforts at systemic reform follow? Can they help us design new educational systems to meet the needs of all citizens in the new century? Can they help us identify critical relationships within the educational system that resist systemic change or afford opportunities for new alternatives? Can we realistically hope for an educational system that will teach large numbers of students to use the new tools for thinking that complexity theory has developed? Can we find ways to make the value of these tools sufficiently evident and attractive to large numbers of students and teachers so that they will seek them out?
If the answers to any of these questions are to be 'yes', we will require collaboration within a diverse new community of researchers seeking a common framework for sharing ideas from different disciplines and approaches to both complex system analysis and to education. There is an urgency to the formation of such a community. If the response of the educational system to the new demands of the public for reform and to the new opportunities technology affords is not guided by the best ideas of the research community, and by research- and data-driven decision-making, it will be guided by other forces, in which we may have less confidence.
The concepts and tools we consider in this article have been put to use in practice and tested in managing complex and ill-defined ecological systems, e.g. in what has been called the “active adaptive management” technique, which could serve as a case study of handling the complexity inherent in a multifaceted system merging scientific knowledge and public-interest goals (see Farr 2000 for definitions and references and Gunderson & Holling 2002 for a related approach). Active adaptive management is a "process of testing alternative hypotheses through management action, learning from experience, and making appropriate change to policy and management practice".
It is not our aim here to present a complete conceptual framework for the analysis of education as a complex system. Our purpose is to begin to define possible agendas for further research toward such a framework. Towards this end, we will try to illustrate the plausibility of defining such a framework, and raise the question of the relation between such frameworks and the crucial aggregation of data across “systemic experiments” (Sabelli, in preparation; Confrey et al,. 2002). Even within what we might eventually agree on as a common framework and terminology for describing such a complex system, there will continue to be room for many existing alternative models and, over an extended period of research, for the emergence of new data-driven models and syntheses appropriate to various specific tasks. We will describe the core issues for a framework under the following headings: Defining the System, Structural Analysis, Relationships Among Subsystems and Levels, Drivers for Change,and Modeling Methods. Much of this sketch towards a possible framework will be presented in the form of questions because we believe that the primary contribution that complex systems theory can make to such an enterprise right now is to guide what questions we ask and how we frame them in relation to one another and to prospective data sources. Questions matter; they are the seeds from which new theories grow, and like all seeds they carry forward the prior theories from which they come.
The U.S. K-16 educational system is conventionally defined as the system of public and private schools and colleges that offer students formal education from kindergarten to college graduation. For research purposes, however, the system must ultimately be defined by our analysis of its constituent elements and environmental dynamics, such as which institutions and social practices and which sources and users of information and material and human resources are tightly enough coupled and interdependent in their behavior that they must be included within the system? Likewise, what are the range of timescales characteristic of the critical processes that enable the system to maintain itself? What are its significant levels of organization, not simply or primarily in terms of lines of authority (control hierarchies), but in terms of characteristic structures and characteristic emergent processes and patterns at each level? What kinds of material resource and information flows connect adjacent and non-adjacent levels? How is information transformed, filtered, re-organized, and added to from level to level? How is information-overload avoided by emergent systems through pattern-recognition that extracts from large data-flows only what matters for the dynamics of the next higher level?
If we consider the longest timescales experienced by students within the system, we will need to extend its definition to consider pre-school education, post-graduate study, and continuing adult education. If we examine all the source institutions that contribute to students' understanding of particular topics within the formal curriculum, we must include informal educational institutions such as science museums and information sources and learning sites afforded by mass media, print publishing, and interactive communication technologies. If we look at resource constraints and decision-making bodies, we will add school boards and trustees and state education authorities. If we include ourselves within the system, we will consider our roles as teachers and researchers, and the relationship between research institutions and sponsors and the communities which make use of research results.
Formal organizational hierarchies propose one starting point for identifying levels within the core educational system: individual learners and teachers, small groups, classrooms, departments, schools, districts (LEAs), states (SEAs), federal agencies, the total system. What would a dynamical analysis propose—one that takes into account the differing timescales at which different levels of the system function? If we analyze the system in terms of dynamical processes and emergent phenomena on different timescales, what would the units of analysis be? How do brief actions by teachers and students add up to coherent activities over periods of minutes and hours, days and months? How do curriculum change processes that occur over periods of years exchange information with classroom activities that occur over periods of minutes? How do learning events in a laboratory or at a computer workstation and those in classrooms and hallways and cafeterias add up to a coherent longer-term process of educational development, or perhaps the development of facility with a particular concept? How do networks of social interaction with peers in the classroom, in the wider neighborhood community, and in virtual online communities contribute to long-term processes of identity development and formation of lasting attitudes and values, which affect decisions and actions on very short timescales? How do the changing priorities, populations and problems of a local community influence the larger educational system's agendas and programs?
Having focused on some of the characteristic educational processes that involve the student, we could raise similar questions about those in which teachers participate but which may not always involve students, and similar questions about supervisors and administrators, teacher educators, curriculum developers, educational materials publishers, and ourselves as researchers.
Whatever level of organization or subsystem is the focus of our concerns at a particular point, we can always ask a series of key questions motivated by the perspectives of complex system theory:
Ø How do all subsystems subject to those constraints interact to constitute the dynamics of the higher level?
Ø What degrees of freedom remain at the focal level after the constraints are allowed for?
Ø What characteristics of those lower level units determine the range of dynamical possibilities at the focal level?
Ø Under what conditions is each attractor dominant for the (sub-) system?
Ø How do new attractors emerge over the history of the system's development and the evolution of this kind of system?
Ø Which regions of the space of possibilities are accessible and which are not?
Ø What manifolds describe the conditions on the range of values of all other parameters that must be met to achieve some value of the parameter of interest?
Ø What kinds of matter, information, and energy do they exchange?
Ø How tightly coupled are they and what is the topology of the coupling network?
Ø What are the significant branchings, closed loops, and connectivity decompositions?
Ø How do system and environment form a supersystem from the viewpoint of some still larger-scale unit or process?
As an example, we will see that any focal pedagogical “innovation” introduced into a tightly coupled [constrained?] school system is in fact a series of embedded innovations at levels above and below the focal intervention, and strategies for all levels have to be considered coherently.
How is the educational system as a whole driven by external events and pressures such as advances in scientific understanding, the increasing complexity of problems addressed by communities and societies, changing technologies, and public demands for reform? How is educational change constrained by resource limitations, standardized curricula and testing, or deeply held cultural beliefs? How is educational change enabled or made possible by bringing new kinds of people into contact with one another or utilizing new technologies (e.g. cross-age tutoring, or tele-mentoring)? How would educational processes be affected by creating new feedback loops, such as research data that systematically describes outcomes back to teachers, students, and parents? How might new educational institutions (e.g. charter schools, online courses) create niches for themselves in the educational ecology? How might new spontaneous networks, such as online communication groups of teachers within a school or across the country, affect the rate of educational change at each scale of organization?
How would we model and analyze issues like these using the concepts and techniques of ecosystem theory and adaptive management, developmental biology, reaction-diffusion chemistry, non-equilibrium statistical mechanics, nonlinear dynamical systems analysis, cellular automata models, artificial life systems, neural networks, parallel distributed computation, agent-based modeling theory, informatics and infodynamics? Given access to data and expertise about the educational system, how would you yourself approach one of these issues? Given collaboration with others who could offer different insights about complex system behavior, how would you and other educators and researchers begin to formulate any one of these problems for actual study?
How well, for example, could we design today a 'SimSchool' or 'SimDistrict' school- or school district- simulation program? Not just to model an existing system, but to enable us to create alternative systems and study their evolution over time, their needs and problems, their probable outcomes? We could then ask, for instance, what kinds of schools would students design if given access to an appropriate version of this software? How would they evaluate various designs proposed by others? Who would we enlist in the team to create such a software package? What research literatures would we want to consult? What is not yet known that would be needed to complete the project?
Do we even know what kinds of data would be needed to realistically attempt such a project? Insofar as we are only interested in easily quantifiable parameters of the system, such as school budgets, teacher qualifications, and student test-scores, we still need to know how much value added there might be from a complex dynamic system model compared to more static statistical analyses. Good use has been made of static and isolated case analyses, particularly when the dynamics of the relation with the environment or with environmental variables are taken into account, usually by statistical methods, and more recently, by the dynamical inclusion of environmental variables (e.g. Jost, 2003.) Agent-based dynamical and simulation models hold the promise of enabling us to explore potential effects of changes in quantitative parameters and assumptions about how variables interact to produce observable statistical relationships. Complex systems models are designed to model change and dynamics, especially qualitative change: the emergence of new social networks, changes in daily routines or actor preferences. In a human social system, these kinds of changes are mediated by the meanings and values assigned by actors, individually and collectively, to the more objectively definable affordances of their environments. To build effective dynamical models of educational institutions we will need to know not just what people do, but why they do it, how they might imagine things being different, and what they would really want to do.
Even if such system models are not predictive in any detailed way, they can still be useful in identifying possible alternatives, potential problems, and overall qualitative features of the change process which may not be intuitively evident to a linear logic of cause and effect. In complex systems every causal chain is mediated, and many chains branch and loop back on themselves in complicated webs of mutual interdependence, self-regulation, and amplification of effects. This conceptualization is consistent with Michael Fullan’s “systems at the edge of chaos” view of education (Fullan, 1999).
No mention of the data needed for analysis and the development of theoretical models should leave out considerations of the importance of sharing information across projects (i.e. across localized case studies). This sharing and aggregation is a major problem for a topic so dependent on localized conditions as education reform is.
We report next on one such effort to begin this process across a set of long-term projects on systemic reform in science and mathematics education in the U.S.
The important report from the Balcones Springs conference (Confrey et al., 2001) summarizes the lessons learned from four major educational reform projects (see Table 1). Such lessons represent hard-won long-term information about the specificity and diversity of implementations of various closely-related curriculum and system-wide reform models. The projects were related in being implemented in urban schools and districts with highly diverse students, being guided by the same meta-model of systemic change “drivers” (NSF, 2001), but using different models of how to achieve sustainable improvement of teaching and learning. Any research effort to develop a complex system model of educational change would do well to take into account this data about what matters to the success of existing systemic reform initiatives.
<Insert Table 1 about here.>
Perhaps the most important of these lessons is that adaptation of models for system reform to local conditions matters more than efforts to replicate successes elsewhere, without extensive knowledge of how the systemic variables differ between environments. This “localization effect” points to the importance of determining whether any single complex system model can be both general and specific enough that it can include design templates to identify key local parameters that need to be set. Alternatively, more heuristic guidance needs to be developed to aid in the design of quite different models for each educational system (i.e. differing structurally and not just parametrically from one another). Here are some of the more detailed lessons learned from these research projects that seem especially relevant to the design of realistic system models.
It was found that in most cases it takes a long time, of the order of 5 – 10 years, to establish effective collaborations between researchers and school systems, and that during this period there may be a need to re-negotiate and re-commit to goals and strategies developed together whenever there are major changes in leadership or personnel on either side of the partnership. The development of effective partnerships takes 5-10 years, and the fruits of reform efforts tend to become visible only after at least 3-5 years. Any evaluation and tests of scalability require at least a second or third cycle of enlargement or replication, implying a minimum of 10 years’ scope for models of effective change.
For reform efforts to be maximally adaptive to changing environmental conditions, an iterative process is needed in which plans are continuously modified in response to issues that only come to light once implementation has begun, or to the mere change of individuals in either the research or implementation personnel. Successful multi-year reform processes include periods of consolidation of gains; these periods provide a respite to plan for needed changes and for people to become comfortable with one set of changes before contemplating others. In this sense, reform should be viewed as a ‘stepwise’ process, in which advances alternate with such periods of reflection and consolidation. This stepwise strategy promotes buy-in from skeptics, allows for non-disruptive change and establishes a culture of continuous improvement. Under these conditions, modeling the likely effects of different “schedules for innovation” may lead to a more integrated and sustainable organization that is resilient with respect to changing future conditions.
Reforms often begin locally and then face the problem of “scaling out”, i.e. including more units at the same level of organization (e.g. from a few teachers, or one grade level, to all teachers in a school or all grades), and also of “scaling up”, i.e. from small-scale systems (e.g. a small suburban district) to much larger scale systems (e.g. a large urban system or an entire state). As reform scales, there is no guarantee that it will maintain validity with respect to its fundamental principles or goals. For this to happen, some self-regulatory feedback must exist within the system to assess whether such validity has been maintained and to provide an incentive for maintaining it.
There were found to be a number of agents of scaling. For example, student cohorts can motivate scaling up as they move through a system, carrying the reform “upward” with them. This type of spread appears to require a critical mass of students and an initial phase that includes a plan for such “vertical” growth. Another model of spread is to systematically plan for horizontal growth, or scaling out. In doing so, pressures on the reform implementation can create situations that indicate problems with the model, or its limits of applicability (e.g., whole school models in contexts where there are not sufficient resources to support that model). Scaling is a useful strategy for testing the robustness of the process, making it more sustainable, and finding its weakest spots. This points to the interdependence of scaling and sustainability as a key issue for any model.
Sustainability, it was found, has two key aspects. The first is the need for a match between stakeholders’ expectations regarding the nature and pace of results and the ability to provide persuasive demonstrations of timely effects. Early successes, as judged by stakeholders, appear to be crucial for sustaining the reform process. The second aspect depends on relationships among the timescales of change processes in different elements of the system and between the system and larger social-political-economic systems in which it is embedded and on which its functioning depends.
Sustainability is threatened by normal processes of change in larger-scale systems within which the educational system operates (e.g. changes in political administrations, new superintendents with new policies, changes in state regulations or funding formulas, etc.) Widespread commitment by many stakeholders and a critical mass of committed practitioners can ensure that maintaining gains in achievement will move the community to keep updating policies and practices needed to sustain reforms while responding to other inevitable social changes.
Many of these lessons point to the importance of multi-scale modeling techniques for educational change, and particularly to multiple timescale models (Lemke 2000a, 2000b). When we consider that many key structural features of educational practice (e.g. student-teacher ratios, use of textbooks, age-grading, local-taxation funding, curriculum areas, teacher training institutions) have been stable on timescales of a century or longer, we can infer that there are powerful system-regulatory relationships maintaining this stability. Reform mandates and implementations, on the other hand, are formulated and expect results on timescales of the order of a decade or less. Complex system models need to help us understand why so many features of the educational system do not change, as well as under what conditions they will change. Many current reform policies assume that no major structural changes are necessary to achieve reform goals. Realistic models, based on detailed case studies of reform efforts, as well as on general system modeling principles, may help us understand if such assumptions are realistic or not. We need to know whether or not current modest reforms have any realistic chance of producing major gains at the large-scale in realistic timeframes. If it should appear that more radical re-engineering of the educational system is needed, we will need to understand the functional roles and interdependencies of current structural features all the more.
The perspective of this paper has been largely instrumental and externalist, continuing the research and philosophical tradition of trying to view systems we are actually a part of as if we were outside them, in order to gain some ability to intervene in them. The logic of complex systems, however, pulls us away from this expected perspective. In the case studies at the Balcones Springs conference it was clear that the researchers were very much a part of the systems they were reporting on, and that how they participated, what their roles were, and how they were perceived by others all mattered very much both to what happened and to how they reported on it. In many cases their most interesting reports included very personal observations, more experiential and phenomenological than objectified and structural. The development of their models, both for change processes and for good practice, often depended critically on the experiential insights that came from direct participation, over extended periods of time, gaining, in a now classic phrase, “a feeling for the organism” (Keller, 1983). The further lesson, across the cases studied, that change is local, and each case as much a unique individual system as an instance of more general processes, was taken very seriously. Complex systems are, in some special sense, “individuals”, whether or not they are also members of some species.
Our philosophical traditions in natural and social science have not prepared us very well to study the internalist, phenomenological, and individual aspects of complex systems. Our traditions doubt the very possibility of a science of the individual, a science of the particular, and yet other traditions, e.g. in the humanities, have long dealt with their objects of study as both unique and typical. Even reflexive sociology balks short of internalism, of putting ourselves squarely inside our models, of stipulating just how systems look different depending on where we sit inside them, or alongside them (Haraway, 1997).
If complex socio-natural systems are individual, surprising, recalcitrant. If we have to get to know them over time, and in person. If we know them only insofar as we participate in them, and then in specifically partial or participation-dependent ways, then our traditional epistemologies may fail us and good science for these systems look somewhat different from what science has imagined itself doing (though perhaps not, as Latour 1987 argues, from what it has actually been doing). Relative to our human viewpoint and ways of participating in them, socio-natural systems of interest may indeed be a little perverse. May actually display an independent sense of humor. The last word about the real has not been said, and the study of complex systems may still be able to teach us new lessons in intellectual humility. Not every frame of mind on the part of an investigator may prove equally fruitful in learning those lessons.
The conceptual basis of complex systems ideas reflects a change in perspective about our world. This perspective emphasizes both the limits of predictability as well as the possibility of understanding indirect consequences of actions taken, both positive and negative, through the modeling of interdependence. The study of complex systems involves experimental, computational, and theoretical approaches for observation, analysis, modeling, and dynamical simulation.
Complex systems concepts are used in science to provide organization for the otherwise bewildering properties of diverse and often unpredictable systems in a common framework and language. These systems are often unpredictable because small changes in one variable may result in major changes in overall outcomes (critical thresholds), while other, larger, local changes might not disrupt the system significantly (robustness).
The application of these concepts in conjunction with computational models and visually compelling data-driven simulations yields unprecedented means for understanding complex phenomena and revealing new, sometimes counter-intuitive patterns and relationships. Such understandings lead to new and essential questions and to viewing educational systems with new eyes. Understanding complex systems also seems to be critical to our ability to apply knowledge and techniques across very different individual contexts.
The education system is one of the most complex and challenging systems for research. Much as we know about cognitive aspects of learning, pedagogical strategies, and reform implementation, we currently lack the modeling capability needed to help practitioners and policymakers explore the potential impact of proposed interventions, although efforts in this area are currently at a very preliminary stage of development. Indeed, in this perspective there are no independent interventions: proposed changes at the classroom level have implications at school and district levels (e.g. for teacher development, parental expectations, school resources, accountability, and so on) and need to be supported by related interventions across multiple levels. A technical ability to explore such dynamic linkages could be a significant tool for educators and policy makers.
Most important perhaps is a change in the paradigms of our thinking about research on education. Away from input-output “blackbox” causal models to modeling the specific, local linkages that actually interconnect actors, practices, and events across multiple levels of organization. Away from single interventions and simplistic solutions to recognition of the need for coordinated changes throughout the system and to its “external” relations to its constraining and enabling contexts and resources. And even perhaps away from the Enlightenment dream of universal laws, perfect predictability and rational control to a new recognition that all genuinely complex systems are individual, surprising, and not a little perverse. Just like us.
Bar-Yam, Y. (1997). Dynamics of Complex Systems. Reading, MA: Perseus.
Confrey, J., Lemke, J. L., Marshall, J., & Sabelli, N. (2001). Conference on Models of Implementation Research in Science and Mathematics Instruction in Urban Schools. Austin, TX: University of Texas.
Confrey, J., Sabelli N., and Sheingold, K. (2002). A Framework for Quality in Educational Technology Programs, Educational Technology, Vol.42, 7-20.
Farr, D. (2000). Defining Active Adaptive Management. [Online] http://www.ameteam.ca/About%20Flame/AAMdefinition.PDF
Fullan, M. (1999). Change Forces: The Sequel. New York: Routledge/Falmer .
Gunderson, L. & Holling, C.S. (2002). Panarchy: Understanding Transformations in Human and Natural Systems. Washington: Island Press.
Haraway, D. (1997). Modest_Witness@Second_Millennium. FemaleMan(c) _Meets_OncoMouse(TM). New York / London: Routledge.
Jacobson, M., Kaput, J., Wilensky, U., & Lemke, J. L. (2001). Complex systems in education: integrative conceptual tools and techniques for understanding the education system itself. [Online] http://edtech.connect.msu.edu/Searchaera2002/viewproposaltext.asp?propID=6203
Jost, J. (2003) External and
Internal Complexity of Complex Adaptive Systems, SFI Working Paper Abstract
2003. Available at
www.santafe.edu/sfi/publications/wpabstract/200312070
Keller, E. Fox (1983). A Feeling for the Organism: The Life and Work of Barbara McClintock. San Francisco: Freeman.
Latour, B. (1987). Science in Action. Cambridge, MA: Harvard University Press.
Lemke, J.L., et al. (1999). Toward systemic educational change: Questions from a complex systems perspective. Working Group 3, Systemic Educational Change. Report of an NSF-funded Workshop, Endicott House, MA. [Online] http://necsi.org/events/cxedk16/cxedk16_3.html
Lemke, J. L. (2000a). Across the Scales of Time: Artifacts, Activities, and Meanings in Ecosocial Systems. Mind, Culture, and Activity, 7(4), 273-290.
Lemke, J. L. (2000b). Opening Up Closure: Semiotics Across Scales. In J. Chandler & G. van de Vijver (Eds.), Closure: Emergent Organizations and their Dynamics (pp. 100-111). New York: New York Academy of Sciences.
National Science Foundation [NSF], Educational System Reform. (2001). Six Critical Drivers. [Online]. http://www.ehr.nsf.gov/esr/drivers/
New England Complex Systems Institute [NECSI]. (1999). Planning documents for a national initiative on complex systems in K-16 education. [Online] http://necsi.org/events/cxedk16/cxedk16.html
Sabelli, N. (in preparation). Crafting a shared framework for systemic change research in education.
Four Major Educational Reform Projects
|
Project |
Research Organization |
Urban Sites |
Participants |
Years in Operation |
|
LeTUS (Learning Technology in Urban Schools)
|
NWU U Michigan |
Chicago IL Detroit MI |
62 schools |
9 |
|
SYRCE (Systemic Research Collaborative for Education) |
U Texas |
Austin TX |
6 schools |
4 |
|
(SFT) School For Thought |
Vanderbilt |
Nashville TN |
125[2] |
6 |
|
http://peabody.vanderbilt.edu/projects/funded/sft/general/sfthome.html |
||||
|
Union City Online |
EDC |
Union City NJ School District |
11 schools |
6 |
[1] This section was written by Jay Lemke and replaces a much longer section IV mainly authored by Nora Sabelli in an earlier version of this paper. The issues it dealt with will be examined in a separate paper. The remainder of the present paper represents a synthesis of the views of both co-authors.
[2] The number is approximate because the configuration of Nashville schools and teachers was in a continuous state of flux throughout the project. Accordingly, many teachers changed schools throughout the project. This makes it difficult to provide an exact number of schools because many of the teachers take the reform with them but the project has no access to data on what they do after leaving, how the reform survives, or the impact on student learning.