Reconceptualizing the Goals and Process …

             Reconceptualizing the Goals and Process
                 of Educational Research Funding:
             Interconnecting Scholarship and Practice




                                      Nora Sabelli
                                Senior Program Director
                  Division of Research, Evaluation, and Communication
                              National Science Foundation1

                                      Chris Dede
                                       Professor
                         Education and Information Technology
                               George Mason University




1
    Any opinions, findings, conclusions, or recommendations expressed in this a
authors and do not necessarily reflect the views of the National Science Fo
                                           Introduction

        In this article the authors reflect on their learning while Senior Program Officers in the
Directorate for Education and Human Resources of the National Science Foundation, where they
have directed a number of grant programs that fund peer-reviewed educational research in science,
mathematics, and technology. Our collective experience includes research into instruction and
policy, learning and intelligent systems, applications of advanced technologies, and networking
infrastructures in education. This article is not a synthesis of the extensive prior literature on the
need to link scholarship to practice, but instead an invitation for dialogue about "the awful
reputation of education research" (Kaestle, 1993; 1997) from a perspective informed by policy and
funding experience. We argue for a new prioritization of research funding initiatives, highlighting
crucial types of educational studies currently undersupported.

       Our approach for increasing the impact of research funding on education is based on
reconceptualizing the relationship between scholarship and practice in the context of an urgent need
for sustainable, scaleable, high quality educational innovation directed towards all students.
Central to our funding strategy for interweaving educational research and practice is recognizing
that:

(a) Educational reform is an iterative process of continuous improvement requiring long-term
    research investments that evolve in their nature.

(b) Parallel to other research situations, scholarly contributions to educational improvement must
    involve an ongoing interplay between theory and experiment.

(c) If the education system is to change, the role of practitioners in the processes of
    experimentation and adaptation must become paramount.

(d) Sustainable improvement depends on transferring the ownership of innovation from the
    scholarly community to practitioners and policy makers in the evolving educational system.

Even when most supportive of practitioners, in the past educational research funding has not
focused sufficient attention on the limits of scholarship as sole mechanism for institutionalizing the
processes of innovation.

        The first two sections that follow summarize how researchers currently attempt to shape
practice, then describe mutually developed implementation studies as an alternative innovation
strategy with greater impact. The next two sections depict how educational practitioners currently
view the role of research in the process of innovation, then delineate alternative policies that could
improve the utilization of scholarship in applied settings. Our discussion focuses on what is
missing from current funding approaches for research directed at enhancing educational


                                                                                                    1
practice--not for all aspects of research.

The Research Context

        Historically, the relationship between researchers and practitioners has taken various
forms, some quite close and supportive (Kennedy, 1997; Wagner, 1997). However, with few
exceptions, researchers' strategies for inducing change in educational practice have emphasized the
transfer to teaching or to policy making of results and insights conceptualized by the scholarly
community. Funded projects have been driven mainly by goals of contributing to the accumulation
of scholarly knowledge; disseminating this knowledge to practitioners as materials, directives, or
rules has been seen as an uncoordinated responsibility of the investigators. This investment
strategy does not foster analyses of synergy and of comparative quality. To enable more scaleable
and sustainable research innovations, we suggest reprioritizing funding strategies to highlight
innovative types of studies that empower the improvement of practice in pervasive ways.

        The most common funding strategies for research-based educational improvement have
several limitations:

1. Because in general funders allocate resources via disconnected, relatively small awards,
   research insights about the educational system are often limited to isolated, minor findings
   about individual practice and collective policy, leading to outcomes that miss crucial systemic
   relationships and don't reinforce links among various components of schooling.

2. In funding program guidelines, learning frequently is assumed to occur only in classroom
   settings (based on curriculum, guided by teachers, managed by schooling organizations),
   rather than as a process distributed across all of everyday life and shaped by interactions with
   families, employers, and communities. This again leads to a narrow, disconnected perspective
   about educational innovation.
3. Scholarly peer-reviewed decision making about funding awards often focuses on research-
   driven issues peripheral to the concerns of practitioners and policy makers, or emphasizes
   studies of highly restricted situations from which results do not generalize to other contexts. .
   Since "all education practice is local," highly restricted studies are important, but do not build
   theories or test general hypotheses. They are design experiments whose reproducibility is
   unclear and therefore are (appropriately) seen by the general audience as of limited value.

4. The dissemination requirements of funding programs frequently do not obligate researchers to
   report their outcomes in ways that stress the types of evidence that educational implementers
   and the public find persuasive. As a result, practitioners seldom develop a deep understanding
   of a research innovation's goals, conceptual framework, implementation process, and



                                                                                                        2
   evaluation strategy.

Because of these and related problems, insights from current research may provide valid new ideas
for improvement in practice, but seldom make a general and sustainable impact on the field or
accumulate into well understood, coherent strategies for implementing multiple innovations
simultaneously.

         Even when research is conducted by practitioners, the use of "classical" investigative
strategies vitiates the impact of their findings. For example, usually teachers' "action research" on
their classrooms is shaped by a research-centric perspective on education as composed of
independent classroom units in which teaching and learning can be improved in major ways
through reflective analysis at the `micro' level. This intrinsically limited approach is mirrored by
parallel misconceptions about a separate policy making sphere in which research on aggregated
patterns and trends provides all the data needed for decision making. Funding programs geared to
practitioner or policy maker research too often replicate this "micro-practice/aggregate-policy"
disconnected model of educational intervention and therefore achieve little lasting impact.

        Despite these problems, funded research has led to many valid, useful models of pedagogy
(Darling-Hammond, 1997; Dede, 1998; National Commission on Teaching and America's Future,
1996), to policy commitments in support of research linked to educational improvement
(President's Committee of Advisors in Science and Technology, 1997), and to advances in our
understanding of how people learn (Bransford, Brown and Cocking, 1999; Donovan, Bransford
and Pellegrino, 1999). Collectively, however, insights from many exciting programs and projects
have not resulted in pervasive, widely accepted, sustainable improvements in actual classroom
practice; in a critical mass of effective models for educational improvement; or in supportive
interplay among researchers, schools, families, employers, and communities. Even when
education research deals with the right issues, too often its results remain invisible outside of the
scholarly community. The root causes of this shortfall rest partially with scholars themselves,
partially with the agencies that fund educational research, partially with the distributed, loosely
coupled nature of the educational system, and partially with a "quick fix with a silver bullet"
mindset in our culture. Also, because policy makers are slow to take into account fundamental
changes taking place in the context of educational practice, their actions often erroneously assume a
dormant societal and educational environment.

        Integrating current research on important pieces of the educational puzzle into next-
generation, overarching models of innovation is a task that calls for different funding scales and
mechanisms than typical today. Even when projects strive for greater descriptive understanding of
educational and societal dynamics, their emphasis is generally on laboratory or design experiment
studies that create atypical "islands of innovation." Only sizable implementation "testbeds" that


                                                                                                    3
deal with scalability, generalizability, local adaptation, and sustainability can advance these ideas
for improvement to the next level. However, despite the fact that practitioners and policy makers
are focused on exactly these issues, funding for such testbeds is currently only a small fraction of
the total research portfolio.

        As studies of the processes underlying educational change document (Cuban, 1990; Fullan,
1993), systemic relationships are crucial determinants of whether the implementation of a strategy
for improvement succeeds or fails in reaching its educational--as opposed to educational
research--objectives. In addition, a lack of systemic analytical frameworks makes it difficult for
researchers and practitioners to develop insights on how attempts at innovation interact. Also,
seldom are research methodologies in use by other scholarly fields brought to bear on educational
systems, even though these analytic methods have contributed to the understanding of designed,
human contexts (e.g., cities, corporations) and of institutions charged with rapid technology-based
improvement (e.g., hospitals). Such theory-based organizational and system-wide frameworks are
needed in order to identify understudied pressure points where new ideas could lead to coherent
strategies for educational improvement.

         For lack of such a systemic perspective, the educational research portfolios of both NSF
and the U.S. Department of Education have not permeated these agencies' implementation
portfolios (materials development, teacher professional development, organizational reform). In
part, this is because systemic issues have been explored almost exclusively in the limited context of
evaluating systemic projects rather than deeper studies of systemic processes in testbed situations.
This approach leaves undeveloped many opportunities for creating ongoing partnerships among
educators, parents, employers, disciplinary experts, and communities, relationships that could lead
to improvements in students' (and teachers') learning.
         The common practice of conducting studies of large-scale educational interventions as
"evaluations" does not create the deep research base needed. Evaluation methodologies and goals
focus primarily on evidence of effectiveness; this often omits process issues and crucial variables
outside the subsystem being studied. Also, such an approach removes the responsibility for
practitioners and policy makers to be active participants in assessing implementation strategies and
results. Too often, experiments and evaluations are imposed on schools in a way that does not
foster reflective processes that could accumulate learning; we see funding for policy maker- and
practitioner-centered processes of experimentation, assessment, and reflection as a crucial enabler
of sustained improvement.

Mutually Developed Implementation Studies as a Better Funding Strategy

       How could investigators and funders transcend these limitations in research funding and


                                                                                                        4
conduct? Reconceptualizing research priorities and processes to focus more on implementation
studies mutually developed by scholars, practitioners, and policy makers is a promising strategy to
develop sustainable impacts on practice. This agenda can be seen as driven by an interactive view
of the role of the target institution:




                          Institution's role            Program focus



                          Passive                       Demonstrations and
                          Dissemination                 Models



                          Interactive                   Adaptation experiments
                          Adaptation                    Seeding Change




We believe too little emphasis is placed on research funding for in-situ adapting, analyzing, and
scaling-up interventions and policies that, as isolated islands of innovation, have been successful in
some other educational context. Even less priority is given to modeling and generalizing the
coherent processes that led these innovations to succeed in design experiment settings.

        To achieve coherent, sustainable, and scaleable change, we posit that understanding the
process of innovating (i.e., of altering standard practices) is as important as studying its outcomes.
In the few instances when we have seen the evaluation of new strategies for change lead to deep
understandings of sustainability as well as a plan of action, the "evaluation" (both formative and
summative) was performed by the project team, led by a strong researcher. In this situation, the
objective was not only to "evaluate" the performance of the project--a worthy goal--but also to
model and institutionalize its gains. Such internal reflective learning could be part of all large-scale
implementations, rather than limited to projects undertaken with a specific innovation research
purpose or subsumed as a minor objective in an external evaluation of an implementation's
performance.

        Certainly, this type of research on the sustainability and scaling-up of reforms is expensive,
people-intensive, and time consuming; but these are not the only reasons why such studies are
seldom done. Implementation (systemic, applied) research does not fit well within conventional
scholarly academic career paths. These studies demand a multiplicity of theoretical and



                                                                                                       5
methodological perspectives and require researchers to share control of the investigative process
with practitioners and policy makers. When such close partnerships are in place, researchers must
not only relinquish sole power over the analytic process, but also act as brokers to guide and
mediate the reflective interactions of other stakeholders. This is a skill for which most
investigators are not prepared--in part because conceptual frameworks for this type of research are
not well developed, in part because this capability is best fostered through supporting changes in
practice within integrated implementation and research "testbeds" (an uncommon experience for
scholars).

As has been recently suggested by the President's Committee of Advisors in Science and
Technology (1997), if our ultimate goal is long-term, pervasive, quality educational improvement,
we must find ways to invest a "critical mass" of funds and human resources in clinical-type
research. This article argues for a definition of such research as reflective interplay between basic
research and practice, a process that is bi-directional and helps both sides evolve towards
increasingly sophisticated objectives.

Such a bi-directional process should not be construed as limiting funding to 'applied' research.
The strategy we propose follows the arguments presented by Stokes (1999) on 'use-driven'
fundamental research, that is, fundamental research on critical and fundamental issues of principled
practice. The example given by Stokes is the close interplay of molecular and clinical research
under NIH sponsorship. In discussing its relevance to learning research, we will use a term that
is in contraposition to 'use-driven': 'curiosity-based' research. Using the word `curiosity' helps to
focus the comparison on the reasons behind the research questions posed. Curiosity-driven
research questions arise from the theoretical and hypothetical dynamics of the research field, with
priorities driven by the investigators' interests. The difference between use-driven and curiosity-
driven research is not the 'basic' or 'applied' nature of the study; rather, it is the community of
practice in which the research objectives are embedded and that first appropriates its results.

An example from research on learning may help clarify what we mean.2 Take research on the
linking of students' activities out of school to their learning in classrooms (use-driven), or the
transfer of knowledge acquired via intuitive and implicit learning into formalized knowledge
(curiosity-driven). Similar research questions are posed in different terms, and the answers would
inform different types of application.(Bransford and Schwartz, 1999)

       In use-driven, clinical research relationship among scholars and practitioners,

2
 The difference between cognitive science research and learning research lies at the c
in NSF. ROLE received 180 preliminary proposals from a wide array of research commun
submited to NSF's education research programs in the past. In multiple cases, the pr




                                                                                                    6
implementation is not dissemination: the expected acceptance of recipes and materials for
innovation developed by others. Implementation is instead the reflective adaptation or recreation of
a process that enabled a similar group to succeed in another educational setting. Focusing on the
process as well as on outcomes enables practitioners to start with objectives consistent with their
own current problems and worldview, then evolve towards increasingly more powerful goals as
they reflectively adapt innovations. This type of interaction among scholars, policy makers,
practitioners, and other stakeholders in quality education undercuts the limitations of conventional
research discussed earlier and lies at the heart of effective, sustainable scaling-up through the
mutual evolution of goals and outcomes.

        The intent of the research endeavor shifts from achieving an expected outcome to planning
for continual, reflective evolution. In our view, even the best of predetermined goals too often lead
implementations that call for either locally unattainable or unimportant outcomes, or worse, to the
co-opting of the original goals. Evolving objectives, on the other hand, require developing a
shared, long-term vision of what excellence in education means. Some central questions in this
process of mutually conducted implementation research are what are the critical insights needed
here and now to plan for and achieve an educational system's long-term goals? What is the
existing knowledge base in a particular situation, and where are the pressure points in that context
for augmenting that knowledge? What types of intermediaries can aid or subvert the
institutionalization of an ongoing relationship between practice, policy making, and basic, applied,
and systemic research?

        Emerging information technologies offer intriguing ways of potentially creating virtual
"knowledge networking" communities among researchers, practitioners, and policy makers. Three
years ago, NSF instituted a new multidisciplinary funding program to examine the potential of
emerging information technologies in fostering "Knowledge and Distributed Intelligence" (KDI).
This initiative [http://www.ehr.nsf.gov/kdi/default.htm] was prompted by fundamental shifts that
new interactive media are creating in the process of science. Scientists are moving away from an
investigative process based on reading others' research results in journal publications as a means of
informing and guiding one's own scholarship. Instead, scientists are engaged in virtual
communities for creating, sharing, and mastering knowledge: exchanging real-time data,
deliberating alternative interpretations of that information, using "groupware" tools to discuss the
meaning of findings, and collectively evolving new conceptual frameworks.

        NSF calls this process "knowledge networking" and is funding a series of KDI
investigations to study these virtual communities both in the context of science and as a

ROLE questions differed from the ones they would have asked if submitting to a discip
program, but would contribute to their overall research plan.


                                                                                                   7
generalizable process that could enhance many forms of reflective human activity. Through
knowledge networking, an emergent intelligence appears in which the virtual community develops
a communal memory and wisdom that surpasses the individual contributions of each participant.
NSF is supporting studies of this process through its "Learning and Intelligent Systems" (LIS)
initiative within KDI. Both knowledge networking and emergent intelligence are important new
capabilities that could potentially transform implementation studies in education.

        Knowledge networking involves creating a community of mind (Dede, 1999). Through
sharing disparate data and diverse perspectives, a group develops an evolving understanding of a
complex topic. Over time, the group's conception of the issues continually expands and deepens,
at times broadening the range of fields and experiences seen as relevant. During these times, the
membership of a networking community grows to include participants who bring new perspectives
and backgrounds. Thus, a network is in longitudinal flux as an ever-larger cast of members
redefines how to conceptualize the topic; this involves a constant collective acculturation into new
ways of thinking and knowing. For example, in the context of educational reform, the participants
in a knowledge network might be teachers, administrators, parents, taxpayers, politicians, teacher
trainers, researchers, school board members, and other policymakers--each bringing differing
perspectives and knowledge across multiple educational settings. Communal learning is at the core
of the networking process.

        This collective learning mirrors the complex initial acculturation process that people who
wish to join a knowledge networking community must undergo to become effective participants.
This acculturation involves:

·   mastering a common language and a generally accepted set of theories and mental models (to
    provide a framework for communication),

·   inculcating communally defined processes of collecting and analyzing data (to enable sharing
    reliable information),

·   developing proficiency in design, reasoning, and argumentation (to facilitate the evolution of
    ideas), and

·   accepting a common set of values, such as respect for others' perspectives (to encourage wide
    participation).

Currently, the absence of these types of acculturation undercuts opportunities for sustained
educational reform. For example, the war between proponents of phonics and advocates of whole
language illustrates the dysfunctional dynamics that can occur without mutual acculturation
processes to enable reflective dialogue that can promote a discussion of when, for which children,



                                                                                                     8
and under what circumstances is each pedagogy more appropriate. Through tools for
representation, collaboration, and community building, new interactive media can create a
framework within which constructive interaction can occur. Advances in information technology
aid knowledge networking through providing rich sources of data, rapid information exchange,
sophisticated analytic tools, and the collective intellectual capacity to tackle the complex problems
that underlie educational innovation.

        The new types of funding initiatives we advocate could create educational "testbeds" to
explore networking design for a class of social knowledge communities that have properties
different from those of virtual scientific research networks. Transfer of knowledge among
members of such educational communities--in ways that enable them to adapt solutions to their
own contexts--is not simply exchange of superficial information about successful innovations. It
requires engagement in rich, artifact-focused dialogue that provides detail about the nature and
processes of education reform. For example, educators can undertake collaborative annotation of
video-based case studies of educational practice that include ancillary information such as student
products and teacher reflections (Jacobs et al. 1997). Knowledge networking technologies
represent a new and powerful strategy for enhancing innovation sharing and adaptation, since
practitioners and policy makers find direct knowledge about each others' practices much more
convincing than conventional forms of research evidence.

        Typically, when islands of innovation in education emerge, they do not scale-up or transfer
and eventually wither away. Evolving small-scale innovations into widespread shifts in standard
operating practices necessitates changes in organizational structures, individual beliefs and values,
and community commitment. By building bridges from reflective innovation to standard practice,
testbed knowledge networks provide an excellent venue for exploring scalability and sustainability
in educational innovation. However, this requires reconceptualizing traditional funding strategies
for linearly transferring innovations from research to practice. Instead, researchers become
intermediaries among practitioners, and practitioners become co-developers of innovative models.

        In this article, by identifying what we see as major gaps in current research studies, our
focus is on defining a balanced portfolio of education research activities, itself embedded in a body
of learning research. To a great degree, the need to revisit how research impacts practice is a
reflection of a general paradigm shift in implementing innovations, focusing on `systems' that
resist or empower change (Senge, 1999). In recent federal funding programs (e.g., the
Interagency Education Research Initiative, IERI, http://www.nsf.gov/IERI; and the Research On
Learning and Education Program, ROLE, http://www.nsf.gov/pubs/2000/nsf0017/nsf0017.html,
NSF 00-17), an important emerging emphasis is exploration of the educational system qua



                                                                                                        9
system. Priority is given to projects that explore the crucial interplay between policy goals,
research outcomes, and actual classroom changes.

         We believe that a greater emphasis should be placed by the educational research community
and its funding organizations on:

1. Developing generalizable conceptual frameworks that coherently advance knowledge in support
   of sustainable improvements in teaching, learning, cognition, motivation, and problem solving
   for all students.

2. Conducting large-scale interventions as design experiments (Collins, 1996, Brown, 1992)
   based on innovations with the potential of enabling all learners to master more complex content
   via sophisticated and adaptable pedagogies, often enhanced through technology.

3. Conducting a reflective ("taking stock") cross-intervention analysis of the implementation
   process itself, so that researchers, practitioners, parents, and policy makers can understand the
   process, modify it, and achieve more effective learning environments that serve the needs of all
   students.

        These goals can be accomplished by a balanced portfolio of research studies, undertaken
not linearly but in a richly intergenerative fashion, categorized by methodology and objectives as
follows:

1. basic research and small, focused laboratory-type experiments: designed to advance
   foundational knowledge (for example, cognitive science research, some aspects of cognitive
   neuroscience and neural networks, studies of how naive learners and experts in a given area
   organize their learning, effects of cognitive tool use in stages of intellectual development, etc.)

2. design experiments and applied research: design experiments to advance the practices of
   teaching, learning, and organizational management (for example, longitudinal studies of the
   uses of scientific and mathematical modeling as pedagogy, environments that support learning
   of increasingly complex tasks, theories of instructional design in environments where learners
   have differing repertoires of learning styles, etc.)

3. data gathering and analysis studies of prior and existing practice and of intervention
   experiments: designed to evaluate and rationalize outcomes across laboratory and intervention
   studies (for example, surveys of teacher certification in the topic being taught, issues of gender
   and career choices, etc.)

4. systemic research on reform consortia and on large-scale intervention experiments:
   designed to advance the implementation of intervention experiments and research-based


                                                                                                     10
   innovations in large-scale and complex educational systems (research and implementation
   testbeds in specific contexts, studies of reform consortia, comparison of the economics of
   scaling-up models, role of motivation and intellectual ownership in sustaining reform, social
   and ethnographic studies, policy research, economic co-dependence of publishers, software
   developers, other vendors and schools, etc.)

In summary, we see the need for reflective and adaptive research methodologies applied to the
study of complex educational systems by these systems themselves, in collaboration with
researchers. But how do practitioners and policy makers currently view the role of research?

The Educational System Context

        Traditionally, the primary goal of educational practitioners and policy makers has been to
efficiently follow standard operating procedures, modifying these only when current learning
outcomes fall dramatically short of society's needs and expectations. In the event that innovations
are necessary to remediate shortfalls, the interventions attempted are seldom based on prior
knowledge, on validated experience, or on research evidence; but instead often emerge from
political expediency or currently popular strategies for change. Implementation projects seldom
monitor relevant research and generally limit their goals to "fine-tuning" current approaches rather
than searching for more ambitious visions. Too often, practitioners and policy makers reject as
"impractical" a reflective, iterative approach to implementation, thereby undercutting the leverage
research might provide in developing strategic objectives and effective evolutionary processes for
improvement.

         As discussed earlier, intervention experiments are one means to introduce alternative,
reflective innovation strategies into a large complex context. However, in order to succeed,
implementation research must be based on insights from design experiments and must focus on
reflective adaptation to given contexts. Unfortunately, in practice this is not often the case; the
balance between "general," "generalizable," and "localized" knowledge acquired from educational
interventions is usually not clear or even present in reports on their effectiveness. In part, this is
because the demand of practitioners and policy makers for immediate results washes out the
development of general ideas or generalizes ideas too soon or rediscovers what is already known.
Interventions mandated by local boards, politicians, or the public are typically not viewed as
applied experiments that build on laboratory trials through iterative refinement to match a particular
situation. Rather, they are seen as `silver bullets' that will magically solve a pressing problem. The
question too frequently asked (what works) is inferior to the perspective how can we make what
works, work for us? Workable, sustainable solutions are not jumps from one stable state to
another, but instead follow a self-correcting, at times meandering, path.



                                                                                                    11
        A variety of problems plague attempts by educators to initiate research-based
implementation studies. Performers and audiences are often different for laboratory research and
implementation studies; in the absence of common participants, building connections among these
types of experiments is difficult. Organizations that contribute to educational inertia, such as
standardized testing corporations and textbook companies, are seldom included in our models of
change; yet involving these types of stakeholders in systemic innovation efforts is often vital to
success. These and other disconnects among basic research, implementation studies, and policy
making hamper aggregating the outcomes of experiments towards a larger goal and undercut the
ownership of change in those who must implement innovations.

        Since a compelling need exists for more clinical and implementation research, the
educational community must undertake the task of capacity building for a new generation of
applied researchers and research-informed practitioners. At present, most scholars know better
how to perform laboratory analyses than how to conduct large-scale implementation studies as
theory-building research (rather than evaluation). Most investigators do not have good mental and
methodological models of intervention and systemic research, which are similar to applied
scholarly activities in medicine and engineering where the relation between general knowledge and
local and individual interventions is crucial. In the absence of sophisticated implementation
strategies, practitioners and policy makers are skeptical of what they see as "ivory tower" remedies
and often inadvertently eschew or co-opt the core of innovations in the process of adapting them.
We need to fund the evolution of innovative, mutually evolved methodologies for this type of
research.

        In the fields of medicine and engineering, integrating new research insights into practice is
appropriately seen as a professional goal mutually achieved by theoreticians and practitioners.
Unfortunately, in education this evolution of practice is instead approached as an issue of control.
For example, the expectations of the medical research community about the role of physicians are
in sharp contrast to the views of educational researchers about teachers: Physicians are seen as
partners in the improvement of public health, but teachers are too often viewed as impediments to
implementing improvements in students' learning. Terms such as "teacher enhancement" illustrate
a perspective that views knowledge about educational practice as a one-way flow from experts and
researchers to teachers. Absent a supportive relationship in which the educational research
community acknowledges teachers and administrators as professionals with insights to offer, and
educators' institutions as learning organizations that shape the evolution of practice, those in the
field have few inducements to incorporate and sustain research-based improvements that reshape
standard operating procedures.

       Lagemann (1997) analyzed the history of educational research from the very useful


                                                                                                   12
perspective of how the professionalization of this field has influenced linking knowledge and
action in education. As a complementary perspective, limits on the professionalization of
practitioners in U.S. education (i.e., not supporting the development of their capacity for
independent judgment and knowledge of their subject matter and its pedagogy, not providing time
for their reflection and sharing ideas with colleagues) hamper linking knowledge and action.
Comparisons of the role of teachers in the U.S. and other countries--or of the role of teachers
relative to practitioners in other fields such as medicine and engineering--illuminate how
underestimating the professionalism of U.S. educational practitioners in turn undercuts the
sustainability and quality of systemic educational reform.

        Beyond these problems of disconnects among basic research, applied research, and
policy-- as well as between researchers and practitioners--the value of innovations is too often
reduced to and measured only by immediate improvements on standardized tests of student
performance. This focus, though useful within its limits, provides too narrow a framework for
evaluating the gains, successes and shortfalls of educational innovation's process and outcomes.
When taken as the only measure of worth--and this is how most discussions frame this issue--
evaluation based on these limited measures does not provide the public and policy makers with
enough knowledge for a continued progression of thoughtful innovations. Sustaining a coherent
strategy for higher-level student learning (outcomes largely not measured by current high stakes
tests) to meet the long-term needs of society then becomes very difficult. For preparing students
to compete in a knowledge-based global marketplace, an all-or-nothing improvement strategy that
is both focused on immediate results and designed for purposes of ranking rather formatively
enhancing learning is self-defeating.

        Change is a process, not an outcome; a `one shot' improvement implies that the
environment that nurtures the current failure is left unchanged. Funding initiatives should explore
ways that current measures of the directions and pace in which schools are moving can alter to
provide a constructive, formative relation among research, policy, and practice. Such funding can
aid in developing shared conceptions of educational reform that allow practitioners and researchers
to assess their continual individual and collective progress based on a dynamic view of the
educational system, in which standardized measures of success are only one part of the assessment
strategy.

Bridging the Research and Educational System Contexts

        We think the central problem in research funding is not only helping to enable success in
current innovation efforts, but also empowering continued, deeper improvements in learning
through an evolutionary process built into the structure of our society's educational system.



                                                                                                    13
Otherwise, educators will be inevitably confronted with recurring, discontinuous calls for
"reform." Other fields provide better models of evolutionary advance than does education, in
which change is seen as sporadic and following a step function. Every two decades the field goes
through a cathartic phase in which immense energy and resources (and political pain) are suddenly
invested in "revolutionizing" education to patch up many years of accumulating shortfalls.

        To avoid such unproductive and demoralizing attempts at instant innovation (Kaestle, 1993;
Kaestle, 1997), the research and policy communities must alter their perspective on the educational
system and reconceptualize schooling as a form of social organization that can learn from itself.
Many of the problems with reform arise from a view of the educational system as dependent on
outside largesse--funding, materials, "teacher enhancement," research, policies--in which all its
practitioners can do is "accept disseminated results" and "deliver instruction." Without mutual
respect and shared expectations among researchers, practitioners, policy makers, and local
communities, no model for educational experimentation and evolution will succeed. Even though
systemic models of innovation dominate organizational evolution outside of schooling and drive
the economics of the workplace, these are largely absent in both practitioners' and researchers'
views of educational innovation.

        In the context of the educational system, we believe enhancing reflective innovation and
evolution requires both researchers and practitioners to consider that:

·   No sustainable, coherent change can take place without reflective, continual experimentation3
    by small groups of students, teachers, and administrators within the target organizational
    context.

·   No seminal experimentation can occur without a research mentality (the formation of testable
    hypotheses, the habit of reflection), supported by appropriate resources and embedded into the
    cultural image of the institution as a learning organization.

·   The research ethos must exist not only inside the "system" as a single unit of analysis, but
    inside each of the critical components of the system (i.e., the practitioner, administrative and
    policy maker communities). Otherwise each component's approach to innovation will be
    shaped by divergent goals and measures. Under these adverse conditions, "reform" is
    necessarily sporadic, dependent on temporary circumstances, and unsustainable.

3
 The Office of Technology Assessment and a recent PCAST report found that R&D in educ
                                                      3
roughly 0.025 percent of all education and training expenditures, and shrinking rela
                                                       . This percentage is small
to R&D investments in other priority areas and to the overall percentage of US GNP in
                                       3
percent; technology-based industries spend While business and industry are taking adva
                                        .   closer to 10%)
the latest technological advances, education and training lag further and further beh
level of investment of 0.025%, coupled with the investment made for the most part to
outside the system itself, has led to a bankrupt, static, system.


                                                                                                       14
·   Not only strategies evolve; the meanings attached to goals also shift as we learn. Worthwhile
    strategic goals must not be set in stone; if frozen in time, the objectives sought will become
    inconsequential, eschewed, or co-opted.

·   Scaling-up does not only mean multiplying practices or disseminating outcomes, but also
    determining the contextual conditions for a similar process of innovation to take place.

·   Any sustainable change must at some point be `owned' by an individual, organization, or
    system rather than being seen as imposed by some external mandate.

How can research funding agencies encourage such a shift in the beliefs and values of researchers,
practitioners, and policy makers?

The impetus for these changes in the culture of educational innovation must initially come from the
research community. But research can play a significant role in sustainable educational innovation
only by developing and testing hypotheses in the challenging real world settings where
implementation problems surface. Funders and researchers must then ensure that the formulation,
testing, and modification of hypotheses and designs from their inception include practitioners'
perspectives and knowledge. For example, educational research in science and mathematics has
gradually moved closer to enactment in contexts of authentic instructional practice. This field has
progressed from laboratory studies and mass testing to detailed research in classrooms and careful
analysis of students' conceptual learning. We call for expanding this trend across all disciplinary
areas by giving much higher priority to research on whole systems of educational practice,
including schools, families, mass media. Such research is currently rare, for all the reasons
described earlier, but urgently needed.4

         To enable conducting the mutually developed, coherent implementation studies suggested
earlier as a crucial strategy for educational improvement, the scholarly community and its funders
must become more reflective and self-critical about the current process and goals of educational
research. Studies of education are similar in some (not all) ways to studies in the sciences, social
sciences, engineering, and mathematics. Analogous to research in engineering and the social
sciences, educational studies involve developing knowledge about designed, human contexts less
constant in their attributes than natural phenomena. As Salomon (1991) describes, educational
reform is a complex, multifaceted enterprise similar to aircraft design as a an activity that must
embrace complexity in order to reach a solution, and whose outcome includes a detailed blueprint
of how to reproduce the process as well as the pitfalls to avoid. The situations studied by
educational researchers can be seen as complex systems with sophisticated feedback and non-linear
causality, similar to biological or ecological systems, and can therefore benefit from integrated




                                                                                                 15
system research strategies (whose development should be supported more strongly by funding
agencies). Beyond independent scholarship, educational researchers also should play a role as
intermediaries who enable experts in other disciplines, educational practitioners, learners, funders,
and policy makers to understand these complex-system perspectives.

        As the field of education changes, the types of research requested and needed alter. In the
1980s, societal concern about educational outcomes led to a variety of descriptive studies designed
to assess and understand problems in performance. At the turn of the millennium, now that the
causes underlying educational dysfunctions are better understood, practitioners and policy makers
are asking researchers to focus on applied studies that improve practice in a sustainable, affordable,
and scaleable manner. Requests from the field for research results that inform practice have thus
far resulted primarily in evaluative studies that provide limited evidence on whether current
educational interventions are worth the cost and trouble involved in implementation. We believe
these questions should instead be answered through research funding focused on implementing
organizational and systemic processes that optimize investments in innovation and increase the
quality and depth of students' learning outcomes.

Another factor influencing the nature of educational research is the rapid advance of information
technology and the affordances new media provide for motivating learning and promoting change.
Studies of alternative approaches to learning that emerging technologies enable are an important
strategic investment for educational research (Dede, in press). As illustrated in our earlier
discussion of "knowledge networking," these innovations also might help resolve current
problems of schooling and of scaling-up change in a more effective and affordable way than
present strategies for improvement.

       In response to all these needs, we believe that a balanced portfolio of educational research
would include greater emphasis than funders currently provide for these types of initiatives:

·     developing a broader base of researchers and scholars capable of studying the complex
      situations characteristics of large-scale educational innovation in complex environments;

·     developing new types of research methodologies to address significant problems of
      practitioners and policy makers at appropriate levels of aggregation;

·     producing a series of research summaries that present validated current research results using
      the types of evidence practitioners and policy makers find convincing;

·     building partnerships with practitioners and policy makers in the design and analysis of
      mutually developed implementation studies;

4
    The authors thank Marcia Linn for contributing this comment.


                                                                                                   16
·   fostering the potential capability of emerging information technologies to enhance learning and
    teaching; to improve the organizational efficiency and effectiveness of schools; to enable
    partnerships for learning among teachers, parents, employers, and communities; and to
    develop new tools and methods that empower the process of knowledge generation and
    communication; and

·   funding proof-of-concept demonstrations that reproduce and validate an insight or innovation
    through pilot implementation in a realistic, authentic setting, rather than simply in laboratory
    trials.

       To assess the quality and potential of educational research, we need a balanced portfolio of
research capable of sustaining educational reform, including:

·   fundamental innovation rather than minor enhancements to existing practices

·   a focus on strategic issues of major importance to practitioners and policy makers throughout
    the multi-year time frame in which research outcomes are developed

·   the development of a conceptual framework based both on current challenges in educational
    practice and on emerging opportunities from potential innovations in technology or theory

·   closer research links to basic scientific research that has the potential to significant improve
    practice, or to shed light on human learning and coping strategies, and that can provide applied
    educational research with productive questions of relevance to human knowledge acquisition,
    retention, and generation

·   a long-term perspective, making convincing assumptions about coming advances in technology
    or knowledge that require immediate, strategic action to prepare educational practice and policy
    for the future

·   a model based on systems thinking rather than reductionist approaches to understanding the
    interrelationships between an educational intervention and the other processes occurring in its
    organizational context

·   an analysis of already existing, related research that enables this project to extend, complement,
    or refute similar studies

·   analysis of project products and outcomes, including testing the falsifiability of research
    hypotheses by controlled studies against "current best practice" alternatives to the proposed
    innovation

·   studying successes to "reverse engineer" their generalizable implications for learning, teaching,
    and organizational structure


                                                                                                    17
·   proof-of-concept demonstrations that validate an insight or innovation through pilot
    implementation in a realistic, authentic setting, rather than simply in laboratory trials

·   the generation of fundamental insights that generalize to advance the conceptual frameworks
    educators use in decision making, or the development of innovations in practice that transfer
    beyond the local context into many similar situations nationally

·   innovations that are affordable, scaleable, and sustainable

·   undertaking assertive and proactive dissemination strategies that both inform a broad base of
    researchers, practitioners, and policy makers and provide support for those interested in
    adopting or extending the innovation

Not all of these criteria are appropriate for every type of educational research, but they suggest a
strategy that could be used when making competitive judgments about what scholarship to propose
or what funding decisions lead to an effective research portfolio.

Conclusion

        Given that our nation's educational system must evolve to meet the needs of a global,
knowledge-based marketplace, then all the groups within that system must contribute towards
mutually agreed innovation strategies and outcomes. Any set of stakeholders that is not given
validity as a respected and empowered contributor towards improvement will (wittingly or
unwittingly) co-opt the change into trusted and known patterns of work and thus defeat any
purpose not mutually developed. This is why the prevalent conceptualization of educational
research as academic and separate from practice has direct deleterious effects on the relationship
between research, policy, and practice.

        Reconceptualizing research funding agendas, priorities, and processes to focus more on
implementation studies mutually developed by scholars, practitioners, and policy makers is a
promising strategy to remedy this situation. For such a reflective, evolutionary innovation process
to succeed, researchers, practitioners, and policy makers must all alter their views on the value and
conduct of research, as well as on the roles played by these three groups in research and
implementation. Acceptance of these shifts in assumptions and actions must initially come from the
research community and its funders; to initiate a dialogue, this article suggests first steps in this
direction.

                                             References

Bransford, J. D. and D. L. Schwartz (1999). Rethinking Transfer: A Simple Proposal with
 Multiple Implications. In A. Iran-Nejad & P. D. Pearson (Eds.), Review of Research in
 Education. Washington: AERA.


                                                                                                     18
Brown, A. L. (1992). Design experiments: Theoretical and methodological challenges in creating
 complex interventions. The Journal of the Learning Sciences, 2, 137-178.

Collins, A. (1996). Design issues for learning environments. In S. Vosniadou, E. D. Corte, R.
 Glaser, & H. Mandl (Eds.), International perspectives on the psychological foundations of
 technology-based learning environments. New York: Springer-Verlag

Cuban, L. (1990). Reforming again, again, and again. Educational Researcher, 19(1), 3-13.

Darling-Hammond, L. (1997). Teachers and teaching: Testing policy hypotheses from a national
 commission report, Educational Researcher, 27(1), 5-15.

Dede, C. (in press). Emerging Influences of Information Technology on School Curriculum.
 Journal of Curriculum Studies

Dede, C. (1999) The Role of Emerging Technologies for Knowledge Mobilization, Dissemination,
 and Use in Education. Washington, DC: U.S. Dept. of Education.

Dede, C. (1998). Learning with technology (the 1998 Yearbook of the Association for Supervision
 and Curriculum Development). Alexandria, VA: ASCD.

Fullan, M.G. (1993). Change forces: Probing the depths of educational reform. Bristol, PA:
 Falmer Press.

Jacobs, J.K., M. Yoshida, J.W. Stigler, and C. Fernandez. "Japanese and American Teachers'
 Evaluations of Mathematics Lessons: A New Technique for Exploring Beliefs." Journal of
 Mathematical Behavior, vol. 16, no. 1, 1997, pp. 7-24.

Kaestle, C. (1993). The awful reputation of educational research, Educational Researcher, 22
 (1), 23-31.

Kaestle, C. (1997). Improving the awful reputation of educational research, Educational
 Researcher, 26(7), 26-28.

Kennedy, M. (1997). The connection between research and practice, Educational Researcher,
 26(7), 4-12.

Lagemann, E.C. (1997). Contested terrain: A history of education and the United States, 1890-
 1990.Educational Researcher, 26(9), 5-17.

National Commission on Teaching and America's Future. (1996). What Matters Most: Teaching
 for America's Future. New York, NY: National Commission on Teaching and America's
 Future.


                                                                                                19
President's Committee of Advisors in Science and Technology. (1997). Report to the president
 on the use of technology to strengthen K-12 education in the United States. Washington,
 DC: USGPO.

Salomon, G. (1991). Transcending the qualitative/ quantitative debate: The analytic and systemic
 approaches to educational research. Educational Researcher,. 20 (6), 10-18.

Senge, P, Kleiner, A., Roberts, C., Ross, R., Roth, G., & Smith. B. (1999). The dance of
 change: The challenges of sustaining momentum in learning organizations. New York:
 Doubleday,

Wagner, J. (1997). The unavoidable intervention of educational research: A framework for
 reconsidering researcher-practitioner cooperation, Educational Researcher, 26(7), 13-22.




                                                                                               20