Promising Practices
Using Narrative Methods to Link Program Evaluation and Organization Development
Charles McClintock, Dean of the Fielding Graduate Institute’s School
of Human and Organization Development, illustrates how narrative methods can
link a program’s evaluation and its organization development.
The field of program evaluation has evolved over the past half century, moving
from focusing primarily on research methods to embracing concepts such as utilization,
values, context, change, learning, strategy, politics, and organizational dynamics.
Along with this shift has come a broader epistemological perspective and wider
array of empirical methods—qualitative and mixed methods, responsive case
studies, participatory and empowerment action research, and interpretive and
constructivist versions of knowledge.
Still, evaluation has remained an essentially empirical endeavor that emphasizes
data collection and reporting and the underlying skills of research design,
measurement, and analysis. Related fields, such as organization development
(OD), differ from evaluation in their emphasis on skills like establishing trusting
and respectful relationships, communicating effectively, diagnosis, negotiation,
motivation, and change dynamics. The future of program evaluation should include
graduate education and professional training programs that deliberately blend
these two skill sets to produce a new kind of professional—a scholar-practitioner
who integrates objective reflection based on systematic inquiry with interventions
designed to improve policies and programs (McClintock, 2004).
Narrative methods represent a form of inquiry that has promise for integrating
evaluation and organization development. Narrative methods rely on various forms
of storytelling that, with regard to linking inquiry and change goals, have
many important attributes:
- Storytelling lends itself to participatory change processes because it relies
on people to make sense of their own experiences and environments.
- Stories can be used to focus on particular interventions while also reflecting
on the array of contextual factors that influence outcomes.
- Stories can be systematically gathered and claims verified from independent
sources or methods.
- Narrative data can be analyzed using existing conceptual frameworks or assessed
for emergent themes.
- Narrative methods can be integrated into ongoing organizational processes
to aid in program planning, decision making, and strategic management.
The following sketches describe narrative methods that have somewhat different
purposes and procedures. They share a focus on formative evaluation, or improving
the program during its evaluation, though in several instances they can contribute
to summative assessment of outcomes. For purposes of comparison, the methods
are organized into three groups: those that are relatively structured around
success, those whose themes are emergent, and those that are linked to a theory
of change.
Narratives Structured Around Success
Dart and Davies (2003) propose a method they call the most significant change
(MSC) technique and describe how it was applied to the evaluation of a large-scale
agricultural extension program in Australia. This method is highly structured
and designed to engage all levels of the system from program clients and front-line
staff to statewide decision makers and funders, as well as university and industry
partners. The MSC process involves the following steps:
- Identify domains of inquiry for storytelling (e.g., changes in decision-making
skills or farm profitability).
- Develop a format for data collection (e.g., story title, what happened,
when, and why the change was considered significant).
- Select stories by voting at multiple levels (e.g., front-line staff, statewide
decision makers and funders) on those accounts that best represent a program’s
values and desired outcomes.
- Conduct a content analysis of all stories (including those not selected
in the voting) in relation to a program logic model.¹
As described by Dart and Davies (2003), one of the most important results of
MSC was that the story selection process surfaced differing values and desired
outcomes for the program. In other words, the evaluation storytelling process
was at least as important as the evaluation data in the stories. In addition,
a follow-up case study of MSC revealed that it had increased involvement and
interest in evaluation, caused participants at all levels to understand better
the program outcomes and the dynamics that influence them, and facilitated strategic
planning and resource allocation toward the most highly valued directions. This
is a good illustration of narrative method linking inquiry and OD needs.
A related narrative method, structured to gather stories about both positive
and negative outcomes, is called the success case method (Brinkerhoff,
2003 [A review of this book is available in this
issue.]). The method has been most frequently used to evaluate staff training
and related human resource programs, although conceptually it could be applied
to other programs as well.
This method has two phases. A very short email or mail survey is sent to all
program participants to identify those for whom the training made a difference
and those for whom it did not. Second, extreme cases are selected from those
two ends of the success continuum and respondents are asked to tell stories
about both the features of the training that were or were not helpful as well
as other organizational factors that facilitated or impeded success (e.g., support
from supervisors and performance incentives). Based on the logic of journalism
and legal inquiry, independent evidence is sought during these storytelling
interviews that would corroborate the success claims.
The purpose of the success case method is not just to evaluate the training,
but to identify those aspects of training that were critical—alone or in
interaction with other organizational factors. In this way, the stories serve
both to document outcomes, but also to guide management about needed organizational
changes that will accomplish broader organizational performance goals. Kibel
(1999) describes a related success story method that involves more complex data
gathering and scoring procedures and that is designed for a broader range of
human service programs.
Narratives With Emerging Themes
A different approach to narrative methods is found within qualitative case
studies (Costantino & Greene, 2003). Here, stories are used to understand
context, culture, and participants’ experiences in relation to program
activities and outcomes. As with most case studies, this method can require
site visits, review of documents, participant observation, and personal and
telephone interviews. The authors changed their original practice of summarizing
stories to include verbatim transcripts, some of which contained interwoven
mini stories. In this way they were able to portray a much richer picture of
the program (itself an intergenerational storytelling program) and of relationships
among participants and staff, and they were able to use stories as a significant
part of the reported data.
Nelson (1998) describes a similar approach that uses both individual and group
storytelling in evaluating youth development and risk prevention programs.
The individual stories elicit participant experiences through a series of prompts,
while the group stories are created by having each group member add to a narrative
about a fictitious individual who participates in the program and then has a
set of future life outcomes. Group storytelling is a means of getting at experiences
an individual is reluctant to claim or at material that might not be accessible
to conscious thought.
Both of these approaches can result in wide differences in the quality and
detail of the stories. Especially with group storytelling, the narrative can
become highly exaggerated. The point of narrative in these instances is not
so much to portray factual material as it is to convey the psychological experience
of being in the program. Analysis can take many forms, depending on the conceptual
framework or evaluation contract, and can include thematic coding, verbatim
quotes, and narrative stories as the substance of the analysis.
Narratives Linked to a Theory of Change
The previous uses of narrative emphasize inquiry more than OD perspectives.
Appreciative inquiry (AI) represents the opposite emphasis, although
it relies heavily on data collection and analysis (Barrett & Fry, 2002).
The AI method evolved over time within the OD field as a form of inquiry designed
to identify potential for innovation and motivation in organizational groups.
AI is an attempt to move away from deficit and problem-solving orientations
common to most evaluation and OD work and move toward “peak positive experiences”
that occur within organizations. AI uses explicitly collaborative interviewing
and narrative methods in its effort to draw on the power of social constructionism
to shape the future. AI is based on social constructionism’s concept that
what you look for is what you will find, and where you think you are going is
where you will end up.
The AI approach involves several structured phases of systematic inquiry into
peak experiences and their causes, along with creative ideas about how to sustain
current valued innovations in the organizational process. Stories are shared
among stakeholders as part of the analysis and the process to plan change. AI
can include attention to problems and can blend with evaluation that emphasizes
accountability, but it is decidedly effective as a means of socially creating
provocative innovations that will sustain progress.
This brief overview of narrative methods shows promise for drawing more explicit
connections between the fields of program evaluation and OD. In addition, training
in the use of narrative methods is one means of integrating the skill sets and
goals of each profession to sustain and improve programs.
¹ A logic model illustrates how
the program’s activities connect to the outcomes it is trying to achieve.
References
Barrett, F., & Fry, R. (2002). Appreciative inquiry in action: The unfolding
of a provocative invitation. In R. Fry, F. Barrett, J. Seiling, & D. Whitney
(Eds.), Appreciative inquiry and organizational transformation: Reports from
the field. Westport, CT: Quorum Books.
Brinkerhoff, R. O. (2003). The success case method: Find out quickly what’s
working and what’s not. San Francisco: Berrett-Koehler Publishers.
Costantino, R. D., & Greene, J. C. (2003). Reflections on the use of narrative
in evaluation. American Journal of Evaluation, 24(1), 35–49.
Dart, J., & Davies, R. (2003). A dialogical, story-based evaluation tool:
The most significant change technique. American Journal of Evaluation, 24(2),
137–155.
Kibel, B. M. (1999). Success stories as hard data: An introduction to results
mapping. New York: Kluwer/Plenum.
McClintock, C. (2004). The scholar-practitioner model. In A. DiStefano, K.
E. Rudestam, & R. J. Silverman (Eds.), Encyclopedia of distributed learning
(pp. 393–396). Thousand Oaks, CA: Sage.
Nelson, A. (1998). Storytelling for prevention. Evergreen, CO: The
WHEEL Council.
Charles McClintock, Ph.D.
Dean
School of Human and Organizational Development
Fielding Graduate Institute
2112 Santa Barbara Street
Santa Barbara, CA 93105
Tel: 805-687-1099
Email: cmcclintock@fielding.edu
> Go to next article
> Go back to Table of Contents
|
