Evaluations to Watch
Strategies for Assessing Policy Change Efforts: A Prospective Approach
Justin Louie and Kendall Guthrie of Blueprint Research and Design outline
the steps for advocacy and policy change evaluators to follow in using a prospective
approach to evaluation.
Prospective evaluation defines a policy change project's short- and long-term
goals up front and then emphasizes evaluating advocates' progress toward those
goals throughout the life of the project. By more deeply integrating evaluation
with program implementation, prospective evaluation provides funders with indicators
of success long before policy change can be achieved. It also collects insights
that advocates can use to refine strategies and document impact to funders and
constituents.
Benefits of a prospective approach include:
- Insuring up-front alignment of expectations
- Providing a framework to assess impact and monitor progress
- Delivering feedback to refine strategy and implementation
- Encouraging advocate engagement in the evaluation process
- Promoting a learning culture
Prospective evaluation involves five basic steps:
1. Understand the context and policy environment. Grantees, funders,
and evaluators involved in assessing progress in policy change need high-quality
knowledge about the environment in which change occurs. Stakeholders should
know the institutions, decision-making process, and culture of public policy,
including:
- The decision makers (legislature, executive, administrative agency, court
system, or general public)
- The potential tactics (education, organizing, litigation, and mobilization)
- The political context (Which party is in power? What external pressures
on decision makers can help or hinder you?)
- Your potential opponents (Who is working against you?)
- Your allies (Who can you count on for help?)
- Your capacity to do the work (Should a window of opportunity open, what
skills, networks, and organizational capabilities can you use to take advantage
of it?)
2. Develop a theory of change. A theory of change explains how
and why a project's activities are expected to lead to desired policy changes.
It provides a road map to policy change, based on an assessment of the policy
environment in which you are working. In many cases, policy change will be just
one component of a larger social change strategy, and a theory of change can
also define how specific targeted policy changes relate to larger social change
goals.
Developing a theory of change need not be complex or time consuming. Often,
answering some guiding questions can help jump-start the process:
- What is the problem you're trying to solve?
- What will be different if you're successful?
- What activities will you undertake to achieve your goal?
- What factors will accelerate or inhibit progress?
A theory of change must adapt to the evolving policy environment. If a strategy
no longer looks like it will work and you decide to take another route, the
theory should map that change.
Related Resources
Available from The California Endowment at www.calendow.org:
Guthrie, K., Louie, J., David, T., & Crystal Foster, C. (2005).
The challenge of assessing policy and advocacy activities: Strategies
for a prospective evaluation approach.
Guthrie, K., Louie, J., & Crystal Foster, C. (2006). The challenge
of assessing policy and advocacy activities: Part II—Moving from
theory to practice.
|
3. Define benchmarks. Generally, policy change is a long-term
effort, demanding many years of work. Defining benchmarks to show progress along
the way is vital to an effective and useful evaluation.
Benchmarks are milestones set in advance that indicate progress. They should
include incremental progress in both achieving policy goals and building internal
capacity for policy advocacy. Capacity-building benchmarks are especially important
markers of long-term progress. These benchmarks indicate growth in an asset
that can be applied to both current and future projects. Advocates have more
control over and therefore can be held more accountable for capacity-building
goals.
4. Collect data. Data collection requires time, energy, money,
and organization. Often advocates feel that collecting data draws them away
from their “real” work. By keeping data collection simple, building
upon the data collection that organizations already conduct, focusing on data
that is meaningful to the advocate as well as the funder, and emphasizing learning
over accountability, you can ease the burden of advocates' data collection.
5. Use findings. Evaluations need to be useful to both funders
and advocates. In particular, evaluations need to answer questions advocates
have about their programs, and findings need to be relayed in time to improve
their work. At the beginning, evaluation questions must include those that most
interest advocates, and, throughout the evaluation, findings need to be presented
as close to real time as possible. Interim reports are likely to be more valuable
to advocates than a final report at project completion.
Justin Louie
Senior Analyst
Email: justin@blueprintrd.com
Kendall Guthrie
Consultant
Email: kkguthrie@comcast.net
Blueprint Research & Design, Inc.
720 Market Street, Suite 900
San Francisco, CA 94102
Tel: 415-677-9700.
> Go to next article
> Go back to Table of Contents
|
