Ask the Expert
Pioneers in the Field: Four Foundations on Advocacy Evaluation
Representatives from four foundations discuss their expectations and approaches
for assessing their advocacy and public policy grantmaking.
The use of advocacy to inform public policy or systems change is an important
grantmaking strategy for many foundations dedicated to achieving sustainable
social change. However, as many articles in this issue attest, advocacy grants
are not easily assessed using traditional program evaluation techniques. Foundations
are eager for evaluation tools and approaches that help them make informed funding
decisions and help advocacy grantees to assess their progress and effectiveness.
Until recently, few resources existed to guide evaluation in this area. In
the last year, however, several foundations have supported the development of
guiding principles and practical tools—many of which are featured in this
issue—that are helping to push the field forward, grounding it in useful
frameworks and a common language. In addition, several have started an informal
“collaborative” to share funding ideas and coordinate their efforts.
This article features interviews with staff at four foundations—The California
Endowment, The Atlantic Philanthropies, the Annie E. Casey Foundation, and the
W. K. Kellogg Foundation—that are helping to build the advocacy evaluation
field. We asked each foundation the following questions about their advocacy
and public policy grantmaking and evaluation:
- What role does advocacy play in your grantmaking?
- What do you want to know from evaluation about your advocacy investments?
- How are you supporting grantees on advocacy evaluation?
- How are you helping to build the larger advocacy evaluation field?
The California Endowment
Astrid Hendricks-Smith, Director of Evaluation, and Barbara Masters,
Director of Public Policy
What role does advocacy play in your grantmaking?
The California Endowment's mission is to make quality health care more accessible
and affordable for underserved individuals and communities and to make fundamental
improvements in the health status of all Californians. We recognize that doing
this in a significant and sustainable way requires policy and systems change.
As a result, public policy work cuts across all of our grantmaking. In addition
to funding advocacy directly, we encourage our direct service grantees to consider
how they can contribute to the policy process.
We fund advocacy at the local, state, and national levels and support a variety
of activities to inform policy, including research, community organizing, coalition
building, and communications. Our funding also connects advocates, grassroots
organizations, and researchers to achieve collectively the kinds of policy and
systems changes we're seeking.
What do you want to know from evaluation about your advocacy investments?
Our evaluation interests lie at multiple levels. On one level, we want to
know how and where our grantees are having an impact in the policy process.
On another, we want to know which of the advocacy strategies we fund are more
or less successful. Finally, we want to know which organizations are most effective
and why so that we can learn how to help all our grantees become better advocates.
Methodologically, we fund a spectrum of evaluation approaches. We expect all
evaluation to credibly and defensibly assess what grantees have accomplished,
but in some respects, our expectations differ across grants. For example, we
expect the external environment to factor in differently for our advocacy and
direct service grants. With direct services, we examine start up and scale up
and consider the factors that affect our grantees' operating environments. With
advocacy, we know that advocates can't control everything within the policy
process. Consequently, we sort out what they can control and monitor that to
see if they have been effective.
How are you supporting grantees on advocacy evaluation?
A major takeaway from the work we've done so far on advocacy evaluation is
that evaluation is a tool and should be integral to the overall advocacy strategy.
We don't want grantees to do evaluation after the fact and rely on memory to
assess impact, and we want to get away from the notion that evaluation is punitive.
Rather, we want evaluation to be seen as a means to help the grantee reflect,
in real time, on their advocacy strategies and assess whether they're working.
Whether theirs is a multigrant initiative or an individual organization, we
want grantees to develop evaluative skills and build evaluation into their day-to-day
work. We are now trying to take this idea from theory to reality.
We also recognize that we have to be partners with grantees to help them develop
the needed skills to achieve policy and systems change. To that end, through
the Center for Healthy Communities, we have developed a Health Exchange Academy,
which offers training modules on advocacy, communications, and evaluation.
Similarly, we are helping our evaluators and program officers develop and utilize
the tools to support advocates and understand the policy world. We are giving
them a better sense of how the policy process works and instruction on how to
help organizations use evaluation for monitoring and management purposes in
unpredictable environments.
Lastly, in conjunction with our policy advocacy grantees that receive general
operating grants, we are carrying out an evaluation to determine the benefits
and downsides of this grantmaking tool. This evaluation will help us learn what
kinds of capacities organizations need in order to be good advocates. The Endowment
provides general support to policy and advocacy organizations very selectively,
primarily to those organizations that are best able to utilize the flexibility
they provide.
How are you helping to build the larger advocacy evaluation field?
We are excited to be involved in this field and are doing several things to
help shape it. Internally, our evaluation and public policy departments do this
work collaboratively, which has enabled us to bring our respective expertise
and perspectives to the project.
We're funding the development of research and tools that advocates, evaluators,
and funders can use. For example, we supported the prospec-tive evaluation framework
that Blueprint Research and Design developed for advocacy and policy change
evaluation (featured in this article). We're also funding
the development of case studies because we tend to learn and want to try new
things when we see other people doing them. Recently we published a case study
about the evaluation of our obesity prevention efforts that resulted in state
laws banning junk food and soda sales in the state's public schools.
Also, it is important that the field have cross-sector conversations about
what we're learning. We need evaluators, policy people, advocates, and funders
engaged in this dialogue. We all speak different languages, and only through
these conversations can we break down barriers and develop evaluation that is
acceptable for all stakeholders. Last year, we held a meeting that involved
all of these groups, and we plan to continue this cross-sector dialogue whenever
possible.
Finally, we want to support other foundations that are not yet funding advocacy.
The California Endowment already is committed to public policy work and sees
it as an effective vehicle for social change. But, for other foundations that
are not yet convinced, we need evaluations that enable funders to understand
how progress is measured and to see its value to achieving the foundation's
goals.
The Atlantic Philanthropies
Jackie Williams Kaye, Strategic Learning and Evaluation Executive
What role does advocacy play in your grantmaking?
The Atlantic Philanthropies are dedicated to making lasting changes in the
lives of disadvantaged and vulnerable people. We focus on critical social problems
related to aging, disadvantaged children and youth, population health, and reconciliation
and human rights. Improving the lives of intended beneficiaries in these areas
requires enhancing their currently marginalized voices. We also need to increase
their access to high-quality services. So, to us, advocacy is important on both
accounts.
Also, in keeping with the philanthropic philosophy of our founder, Atlantic
believes that committing resources over a limited period will maximize impact
and plans to complete active grantmaking by 2016. Therefore, we seek changes
that will endure beyond the foundation. Policy change is a strategic component
for achieving that objective.
Atlantic supports several kinds of advocacy for both policy change and increased
access to effective service delivery models. We support judicial advocacy through
strategic litigation and campaigns, legislative advocacy to enact and implement
policy, targeted advocacy campaigns to reach specific decision makers, and broader
awareness and education campaigns.
What do you want to know from evaluation about your advocacy investments?
We want to help our grantees improve their work and to increase Atlantic's
effectiveness as we spend down. This translates into wanting practical knowledge
that can be applied. Our approach is action oriented rather than academic (although
we believe that action should be research based). We want data that can be used
quickly so people can make decisions and shift strategies as appropriate.
When we explore possible grants with organizations, we don't expect them necessarily
to have a strong evaluation system in place because we understand that there
are resource issues involved. Instead, we look for a commitment to evaluation
and an ability to articulate the questions they'd like answered. When we find
that mindset, we support evaluation that helps answer their questions.
Evaluation of advocacy is interesting because the end goal usually is clear
and easy to measure. Less clear is what happens along the way and the lessons
for advocates working toward different policy outcomes. We want advocacy and
advocacy evaluation to have a clear rationale and theory of change, but we also
recognize that the most useful learning comes from understanding how advocacy
campaigns can be flexible in their operations and tactics. Now we are seeing
theories of change that reflect contingencies about how policy change might
occur. People are thinking about what could happen and the various pathways
they might take to achieve intended policy outcomes. Evaluation brings a mindset
and ability to think about those strategic issues; it elicits a “what if”
mentality.
How are you supporting grantees on advocacy evaluation?
My personal desire is to eventually have evaluation integrated into nonprofit
work, so that on a grantee's logic model, for example, the activities column
includes evaluation as a core organizational activity that supports all others.
Integration to me means more than developing internal evaluation capacity
to replace external evaluations. Grantees can integrate evaluation by partnering
with a good external evaluator. Grantees should have the internal systems and
skills so that they don't have to rely on an external evaluator to help
them access data when they need it. But I also believe that, too often, we expect
organizations to take on more than they are able. There are reasons that evaluators
have specialized skills; they have education and training that often make it
useful to have external evaluators step in. We want Atlantic program executives
and grantees to have the evaluation skills and knowledge that help them decide
when to do things themselves and when more expertise is needed.
How are you helping to build the larger advocacy evaluation field?
I think our approach to grantmaking and evaluation is helping to build this
field with other funders. There are three elements I think funders should consider.
First is time frame. We need fewer funders asking for multiyear outcomes with
single-year grants. Either funders should provide support over multiple years
or align expectations with the grant's time frame. We want grantees' evaluation
plans to be realistic. We understand the long-term nature of policy change,
and we give grantees permission to focus on intermediate outcomes. We also think
this focus helps build the advocacy field, because how advocates achieve their
intermediate outcomes is often where the most transferable learning lies.
Second, Atlantic provides capacity-building support, not just project support.
It is hard for grantees to integrate evaluation if we fund only project-based
evaluation. Project evaluations give no incentive to invest in evaluation systems
that are useful in the long term.
Third, we provide direct evaluation support. Overall in philanthropy, very
few dollars go directly for evaluation. Certainly there are project evaluations,
but funders often fail to help organizations really commit to evaluation. For
example, Atlantic supports information technology systems, which few people
think of as evaluation related. If we want grantees to use data to make decisions,
they need the systems that enable them to do that.
The Annie E. Casey Foundation
Thomas Kelly, Evaluation Manager
What role does advocacy play in your grantmaking?
The Annie E. Casey Foundation believes that policy and systems change are
avenues for achieving large-scale results for vulnerable children and families.
For that reason, advocacy to achieve such change is a central part of our grantmaking.
Many of our initiatives, place-based grants, and individual grants support
community, state, or national advocacy. For example, our major initiative, KIDS
COUNT, is a network of state child advocates in all 50 states, the District
of Columbia, Puerto Rico, and the Virgin Islands. Our grantmaking funds a range
of advocacy activities, including community organizing, outreach campaigns,
targeted issue advocacy (e.g., child health insurance, predatory mortgage lending),
and research.
What do you want to know from evaluation about your advocacy investments?
Our desired outcomes foundation-wide are in three areas—impact, influence,
and leverage. Our challenge is to more clearly define what these areas mean
and how to measure them.
Currently, we are looking at impact, influence, and leverage across all of
our major investment areas, including advocacy. We know we can't be clear about
our expectations for our advocacy grantees until we're clear about our expectations
for ourselves. We want evaluation to help us be more transparent about our work
and to instruct us on how best to invest our limited resources. For example,
we want to know which outreach strategies not only raise public awareness but
also generate the public will that helps moves issues forward in the policy
process.
Because we know that advocacy is complex to measure, our measurement expectations
for advocates have been fairly low. We're in the process now of applying the
same rigor that we use for our more traditional service delivery work to advocacy.
Rigor doesn't mean prescribing a specific approach; it means getting clearer
about evaluation outcomes, measures, methods, audiences, and uses.
How are you supporting grantees on advocacy evaluation?
Our evaluation conversations are tailored to the different kinds of advocates
we fund. Many grantees are not traditional advocacy organizations but are neighborhood
service providers who can be politically influential.
The more traditional advocates, like KIDS COUNT grantees, already are having
conversations about getting better at measuring progress (see the article
with Kay Monaco, a New Mexico KIDS COUNT grantee). Because they don't want
cookie-cutter approaches that feel like directives, they are very engaged in
helping us figure out our approach. They generate evaluation ideas and provide
feedback on what we're developing, such as our new tool, A Guide to Measuring
Advocacy and Policy (featured in this article).
As we get clearer about our approach to advocacy evaluation, this group will
be the first to test it.
For the less traditional advocates, our conversations begin in a different
place. We use evaluative questioning to help them define their advocacy approaches.
This gets them focused on, for example, not only what they are advocating for,
but who their target audiences are and who they need to work with. Then, we
move onto how to measure what they are doing.
In the future, we'd like to use evaluation to help shape our training and
technical assistance for grantees new to advocacy. Knowing more about effective
tactics and strategies will help us know what knowledge and skills to transfer.
To date, we've been highly reliant on smart advocates who know how to do this
intuitively. Now, we want to identify more systematically how and why they are
effective in a way that is teachable.
How are you helping to build the larger advocacy evaluation field?
As a field, we shouldn't be handcuffed by the fact that we don't yet have
the perfect advocacy evaluation approaches and measures. We need to start somewhere
and test our ideas. That's where we are as a foundation.
We also are communicating and collaborating with other funders on what we're
doing and learning. One thing that that these discussions should do that they
don't now is challenge the assumption that all advocacy is good. Advocacy is
accountable for achieving policy-related outcomes. But should it also be accountable
to the communities and constituencies it serves? Should it strengthen participatory
democracy, for example, so that even if a policy isn't achieved, more residents
will have been involved in the political process? We need to examine advocacy's
accountability to audiences beyond funders.
A foundation colleague raised another question that I think is important for
the field. How can advocacy evaluation help people make more strategic choices
around what to advocate for and not just how to do it? For example, is
advocating for new child care service funding every year the best way to support
working families, or should we look at expanding child care tax credits to reach
even larger numbers of families? The field currently is focused on using evaluation
to do advocacy better and rightly so. But whether and how evaluation can help
us make strategic choices about advocacy positions are field-level questions
that deserve a placeholder for future deliberation.
W. K. Kellogg Foundation
Sheri Brady, Director of Public Policy
What role does advocacy play in your grantmaking?
Public policy and advocacy play an important role at the Kellogg Foundation
because many of the initiatives across our main program areas (health, food
systems and rural development, youth and education, and philanthropy and volunteerism)
are working toward systems change, which often requires changes in policy. We
fund grantees' efforts to realign public and private systems in ways that benefit
the communities they represent and serve.
We support advocacy that leads to long-lasting changes, leverages resources,
strengthens the voices of communities, and helps the Foundation to achieve its
mission and grantees to reach their goals. As a part of this support, we educate
grantees on how to comply with the rules and regulations that govern public
policy activities.
What do you most want to know from evaluation of your advocacy investments?
Evaluation has always been important to the Kellogg Foundation and is expected
for all of our grants, including those involving advocacy. Different types of
work, however, call for different ways of looking at evaluation.
With advocacy, we have to be careful. We can't count how many people were
fed or how many kids got shots. Ultimately, we want to know whether an appropriation
increased, if an existing policy position was sustained, or whether a new policy
was enacted. But getting there sometimes takes a long while. For that reason,
we need to look at what happens during the life of the project, capturing markers
or indicators that tell us if we are on the right track.
Many people are critical of the indicators advocates often track, saying they
are too output focused and not meaningful. For example, they often pick on things
like the number of newspaper mentions or number of legislators at a briefing.
I don't like the term “meaningful indicators.” Whether indicators
are meaningful depends on the organization doing the advocacy, how difficult
their issue is given the current policy climate, and their strategy. For example,
say an advocate had 10 legislators attend a briefing. For some issues, this
number might be low. For others, especially issues not currently at the top
of the policy agenda, it might be a major win. It might mean that a new issue
is gaining momentum and that the grantee is a recognized expert in that area.
Judging an indicator without context is dangerous. We shouldn't necessarily
assume that there are measures that are meaningful across all advocacy.
The Kellogg Foundation also expects advocacy to involve the people being affected
by the policies in question. One of our core values is that all people have
the ability to effect change in their lives and their communities. We want to
know whether and how advocacy efforts make that happen.
Related to this, we are interested in whether communities are better off when
our grants end. We not only want to know whether grantees informed policy; we
want to know the actual or likely effects of those policies on people and communities.
How are you supporting grantees on advocacy evaluation?
First, we make sure that all of our advocacy and public policy work is within
the guidelines of what the IRS allows. We educate both staff and grantees about
available advocacy options.
We also support evaluation. On one level this means with dollars; we can't
expect people to do evaluation if we don't pay for it. On another level this
means giving grantees the tools and technical assistance to do it. This includes
making sure they see a clear reason for doing evaluation in the first place.
Most advocates won't advocate for evaluation. I was recently on a panel, talking
about this topic. The speaker was introducing the panel and by the time he said
“evaluation,” people already were asleep.
We don't want to put more and more pressure on grantees to always do more.
But it's revealing that evaluation typically is considered a bur-den while other
kinds of “asks” like communications or collaboration generally are
not. We need to help advocates understand evaluation's benefits and how it will
help them figure out what they are doing right and where to adjust. Right now,
either we're not making that case effectively, or many advocates aren't hearing
it.
How are you helping to build the larger advocacy evaluation field?
We try to demonstrate with our grantmaking that doing policy work and systems
change requires longer term investments. Funders tend to make decisions based
on their own grantmaking cycles rather than the needs of the fields their investing
in. With policy and systems work, we can't just say arbitrarily, “Okay,
it's been 3 years, and it's time to move on.”
Also, the Kellogg Foundation has developed a number of tools on evaluation
generally that I think are useful in this arena. Moving forward, we are interested
in being involved in field-level discussions about how to adapt evaluation to
the unique needs of advocates, and about the roles funders can play in continuing
to build the field.
HFRP Staff
Email: hfrp_pubs@gse.harvard.edu
> Go to next article
> Go back to Table of Contents
|
