Issues and Opportunities in Out-of-School Time Evaluation Briefs

Why, When, and How to Use Evaluation:
Experts Speak Out

Number 5, June 2003

Heather Weiss and Priscilla Little, Harvard Family Research Project

Download this brief:

50KB Acrobat file

Help for Acrobat file

This brief is also available in hard copy for free. Please go to our publications list to learn how to order it.

Harvard Family Research Project’s (HFRP) Issues and Opportunities in Out-of-School Time Evaluation briefs are short, user-friendly documents that highlight current research and evaluation work in the out-of-school time field. These documents draw on HFRP’s research work in out-of-school time in order to provide practitioners, funders, evaluators, and policymakers with information to help inform their work. This latest brief offers expert commentary on the implications of When Schools Stay Open Late: The National Evaluation of the 21st-Century Community Learning Centers Program, First Year Findings1 for future evaluation and research.

Introduction

The 21st Century Community Learning Centers (21st CCLC) program began in 1998 under the Elementary and Secondary and Education Act, with $40 million awarded to 99 grantees in 34 states, supporting programs in about 360 schools. Reauthorized under the No Child Left Behind Act, the program received $1 billion in 2002. On February 3, 2003 the U.S. Department of Education released the first-year findings from the national 21st CCLC program evaluation. Conducted by Mathematica Policy Research, the national evaluation examined the characteristics and outcomes of typical 21st CCLC programs. Simultaneous with the report’s release, the President’s Fiscal Year 2004 Education Budget Summary and Background Information cited the “disappointing initial findings from a rigorous evaluation of the 21st Century Community Learning Centers program” as a rationale to request a decrease in funding for the program by 40 percent.2 According to the budget summary, “the evaluation indicates that the centers funded in the program’s first three years are not providing substantial academic content and do not appear to have a positive impact on student behavior.”3

The decision to recommend a 40 percent reduction in the 21st CCLC budget has stimulated renewed commitment to evaluation as well as efforts to examine this one evaluation in the larger context of other relevant evaluation and research. It also has catalyzed interest in developing a strong future research and evaluation agenda to support program development, improvement, and accountability. This brief provides commentaries from researchers, evaluators, and practitioners about new directions for research and evaluation in order to reframe the “gotcha accountability” game into one of learning for continuous improvement and accountability. Specifically, we asked, Given the recent push for science-based research, coupled with the release of the first-year evaluation findings from the 21st CCLC programs, where do we go from here to use research and evaluation to support the development of high-quality out-of-school time programs?

In a world where the principles of scientific research are increasingly invoked to guide policy, the decision to use the first-year data from an ongoing evaluation to cut funding is producing considerable scientific scrutiny and critique of the study’s methodology and findings—scrutiny that is crucial for scientific as well as policy advancement. We believe there should be an ongoing exchange regarding methodological choices, trade-offs, and issues about how to deal with the inevitable issues of “real-world” implementation in this and other large-scale evaluations. Therefore, this brief includes a commentary on the methodological issues in this particular evaluation and their implications for current policy as well as for future evaluation design, implementation, and use. It is our hope that this will stimulate a larger conversation about a productive research and evaluation agenda, about solutions to methodological problems, and about how to maximize the use of research and evaluation to support policy formation and service improvement.

Steve Gunderson
Manager, Washington Office, The Greystone Group, Inc.
Former Congressman from Wisconsin

In 1996, as part of the Congressional reauthorization of the Elementary and Secondary Education Act, I introduced legislation to create “community learning centers.” Our goal was to find ways to more efficiently use school resources, especially in rural and inner-city areas, for all citizens all year. The Clinton administration strategically directed this broad language to create today’s after school program, funded at $1 billion annually in FY 2003.

In the No Child Left Behind Act the law transitioned to a state grant program. Now, the administration seeks to add new standards to all federally funded programs which I call the “three A’s”—academics, access, and accountability. We’d be wise to positively respond to this new focus. After school programming is an important and growing component in the development of today’s youth. Yet we need to target these programs to those most in need, in ways that will enhance a student’s academic progress, assuring limited public dollars meet the test of accountability.

New research and evaluation is desperately needed to improve federal support for this program. Certainly one study (Mathematica’s) does not justify ending the program. But with limited resources and the new focus on academics, we must learn what works—especially for at-risk students. Then, we must restructure our programs to best achieve this goal. So, let’s get on with improving a good idea rather than defending the status quo. To do anything less is to contribute to the death of the most significant expansion in federal support for any K–12 education program in recent years.

Kathleen McCartney
Professor, Harvard University Graduate School of Education
Cambridge, Massachusetts

Hard-won lessons of evaluation research have been lost in the administration’s response to the Mathematica evaluation of the 21st CCLC program. To evaluate the administration’s response, ask yourself these five questions.4

1. Were the findings used as part of an ongoing innovation cycle? The answer is clearly no. Many child advocates had hoped that this evaluation would be used to promote continuous improvement. Instead, the administration has acted based on first-year data, collected during the implementation phase of the study.
2. How were the effect size data interpreted? The Mathematica researchers highlight in their executive summary that the small effect sizes were most likely due to the low attendance rates, the length of the follow-up period, and the lack of sustained, substantive academic support in most programs. Although it is easy to dismiss the effects as small, this conclusion is no doubt premature, especially in light of the fact that this is an ongoing evaluation.
3. Were the findings from the Mathematica study synthesized with existing data on after school programs in order to make an informed decision? No, again. Instead, the administration embraced the Mathematica report as providing the only relevant information with which to inform funding considerations.
4. Did the administration have fair and reasonable scientific expectations? Scholars agree that no one should expect the 21st CCLC program evaluation to yield short-term effects on tests scores, echoing Zigler’s early warnings concerning Head Start.5 By what criteria are the findings “disappointing”?
5. Were the findings subjected to professional scrutiny? Given that the administration’s recommendations coincided with the release of the report, the answer is no. This is the most troubling aspect of the administration’s response. Policy recommendations should not precede reactions from the scientific community.

Accountability efforts and scientifically based research can either be used to generate knowledge that informs effective practices or to serve as a political lever to cut programs and expenditures on child and family services. Here we have a sad example of that latter—another case of death by evaluation.6

Karen J. Pittman
Executive Director, Forum for Youth Investment
Washington, D.C.

The administration’s proposed cut to the 21st CCLC budget is not surprising. It is a rare elected official who expands rather than downsizes the pet programs of a predecessor from the opposing party. Dozens of social programs will suffer cutbacks in the next budget. What is surprising is that the administration has broken its own rules for bringing science into policy discussions. By announcing the cuts just as it released the report, opportunities for the research and policy community to apply the new rules of scientifically based research—conducting a rigorous, objective, and scientific review, discussing the findings, and debating responses in light of findings from other equally scientific studies—were effectively cut off.

Research should play a more central role in decisions to expand, redefine, or reduce programs. When used correctly, it can be a powerful counterweight to limit the big pendulum swings frequently associated with popular programs, to accelerate the growth of effective programs, and even to curtail the expansion of popular but ineffective programs. The Mathematica report includes promising findings and valuable lessons that can inform both practice and policy. This and other studies should serve as platforms for much needed conversations about how to augment program quality and encourage longer and more intense participation. By using the study to justify cuts, the administration has curtailed conversation about a range of responsible strategies for improving the program, given these and other findings. Our concern should be the same if the proposed program budget had been doubled.

For a more detailed version of this commentary, visit the Forum for Youth Investment website at www.forumforyouthinvestment.org/resspeech.htm.

Mindy DiSalvo
Program Director, Family Technology Resource Center
Decatur, Georgia

Because we want to know if our after school programs are the best that they can be, we eagerly welcomed the opportunity to be a part of the national evaluation of the 21st CCLC program. From the onset of the evaluation we were candid about our strengths and weaknesses, providing honest information about a program in its infancy. As a participant in the evaluation, I have two concerns related to the emphasis and use of the evaluation results.

First, there were very positive findings in the 21st CCLC evaluation—findings that could serve as a road map to improve existing and new programs, not close the doors to them. Based on our concurrent evaluation data, we expanded our curriculum, developed a parent/teacher/student homework completion policy, translated materials into two languages, extended hours of operation, and hired a nurse. We learned how to make our program better by using data. Therefore, it seems that the emphasis should be on how we can learn from the new evaluation report, and not what the report told us about academic impact.

Second, student achievement isn’t, and never will be, solely a result of after school programs. Student achievement isn’t a result of textbooks either, but we spend a fortune on them and no one is talking about cutting them from a budget! Improved student achievement is a result of a combination of components in a child’s life, including how they spend their nonschool hours. Before student achievement becomes a priority for many of our after school programs, a safe place with a caring adult, friends, a healthy snack, and a promise of security comes first.

Tiffany Berry
External Evaluator, LA’s BEST
Los Angeles, California

LA’s BEST uses evaluation data by transforming program outcomes into organizational tools for program improvement Since the program’s inception in 1988, LA’s BEST has placed a high priority on evaluation and we encourage feedback from the program’s diverse stakeholders. Monitoring of program quality has been accomplished by leveraging internal sources (e.g., random site visits by the board of directors, site activity logs, opportunities to communicate between field and management staff, etc.), as well as data from external sources (e.g., the Center for the Study of Evaluation at University of California Los Angeles). These data sources have yielded valuable insights, which have been fed back into program operations

One of the most robust findings of the LA’s BEST program relates to the duration and intensity of participation. Our evaluation reports indicate that when compared with nonparticipants, LA’s BEST participants have fewer days of absences from their regular school, higher achievement on standardized tests in mathematics, reading, and language arts, and higher language redesignation rates to English proficiency.7 In addition, we have found that the relationship between participation intensity in one academic year and academic achievement was mediated by regular school attendance. This suggests that participating in LA’s BEST resulted in better school attendance, which in turn related to higher academic achievement.

James P. Connell
President, Institute for Research and Reform in Education
Toms River, New Jersey

Common sense tells us that public investment in programs serving youth should start with a research-based rationale, or theory of change. This theory should tell us how and why the proposed activities, in this case after school programs, can reasonably be expected to produce the designated academic and social outcomes.

In the absence of alignment between program activities and expected outcomes, the failure of the 21st CCLC program to produce its desired outcomes was virtually preordained. Two remedies present themselves: start with the outcomes you want and change program activities to those with a reasonable shot at achieving the outcomes, or start with the activities you have and adjust your expectations to outcomes they can achieve.

Either course makes some sense. The 21st CCLC evaluators call for remedy number one—enriching after school programs with more research-based activities tied to the desired outcomes. Many commentaries on the evaluation make energetic pleas for remedy number two—holding programs accountable simply for providing positive activities for young people while many of their parents are at work.8

The following steps could lead to better alignment between after school programs, academic outcomes, and evaluation: (1) develop educational and recreational activities to help students meet a small number of broad academic standards that reflect the schools’ goals for their students, (2) give staff the resources to actively engage young people in these activities in different ways, and (3) assess the quality of implementation of these activities, their intended outcomes, and the connection between the two. We have seen such a theory of change approach help bring both realism and accountability to the work of changing public education. We expect it could do the same for after school programming.

Jacquelynne S. Eccles
McKeachie Collegiate Professor of Psychology, Women’s Studies, and Education, University of Michigan

As Chairperson for the National Research Council (NRC) committee that produced Community Programs to Promote Youth Development,9 my comments reflect my concern over the administration’s decision to cut the funds for the 21st CCLC program based on one evaluation report. This seems a very strange decision for an administration that stresses both the need for evidence-based practice and the importance of supporting healthy adolescent development. Our comprehensive report outlined the characteristics of many programs shown scientifically to have positive effects on many different aspects of adolescent development and provided examples of many high-quality programs with rigorously demonstrated effectiveness. We also discussed what is needed for adequate evaluation to improve these programs and make sound policy decisions. We proposed that the real challenge for the field is to increase the availability and sustainability of high-quality programs, especially in the context of unpredictable funding streams. We concluded that increased funding for the 21st CCLC program was one step the federal government could take to help increase the predictability of funding.

Consequently, I was appalled at the decision to cut these funds based on one quite-limited evaluation. Very little attention was paid in this evaluation to the characteristics of the programs being evaluated. Instead great attention was paid to the quasi-experimental and experimental evaluation designs used. As we discussed in the NRC report, these designs are powerful methodological tools, but they are not particularly useful if we do not know the quality of the programs being evaluated. If members of this administration truly value evidence-based practice, then they should pay more attention to the evidence-based reports that are so carefully put together by the NRC rather than use the results of one report to justify funding cuts. The existing evidence suggests to me that the administration should increase the funds for 21st CCLC program, but require better specification of the exact characteristics of the programs eligible for funding.

Where Do We Go From Here?
Heather Weiss, Director, Harvard Family Research Project
Cambridge, Massachusetts

We are now playing in a “new evaluation game” with new players and new rules. The game is different because not only is research and evaluation helping to shape policy, but the reverse also is true. Evaluation has always been played out within a political frame, but the No Child Left Behind Act helped define that frame by setting new rules or standards for research and evaluation with its five principles for scientifically based research (SBR) in education.

Not everyone has agreed that the new evaluation game is being played with the right set of rules. Some would argue that even if the end goal is to comply with this set of rules, there needs to be a learning curve where programs are held incrementally accountable for implementing the new scientifically based standards. For example, the Mathematica impact evaluation, underway well before the new SBR principles were signed into legislation, was prematurely subjected to the new rules and the result has had potentially dire policy consequences. As Pittman, Eccles, and McCartney point out in their commentaries, using one evaluation report to justify a policy decision runs counter to the new rules of SBR—in fact, they argue that subjecting the impact report to the new rules violates some of those rules’ very premises. Further, as Connell points out, in order to accurately assess program impact, there must be alignment between program activities and desired outcomes. In the case of the 21st CCLC programs, old programs were held accountable for new outcomes, thereby almost “preordaining” failure. Moving forward, all players must strive for alignment between desired outcomes and program strategies.

Despite disagreement over the rules, we are now in a position where, like it or not, the new rules are in play and we have to learn how to “get in the game.” So how do we play in this new game? There are at least three approaches.

First, as Gunderson points out, “we must learn what works,” and we must acknowledge the new game rules require after school programs to be accountable, demonstrate results, and improve their quality. However, as noted evaluator Mark Lipsey points out, “individual evaluation studies, however useful they may be to sponsors and stakeholders, yield approximate estimates of intervention effects and the relationships of those effects to the features of the program under assessment.” Further, he points out that perhaps the most useful and informative contribution to program managers and policymakers alike may be the consolidation of our piecemeal knowledge into broader pictures of the program and policy spaces at issue, rather than individual studies of specific programs.10

Second, we need to shift from a system of “gotcha accountability” to a system of learning for continuous improvement and accountability. In a recent Washington Post commentary, former US Secretary of Education Richard Riley notes that the original purpose of the evaluation was for continuous improvement, not accountability. He points out that the “Clinton administration did commission this study, but the purpose was to learn how to make after school programs even more effective… [not to provide] political cover”11 for proposed budget cuts. Our two practitioner commentators (DiSalvo and Berry) agree that, even in the context of heightened accountability, to be good evaluation players requires a commitment to using data for continuous improvement as well as to show impact. A thorough reading of the Mathematica report reveals many promising implementation findings that need to be brought into the light and used for program improvement.

Third, there is increasing precedent for independent researchers to reanalyze national evaluation data, particularly when there are substantial policy implications. Mathematica’s decision to make public the entire database of results from its New York voucher study to independent researchers for reanalysis led to discovery of methodological problems on the part of Mathematica’s research partner. This, in turn, substantiated Mathematica’s cautions about the study’s findings and refuted their research partner’s conclusion that voucher use among poor black children significantly improved test scores.12 As Vandell points out in her methodological commentary, some of the issues that she feels plague the recent 21st CCLC study could be addressed through a reanalysis of the data. Years of evaluation research have taught us lessons that are too expensive to learn again, such as don’t make large-scale investments in evaluation unless you are learning about program implementation along the way and don’t evaluate a program until it is proud.13 Moving forward, our responsibility as evaluators is to take advantage of the unintended window of opportunity provided to us by both old and new administrations to engage in a dialogue about how to apply these lessons to the new game to ensure that future research and evaluation of after school programs “plays by the rules,” and is used to improve the overall quality of after school programs.

This Issues and Opportunities in Out-of-School Time Evaluation brief was compiled by Heather Weiss, Director, and Priscilla Little, Project Manager of Harvard Family Research Project. We welcome additional methodological commentary and will post them to our website as they become available.

Notes

1 US Department of Education, Office of the Under Secretary. (2003). When schools stay open late: The national evaluation of the 21st-Century Community Learning Centers program, first year findings. Washington, DC: Author. Available at www.ed.gov/pubs/21cent/firstyear.

2 US Department of Education. (2003, February 3). Fiscal year 2004 education budget summary and background information. Retrieved March 24, 2003, from www.ed.gov/about/overview/budget/budget04/summary/edlite-section2a.html#clcs

3 Ibid.

4 Weiss, H. B., & McCartney, K. (2003, March). Data in a democracy: The evolving role of evaluation in policy and program development. Paper presented at Child Development and Social Policy: Knowledge for Action, a Festschrift in honor of Edward Zigler, Washington, DC. Publication forthcoming.

5 Zigler, E., & Muenchow, S. (1992). Head Start: The inside story of America’s most successful educational experiment. New York: Basic Books.

6 Datta, L. (2001, January). Avoiding death by evaluation in studying pathways through middle childhood: The Abt evaluation of the Comer Approach. Paper presented at the MacArthur Invitational Conference on Mixed Methods Research, Santa Monica, CA.

7 Huang, D., Gibbons, B., Kim, K. S., Lee, C., & Baker, E. L. (2000, June). A decade of results: The impact of the LA’s BEST after school enrichment program on subsequent student achievement and performance. Los Angeles: University of California Los Angeles, Center for the Study of Evaluation.

8 See, for example, commentaries posted on the Afterschool Alliance website at: www.afterschoolalliance.org/voices_budget_cut.cfm.

9 Eccles, J., & Gootman, J. A. (Eds.). (2002). Community programs to promote youth development. Washington, DC: National Academies Press.

10 Lipsey, M. (1997, Winter). What can you build with thousands of bricks? Musings on the cumulation of knowledge in program evaluation. New Directions for Evaluation, 76, 7–23.

11 Riley, R. (2003, April 8). School children left behind. The Washington Post, p. A32.

12 Winerip, M. (2003, May 7). What a voucher study truly showed and why. New York Times, A27.

13 For a review of these and other lessons learned from evaluation, see Weiss, H. B., & McCartney, K. (2003, March). Data in a democracy: The evolving role of evaluation in policy and program development. Paper presented at Child Development and Social Policy: Knowledge for Action, a Festschrift in honor of Edward Zigler, Washington, DC. Publication forthcoming.

Acknowledgements

Preparation of this brief was made possible through the support of the Charles Stewart Mott Foundation. We would also like to thank Phil Sparks of the Communications Consortium Media Center for his review of this brief.

All rights reserved. Information on this site may not be reproduced in whole or in part without written permission from Harvard Family Research Project.

© 2006 President and Fellows of Harvard College
Published by Harvard Family Research Project
3 Garden Street
Cambridge, MA 02138
Tel: 617-495-9108
Fax: 617-495-8594
Email: hfrp@gse.harvard.edu