Issues and Opportunities in Out-of-School Time Evaluation Briefs

Evaluation of 21st Century Community Learning Center Programs: A Guide for State Education Agencies

Number 2, April 2002

Priscilla Little, Flora Traub, Karen Horsch, Harvard Family Research Project

Download this brief:

140KB Acrobat file

Help for Acrobat file

Download the addendum to this brief:

35KB Acrobat file

Help for Acrobat file

This brief is also available in hard copy for free. Please go to our publications list to learn how to order it.

Heading Index:

• Why Evaluate 21st Century Community Learning Center Programs?
• 21st CCLC Evaluation: Accountability and Program Evaluation
  • Performance Measurement for Accountability
  • Program Evaluation
  • State Program Evaluation
  • Models Available to Help SEAs with Program Evaluation
  • Local Program Evaluation
  • Technical Assistance
  • The Five-Tiered Approach
  • Continuous Improvement Management
  • Pulling It All Together
• 21st CCLC Data Requirements
  • Reduce Data Burden Through Alignment
• 21st CCLC Evaluation Infrastructure
  • SEA Oversight
  • Funding for Evaluation
• Conclusion
• Appendix A
• Appendix B
• Addendum
  
  

The Harvard Family Research Project’s (HFRP) Issues and Opportunities in Out-of-School Time Evaluation Briefs are short, user-friendly documents that highlight current research and evaluation work in the out-of-school time field. These documents draw on HFRP’s research work in out-of-school time in order to provide practitioners, funders, evaluators, and policymakers with information to help inform their work. This second brief, Evaluation of 21st Century Community Learning Center Programs: A Guide for State Education Agencies, offers an in-depth look at 21st Century Community Learning Center (21st CCLC) evaluation requirements and provides practical suggestions about how to implement state and local 21st CCLC evaluation.

Why Evaluate 21st Century Community Learning Center Programs?

For many states, after school programming is uncharted territory. Because of their newness, relatively little is known about after school best practices, program implementation, cost effectiveness, and impact. However, in these times of decreasing public resources and increasing and competing demands for public investments, it is necessary for funders, policymakers, and their constituents to know which investments are effective and how programs can be improved.

This situation makes it imperative that those developing policies and implementing after school programs are able to learn, over time, whether after school investments are working, how they can be improved, and whether they should be expanded. In other words, states need to begin to grapple with the issue of evaluation. Many funders, including the Department of Education (DOE), through the 21st Century Community Learning Center (21st CCLC) program, are mandating evaluation as part of their grant making.

As 21st Century Community Learning Center funds devolve to the states, those that evaluate their after school programs will be in the forefront, able to capitalize on evaluation results to create sustainable after school programs that best serve the children and youth of their state in a cost-effective manner. Results from after school evaluations nationwide indicate that after school programs do make a difference—they contribute to increased student performance, provide a safe haven for children and youth during non-school hours, and reduce school violence.1 The depth and breadth of the 21st CCLC grants offer a unique opportunity to collect data nationwide and build a strong, data-driven case to support after school programs throughout the country. It is essential, therefore, that state education agencies (SEAs) administering 21st CCLC grants develop a manageable plan for designing and implementing evaluation.

The core components of the 21st Century Community Learning Centers evaluation are mandated through existing federal legislation.2 This guide, geared for state education agencies who are charged with implementing 21st CCLC programs, examines the 21st CCLC legislation and provides SEAs with information about what it means for them. It addresses three main issues relevant to designing and conducting evaluation of 21st CCLC programs:

1. 21st CCLC evaluation—balancing demands for accountability with the need to evaluate for continuous improvement and program effectiveness
2. 21st CCLC data requirements—understanding and aligning data reporting
3. 21st CCLC evaluation infrastructure—SEA oversight and funding

This guide concludes with a “getting started” checklist to help state education agencies begin to make important evaluation decisions. A glossary of common after school evaluation terms is provided at the end of the guide.

^ Back to Top

21st CCLC Evaluation: Accountability and Program Evaluation

According to Federal 21st CCLC evaluation guidelines, SEAs must ensure that programs:

• meet the principles of effectiveness based on the assessment of objective data, an established set of performance indicators, and scientifically-based research on helping students meet a state’s high academic achievement standards;
• use performance indicators and performance measures for evaluation;
• conduct a periodic evaluation of how the program or activity is providing high quality academic enrichment;
• use evaluation findings for continuous improvement of the program, broader dissemination of promising practices, and for the general information of the public;
• receive ongoing technical assistance and training that enables them to implement effective program and evaluation strategies.3

Additionally, SEAs must conduct state-level evaluation to determine the effectiveness of the 21st CCLC program throughout the state. As with local grantee program evaluation, statewide evaluation must use performance indicators and measures for evaluation.

These guidelines indicate that the goals of the 21st CCLC evaluation, and the consequent responsibilities of the SEAs are twofold. First, SEAs must conduct performance measurement for accountability. Second, they must ensure that 21st CCLC programs are effective and continuously improving through program evaluation carried out at both the grantee and state levels.4 Both of these evaluation approaches, and their implications for SEAs, are described below.

^ Back to Top

Performance Measurement for Accountability
Performance measurement (also called performance monitoring) is defined as “the ongoing monitoring and reporting of program accomplishments, particularly progress toward pre-established goals” (sometimes also called outcomes).5 Performance measurement is typically used as a tool for accountability and the 21st CCLC program is no exception.

As in former 21st CCLC evaluation guidelines, local sites are required to submit Annual Performance Reports (APRs) that provide SEAs with center-level data collected on most of the 21st CCLC performance indicators listed in the box below. While only discretionary grantees (i.e., grantees that are receiving 21st CCLC funding under the “old” legislation6) are still required to report on all the indicators, all grantee local sites, new and old, are encouraged to collect data on all performance indicators.

The performance measurement data from APRs submitted by local sites will be used for accountability at the state level. As SEAs begin to collect performance measurement data on 21st CCLC participants, they can assess and report to the public on the well-being of children in 21st CCLC after school programs. Further, they can strengthen the argument that public investments in after school contribute to that well-being. However, it is important to stress that performance measurement does not provide conclusive evidence that the program has caused changes in performance. To determine this, 21st CCLC evaluation guidelines also require program evaluation.

^ Back to Top

Program Evaluation
Program evaluation is complementary to performance measurement. While performance measurement provides a broad, shallow snapshot of program functioning, program evaluation provides a narrower, deeper examination of program functioning. Because of the greater rigor of inquiry and depth of knowledge that program evaluation provides, it, rather than performance measurement, is called for when policymakers need “evidence” that a program works or validated information about the way in which programs are operating. Table 1 offers a comparison of performance measurement and program evaluation.

As in current 21st CCLC evaluation guidelines, there are two levels of program evaluation required in the new authorizing legislation: comprehensive state level program evaluation and local grantee level periodic evaluation. These two levels of program evaluation will serve to assess the effectiveness of the 21st CCLC program in achieving its goals and provide information necessary for continuous improvement at both the local and statewide program levels.

^ Back to Top

State Program Evaluation
According to non-regulatory 21st CCLC Guidance, states must “conduct a comprehensive evaluation (directly or through a grant or contract) of the effectiveness of programs and activities provided with 21st CCLC funds.”7 This evaluation will allow SEAs to learn about the implementation of the 21st CCLC program across the state, the cumulative impact of the program on all the state’s children, and circumstances that are fostering and hindering success in the achievement of statewide goals for the program. Unlike the performance measurement that the SEAs will be overseeing, state-level program evaluation does not necessitate the involvement of all 21st CCLC grantees—a sampling of a select group of grantees may be more appropriate to the evaluation goal.

To meet program evaluation requirements, SEAs must acquire and utilize “scientifically-based” information about the 21st CCLC program statewide. They must also ensure that local sites use scientifically-based information to meet principles of effectiveness. Title IX of the reauthorized Elementary and Secondary Education Act (ESEA) defines scientifically-based research as having the following components:

1. Systematic, empirical methods that use both experiments and observation
2. Methods that provide reliable data across evaluators, measurements, and different studies
3. Rigorous data analyses that are adequate to test hypotheses
4. Experimental and quasi-experimental designs, or at a minimum, designs that allow for comparisons among groups
5. Sufficient detail of evaluation design to allow for replication across the state
6. Scientific review and approval by independent experts in the field

In essence, these guidelines strongly urge SEAs to conduct replicable program evaluation studies that allow for rigorous statistical analyses and generalizable conclusions. This may require the allotment of funds for external evaluation consultants. Table 2 provides a comparison of the main features and benefits of using different evaluation designs. While the legislation does not require the use of experimentally designed evaluation, it is clear from Table 2 that experimental studies are the most desirable when trying to make generalizable statements about effectiveness.

^ Back to Top

Models Available to Help SEAs with Program Evaluation
When considering a program evaluation that will meet their own states’ needs, SEAs can turn to the national 21st CCLC program evaluation model. As described in greater detail in the box below entitled, “Current 21st CCLC Evaluation: A Blend of Approaches,” the 21st CCLC national evaluation is a large-scale effort, conducted by an independent evaluator working with the DOE to determine the effectiveness of the 21st CCLC program in raising student achievement and fostering positive youth development (including safety, self-care, and behavior).

States that are already evaluating their after school programs, independently of the 21st CCLC program, can also serve as models to others newly charged with evaluation under the 21st CCLC program in structuring and legislating evaluation of after school programs. The excerpts in the box below, from actual state legislation, may help states craft proposals that meet their needs.

^ Back to Top

Local Program Evaluation
In addition to conducting state-level program evaluation, SEAs must help local sites conduct some form of evaluation that will contribute to their own program improvement and/or demonstrate that their site is effective in producing positive academic outcomes for participants. This necessitates that 21st CCLC grantees conduct program evaluation to answer questions about their programs and their effectiveness. Program evaluation at the local level is subject to the same standards of scientifically-based research as evaluation at the state level. The legislation gives some leeway in the types of evaluation questions that grantees can explore. Below is a list of questions commonly addressed in after school program evaluation:8

• Is my after school program responding to participant needs?
• Who is participating in my after school program?
• What is the staffing for my program? What training do they have/need?
• What after school resources are available in my community? What are the service gaps?
• What services does my after school program provide?
• How can I improve my after school services? Are participants, families, and the community satisfied?
• What is the impact of my after school program? Academic achievement (e.g., increased school attendance and homework completion)? Youth Development (e.g., increased positive school behavior)? Workforce Development (e.g., employment preparation and supporting families transitioning from welfare to work)? Prevention (e.g., reduced drug and alcohol abuse)?
• What are the costs of my after school program?

^ Back to Top

Technical Assistance
Providing technical assistance about evaluation to local grantees is a key responsibility of SEAs under the 21st CCLC guidelines. One element of this technical assistance will be to support programs in their performance measurement and their regular reporting of performance indicators through the APR by helping them handle the collection and reporting of large quantities of data. Since the 21st CCLC authorizing legislation does not prescribe a specific way in which programs must conduct program evaluation, there is more uncertainty as to the type of program evaluation technical assistance that programs will require. Some grantees may carry out program evaluation themselves, while others may require an outside evaluator to answer the types of evaluation questions that interest them. Consequently, programs will need technical assistance both in the area of conducting program evaluation and in the area of knowing when, and if, to hire an outside evaluator.

There are dozens of approaches to evaluation. Descriptions of two approaches that are well-suited to after school program evaluation are presented below: the five-tiered approach to evaluation and continuous improvement management.

^ Back to Top

The Five-Tiered Approach
As its name suggests, the five-tiered approach9 to program evaluation describes the evaluation process as a series of five steps that organizations can take to evaluate their program. The step on which a program starts an evaluation is largely determined by its developmental stage. For example, a new program may want to begin by assessing its needs, while an older, more established program might focus on refining its services and disseminating program results. The five steps are:

1. Conduct a needs assessment. Identify the after school services already available in the community as well as services needed, but not provided.
2. Document your service. Monitor the program by systematically documenting the after school services it provides; establish indicators of success and a set of benchmarks to record progress.
3. Compare program intent to actuality. Look at original goals and who the program is serving and gauge whether the program is meeting benchmarks.
4. Fine-tune program services and evaluation methods. Discuss evaluation data with staff and brainstorm about program improvement; refine indicators and benchmarks; begin to document program outcomes.
5. Produce evidence of effectiveness. Determine research design and evaluation measures; demonstrate your program’s worth to participants, the community, and funders.

The five-tiered approach underscores some important points about evaluation. First, all programs are able to do at least some evaluation—tier 1, conducting a needs assessment, is something that every program can do. Second, a program’s information needs evolve as it develops and evaluation must be nimble enough to accommodate program change. Finally, evaluation is an ongoing, cyclical process—feedback from one phase of the evaluation can be used to shape the next phase.

^ Back to Top

Continuous Improvement Management
A second, and not necessarily mutually exclusive approach to evaluation, is evaluation for continuous improvement. Simply stated, evaluation for continuous improvement answers the question, “What can I do even better?” While the demonstration of program effectiveness (tiers 4 and 5 above) may require the help of an outside evaluator, evaluation for continuous improvement can be conducted independently by programs. To help with this process, the U.S. Department of Education has prepared A Guide to Continuous Improvement Management (CIM) for 21st Century Community Learning Centers.10 This guide is a complete resource on how to conduct an evaluation that will help 21st CCLC sites strengthen program design, monitor and reflect on program implementation, and assess and communicate program results. It outlines a three-step process for continuous improvement management, and includes tools and resources that can be replicated and used in multiple 21st CCLC sites.

^ Back to Top

Pulling It All Together
The task of administering and conducting both performance measurement and program evaluation may seem daunting at first, but there is already a good evaluation model in place to help SEAs. The current national 21st CCLC evaluation shows how performance measurement and program evaluation can be conducted simultaneously to meet the dual purposes of providing accountability and assessing program impact. The box below describes this evaluation and illustrates the blended approach to evaluation that SEAs can adapt for use in their states.

^ Back to Top

21st CCLC Data Requirements

Data collection is an important part of any evaluation. To assist SEAs in the collection of performance measurement data, the U.S. Department of Education has already developed an Annual Performance Report (APR) that is required of all 21st CCLC sites under current legislation. The APR is a set of forms that is used to (1) demonstrate substantial progress has been made toward meeting the objectives of the project as outlined in the grant application and (2) collect data that addresses the performance indicators for the 21st CCLC program. Annual submission of the APR is a requirement of 21st CCLC grants. 21st CCLC sites are required to submit annual reports that coincide with the school year and include the summer prior to the school year. For example, the APRs due in April 2002 cover the reporting period beginning in June 2001 and continuing through most of the 2001–2002 school year. All 21st CCLC grantees are now expected to complete the web-based version of the APR, available online at www.ed.gov/21stcclc/index.html.

The APR requires the collection of two types of data: descriptive data at the grant and center levels, and achievement data on participants who attend the program on a regular basis. Additionally, the APR requires centers to administer teacher surveys and includes optional participant surveys for all regular participants in fourth grade or above. Table 3 lists the specific types of information collected using the APR. Although the new legislation requires that SEAs collect data on fewer performance indicators, they are encouraged to collect all APR data to get a comprehensive assessment of program quality.

In addition to the APR data, local sites may want to consider administering surveys to 21st CCLC staff and parents. While teacher surveys provide an important perspective on changes in school-day behavior and academic performance, it is also important to tap into the knowledge that 21st CCLC program staff have about youth experiences in their after school programs and how these experiences have helped improve attitudes toward school. Similarly, parent satisfaction is an often overlooked measure of program quality. Collecting data on how parents perceive their child’s after school experiences can be valuable information to feed back into programs interested in continuous improvement.

On a local level, some 21st CCLC sites currently use commercial software packages such as Microsoft Access or Excel to assist in the collection and management of APR data. A few sites work with data management consultants.11 The benefit of using these types of management information systems is that they help consolidate large quantities of multi-site data. A disadvantage is that you need to expend resources to train staff to use the systems properly. Some states have already invested in statewide management information systems to help programs collect and report information in a cost effective way. As SEAs contemplate management information systems for data reporting, they can investigate what technology is already available in the state.

^ Back to Top

Reduce Data Burden Through Alignment
Data burden is a reality for most after school programs. 21st CCLC programs typically receive funding from many different sources, which have different data requirements and reporting cycles. Data requirements that are aligned across funding sources go a long way toward both reducing data burden and increasing the likelihood of more accurate data. Using already existing data can also ease the burden of data reporting.

Align Data Requirements with State Accountability System. SEAs can play an important role in ensuring that data collection is not overly burdensome to after school programs. One way to do this is to align 21st CCLC data reporting with reporting already established as part of the state’s larger accountability system. As Table 3 shows, the APR requires standardized test scores for those sites that are already required to administer achievement tests as part of their district or state accountability system. This means that sites are only collecting one set of achievement data for multiple purposes, and students are not subjected to multiple tests for different reporting requirements.

Align Data Requirements with Other State After School Programs. In addition to alignment with state accountability systems, 21st CCLC sites should consider the data being collected by other after school initiatives in the state. The box below shows reporting requirements for after school programs already underway in California and Kentucky. Like the 21st CCLC data reporting requirements, both states require the collection of student performance data. In addition, like the 21st CCLC APR, the California after school program also collects information on student behavior and attendance using the 21st CCLC student survey. As 21st CCLC grant money is awarded to states, it is helpful for SEAs to understand what state after school programs already exist, and how they can work to align both service delivery and effective data management.

Coordinate Data Requirements with Title I Reporting Requirements. Another way for SEAs to align data reporting is to coordinate performance indicator data requirements with Title I requirements. Not only is Title I the largest single federal investment in education, but over 47,000 schools nationwide use Title I funds to provide extra academic support and learning opportunities, e.g., after school programs, to help low-income students meet their state’s academic program standards.12 Recent changes to Title I place an increased emphasis on assessment and accountability, and require states to implement annual reading and math assessments for children in grades 3–8 in schools receiving federal funds. Further, state education officials must produce an annual report card assessing the state’s progress compared to that of other states. Under ESEA legislation, each Title I program must coordinate and integrate federal, state, and local services and programs, including 21st CCLC. Part of this coordination ought to be around performance indicators and the collection of data.

Complete coordination among all service sectors providing after school programming in a state is probably unrealistic. However, SEAs administering 21st CCLC grants can show leadership by aligning data reporting with other state and local agencies to help reduce data burden for after school programs.

^ Back to Top

21st CCLC Evaluation Infrastructure

Evaluation infrastructure includes all the systems, individuals, and resources that go into carrying out an evaluation and disseminating the results. In the case of 21st CCLC programs, state education agencies will provide the infrastructure necessary to implement evaluation at the state and local levels.

^ Back to Top

SEA Oversight
It is important to clarify the roles of the SEA as well as its relationship with other state agencies prior to implementing the evaluation. As the SEA oversees the performance measurement component of the evaluation, it makes process decisions. For example, it decides how often to report data to other entities, coordinates inter-agency cooperation necessary for success, and designates the agency responsible for providing technical assistance to programs should they require it.

When the SEA oversees program evaluation, its task is dual: to conduct a statewide evaluation and to oversee local program evaluations. The statewide evaluation requires SEAs to first articulate the evaluation purpose in an RFP (request for proposal). If necessary, it must then locate and select an evaluator. Once selected, the SEA works with the evaluator to assist with access to and standardization of key data, serves as a liaison between programs, policymakers, and the evaluator, and ensures evaluation technical assistance is provided to programs during the evaluation if necessary.

In providing oversight to local grantees as they conduct program evaluation, the SEA needs to support local programs though the provision of technical assistance. Technical assistance may be needed to help programs articulate evaluation goals, select an evaluator as needed, and/or conduct the evaluation themselves, ensuring that it meets the standard of scientifically-based research as defined in the ESEA.

^ Back to Top

Funding for Evaluation
An integral component of evaluation infrastructure is funding. SEAs must set aside adequate resources for the evaluation, including funding the development of data collection instruments and other research tools as needed. The 21st Century Community Learning Center legislation states that a state educational agency may use not more than 3% of its grant for any and all of the following activities: monitoring and evaluation of after school programs; comprehensive evaluation to determine effectiveness; and capacity building, training, and technical assistance to eligible grantees, including technical assistance for evaluation.13

Since performance measurement is more strictly defined by 21st CCLC guidelines, the cost of program evaluation is more variable. This means that SEAs need to think through their evaluation needs carefully, understanding that the issue of program evaluation cost is intricately related to the goals of the evaluation. Different types of evaluation questions cost different amounts to answer. Generally speaking, program evaluations that are conducted over a longer period of time, that have more rigorous designs (i.e., experimental outcomes studies), and that involve more program sites are more expensive than those that do not meet these criteria. Using pre-existing data sets from state and national organizations can greatly reduce data collection costs, freeing up scarce resources for other evaluation purposes.

^ Back to Top

Conclusion

Evaluation of the 21st Century Community Learning Centers program is critical to the success of after school programming nationwide. As state education agencies assume the responsibility for administering 21st CCLC grants, they must implement both performance measurement for accountability and program evaluation for continuous improvement and assessing program effectiveness. To do so effectively, they must align their data requirements with other programs serving school-age children, such as Title 1 and other state after school initiatives, as well as with state accountability systems.

The state examples inserted throughout this document serve to underscore an important point about evaluation and accountability at the state level—states are entering the realm of 21st CCLC administration with a range of evaluation experiences that they can draw on as they craft their 21st CCLC programs. Taking stock of what specific states already have in place, who is already working in after school programming, and what kind of data is already being collected will make the task of SEAs just starting out that much easier. In other words, to get the most out of your after school evaluation, take a “don’t reinvent the wheel” approach.

The changing context of the 21st Century Community Learning Centers program means that state legislatures will play an increasingly important role in program design and sustainability of new after school programs. This offers state education agencies the unique opportunity to capitalize on their strengths and draw from past experience to shape the outcomes for school-age children and youth.14 States that take evaluation seriously will be able to place after school programming at the top of the political and public agendas, while ensuring positive experiences and outcomes for the children and youth of their state.

Acknowledgements

Preparation of this brief was made possible through the support of the Charles Stewart Mott Foundation. We also want to acknowledge An-Me Chung of the C.S. Mott Foundation and Adriana de Kanter, Partnership Liason, OSES/AIDP, U.S. DOE, for their valuable feedback on earlier drafts.

Notes

1 U.S. Department of Education. (1998). Safe and smart: Making the after-school hours work for kids. Washington, DC: Author. Available on the Internet at www.ed.gov/pubs/SafeandSmart.

2 Unless otherwise noted, current federal 21st CCLC evaluation guidelines will remain the same when the program becomes block granted to the states.

3 Excerpt from 21st CCLC program manual, in draft. Provided by U.S. Department of Education staff, January 2002.

4 Title IV, Part B of the 2002 Elementary and Secondary Education Act also authorizes the Secretary to reserve up to one percent for, among other things, national evaluation activities. This discussion is beyond the scope of this guide.

5 U.S. Government Accounting Office, April 1998.

6 Prior to the changes made to the program in the No Child Left Behind Act of 2001.

7 Department of Education. (February 2002). 21st CCLC guidance. Washington, DC: Author.

8 Based on a review of 21 after school program evaluations in the Harvard Family Research Project Out-of-School Time Evaluation Database. Available on the Internet at: www.gse.harvard.edu/hfrp/projects/afterschool/evaldatabase.html.

9 Jacobs, F. (1988). The five-tiered approach to evaluation: Context and implementation. In H. B. Weiss & F. H. Jacobs (Eds.), Evaluating family programs (pp. 37–68). Hawthorne, NY: Aldine de Gruyter.

10 Pane, N., Mulligan, I., Ginsburg, A., & Lauland, A. (1999). A guide to continuous improvement management (CIM) for 21st Century Community Learning Centers. Washington, DC: U.S. Department of Education. Available on the Internet at www.ed. gov/offices/OUS/PES/21cent/cim226.pdf.

11 Information on 21st CCLC software use is based on Harvard Family Research Project monitoring of the after school listserv hosted by the Charles Stewart Mott Foundation.

12 The Finance Project. (2002). Using Title I to support out-of-school time and community school initiatives. New York: Author. Available on the Internet at www.financeproject.org/Publications/Brief10.pdf.

13 H.R. 1. Part B—21st Century Community Learning Centers. Sec. 4202, b(3). January 2002.

14 Langford, B. (2001). State legislative investments in school-age children and youth. New York: The Finance Project. Available on the Internet at www.financeproject.org/publications/oststatepaper.pdf.

^ Back to Top

Appendix A

Glossary of Selected Evaluation Terms

Accountability
A public or private agency, such as a state education agency, that enters into a contractual agreement to perform a service, such as administer 21st CCLC programs, will be held answerable for performing according to agreed-on terms, within a specified time period, and with a stipulated use of resources and performance standards.

Benchmark
(1) An intermediate target to measure progress in a given period using a certain indicator. (2) A reference point or standard against which to compare performance or achievements.

Data Collection Methods
Document Review: A review and analysis of existing program records and other information collected by the program. Information analyzed in a document review is not gathered for the purpose of the evaluation. Sources of information for document review include information on staff, budgets, rules and regulations, activities, schedules, attendance, meetings, recruitment, and annual reports.

Interviews/Focus Groups: Conducted with evaluation and program/initiative stakeholders, including: staff, administrators, participants and their parents or families, funders, and community members. Can be conducted in person or over the phone. Questions posed are generally open-ended. The purpose of interviews and focus groups is to gather detailed descriptions, from a purposeful sample of stakeholders, of the program processes and the stakeholders’ opinions of those processes.

Observation: An unobtrusive method for gathering information about how the program/initiative operates. Observations can be highly structured, with protocols for recording specific behaviors at specific times, or unstructured, taking a more casual “look-and-see” approach to understanding the day-to-day operation of the program. Data from observations are used to supplement interviews and surveys in order to complete the description of the program/initiative and to verify information gathered through other methods.

Secondary Source/Data Review: Sources include data collected for other similar studies for comparison, large data sets such as the Longitudinal Study of American Youth, achievement data, court records, standardized test scores, and demographic data and trends. Data are not gathered with the purposes of the evaluation in mind; they are pre-existing data that inform the evaluation.

Surveys/Questionnaires: Conducted with evaluation and program/initiative stakeholders. Usually uses a highly structured interview process in which respondents are asked to choose answers from those predetermined on the survey and administered on paper, through the mail, or more recently, through email and on the Web. The purpose of surveys/questionnaires is to gather specific information from a large, representative sample.

Tests/Assessments: Data sources include standardized test scores, psychometric tests, and other assessments of the program and its participants. These data are collected with the purposes of the evaluation in mind.

Evaluation Design
Experimental Design: Experimental designs all share one distinctive element—random assignment to treatment and control groups. Experimental design is the strongest design choice when interested in establishing a cause-effect relationship. Experimental designs for evaluation prioritize the impartiality, accuracy, objectivity, and validity of the information generated. These studies look to make causal and generalizable statements about a population or impact on a population by a program or initiative.

Non-Experimental Design: Non-experimental studies use purposeful sampling techniques to get “information rich” cases. Types include: case studies, data collection and reporting for accountability, participatory approaches, theory based/grounded theory approaches, ethnographic approaches, and mixed method studies.

Quasi-Experimental Design: Most quasi-experimental designs are similar to experimental designs except that the subjects are not randomly assigned to either the experimental or the control group, or the researcher cannot control which group will get the treatment. Like the experimental designs, quasi-experimental designs for evaluation prioritize the impartiality, accuracy, objectivity, and validity of the information generated. These studies look to make causal and generalizeable statements about a population or impact on a population by a program or initiative. Types include: comparison group pre-test/post-test design, time series and multiple time series designs, non-equivalent control group, and counterbalanced designs.

Formative/Process Evaluation
Formative evaluations are conducted during program implementation in order to provide information that will strengthen or improve the program being studied—in this case, the after school program or initiative. Formative evaluation findings typically point to aspects of program implementation that can be improved for better results, like how services are provided, how staff are trained, or how leadership and staff decisions are made.

Indicator
An indicator provides evidence that a certain condition exists or certain results have or have not been achieved. Indicators enable decision-makers to assess progress towards the achievement of intended outputs, outcomes, goals, and objectives.

Performance Measurement (also called Performance Monitoring)
According to the U.S. Government Accounting Office, it is “the ongoing monitoring and reporting of program accomplishments, particularly progress toward pre-established goals” (sometimes also called outcomes). Performance measurement is typically used as a tool for accountability. Data for performance measurement is often tied to state indicators and is part of a larger statewide accountability system.

Summative/Outcome Evaluation
Summative evaluations are conducted either during or at the end of a program’s implementation. They determine whether a program’s intended outcomes have been achieved. Summative evaluation findings typically judge the overall effectiveness or “worth” of a program based on its success in achieving its outcomes, and are particularly important in determining whether a program should be continued.

^ Back to Top

Appendix B

Getting Started: Issues to Consider When Designing an Evaluation of Your State’s 21st CCLC Program

The following is a checklist of issues to consider as you begin to craft an evaluation of your state 21st Century Community Learning Centers program. (Asterisks indicate requirements of the Elementary and Secondary Education Act guidelines.)

^ Back to Top

Addendum

The Department of Education (DOE) is currently reviewing its 21st Century Community Learning Center reporting requirements and will most likely reduce the number of required indicators to one—indicator 1.1 which relates to academic achievement. While states are encouraged to report information on the full set of indicators, it is unlikely that they will be required to report on the subset suggested in the Guide. Please check back here for the DOE's final decision on indicator reporting requirements or you can register to be notified of all changes to our Out-of-School Time website.

All rights reserved. Information on this site may not be reproduced in whole or in part without written permission from Harvard Family Research Project.

© 2006 President and Fellows of Harvard College
Published by Harvard Family Research Project
3 Garden Street
Cambridge, MA 02138
Tel: 617-495-9108
Fax: 617-495-8594
Email: hfrp@gse.harvard.edu