Out-of-School Time Evaluation Snapshot
A Review of Out-of-School Time Program Quasi-Experimental and Experimental
Evaluation Results
Number 1, July 2003
Priscilla M. D. Little
and Erin Harris
Download this brief:
This Snapshot is also available in hard copy for free. Please go
to our publications list
to learn how to order it.
Heading Index:
Harvard Family Research Project’s series of Out-of-School Time
Evaluation Snapshots distills the wealth of information compiled in our Out-of-School
Time Program Evaluation Database1
into a single report. Each Snapshot examines a specific aspect of out-of-school
time (OST) evaluation. This Snapshot provides an overview of what the quasi-experimental
and experimental evaluations in the database reveal about the impact of out-of-school
time programs on an array of academic, prevention, and youth development outcomes.
It also includes a resource list of other OST evaluation reviews and related
evaluation information.2
The past five years have witnessed overwhelming public support for the funding
of out-of-school time (OST) programs. This support has been fueled by public
concern that young people need safe places in the out-of-school hours—places
that provide supervision by caring adults and productive activities that support
school success as well as broader development. Given the amount of resources
now being allocated to OST programming, stakeholders have a growing interest
in knowing if these programs are attracting those most in need of services and
if youth are acquiring the intended benefits of program participation, such
as improved school performance, lower risk taking, and positive youth development.
This interest, coupled with increasing pressure from policymakers for programs
to demonstrate research-based results, has sparked interest in OST evaluations
that have employed rigorous research designs to examine program outcomes.
| |
Definitions of Research Designs of Reviewed Evaluations
Experimental Design (also known as
Randomized Control Trial) – Random assignment of individuals to either
a treatment or control group. A comparison between groups is made to determine
program effects.
Quasi-Experimental Design – Nonrandom
selection of individuals to treatment and comparison groups or conditions,
as well as the use of controls to minimize threats to the validity of
conclusions drawn. Subject to selection bias. Types of quasi-experimental
designs include: comparison group pretest/posttest design, time-series
and multiple time-series designs, matched pairs, nonequivalent control
group, and counterbalanced designs.
|
Our Sample and Methodology for Review
This brief provides an overview of the 27 evaluations in the HFRP Out-of-School
Time Program Evaluation Database that used experimental and quasi-experimental
research designs to make statements about program outcomes.3
(See the box for research design definitions.) As of June 2003, evaluations
of 54 OST programs were profiled in our database. Of these 54 programs, 27 used
quasi-experimental designs and 11 used experimental designs during some phase
of their evaluation; 3 programs used both. Many also included a non-experimental
design component as well. The set of evaluations reviewed for this Snapshot
represents a range of programs from small single-site programs to city- and
statewide programs operating multiple sites in multiple locations to national
programs such as the 21st Century Community Learning Centers program. Appendix
A provides a brief description of all programs reviewed, as well as citations
for the evaluation reports. In most cases, the programs also collected implementation
data that was used to help shape program execution and inform future evaluation
efforts.
The results presented in this brief represent statistically significant outcomes
(p<.10) found to be linked to overall program participation as reported in
the set of 27 evaluations reviewed.4
However, there are some statements of caution about interpreting the results
of OST impact evaluations. First, due in part to the diversity of OST programming,
most OST evaluations assess overall program impact, answering the question,
Did the combined results of the various program components result in
changes in participant outcomes? Few evaluation studies have attempted to link
specific program activities with outcomes—and this set of 27 evaluations
is no different. This means making statements about causal links between
specific OST activities and specific outcomes is not possible from this set
of evaluations.
Second, this summary is based on a review of evaluation reports and the results
presented herein reflect what evaluators and program leaders chose to print
in those reports and make available to the public. In many cases, the findings
in this set of evaluation reports are predominantly positive and sometimes neutral,
but seldom negative. A review of the complete evaluation reports, including
the formative findings, reveals that programs identify many areas for self-improvement.
This information, however, does not lend itself to quantification using statistical
analysis and therefore is beyond the scope of this review.
Also beyond the scope of this review are the rich implementation findings included
in each evaluation report, which are essential for interpreting the program
results in the context of the program.
Finally, it is important to note that statistically significant results often
held only for a subgroup (e.g., middle school versus elementary school students,
girls versus boys, etc.). In addition, significant findings often varied by
situation. For example, positive findings for participants may only apply to
one particular school year or the results of one particular test, but other
school years and tests might have different results. The results presented below
are a synthesis of results across subgroups. For a complete reporting of the
statistically significant results by program evaluation, including information
about subgroups, see Appendix B.
^ Back to Top
Results From Programs That Assess Academic Outcomes
Evaluators use a broad array of performance measures to assess academic outcomes.5
They range from grades to standardized testing to homework completion. Results
from the 25 evaluations that assessed academic outcomes in this sample indicate
that participation in OST programs is linked to:
| |
In Focus: A Quasi-Experimental Evaluation of Academic
Outcomes
The Foundations organization operates extended-day
programs in large urban areas in the Mid-Atlantic and Northeast, serving
1,200 children in 41 different sites. The programs feature a curriculum
emphasizing academic subjects as well as experiences designed to foster
physical and emotional development. Participants also spend time on field
trips, homework assistance, and in the computer lab, and family involvement
is encouraged. Results from its two-year quasi-experimental evaluation
of fourth grade participants indicate that participation in a Foundations
program improves school grades and performance as measured by the Terra
Nova Reading/Language Arts and Mathematics Computation Tests.
For the full profile of this evaluation see the Harvard
Family Research Project Out-of-School Time Program Evaluation Database
at www.gse.harvard.edu/hfrp/
projects/afterschool/evaldatabase.html.
|
- Better attitudes toward school and higher educational aspirations
- Better performance in school, as measured by achievement test scores and
grades
- Higher school attendance (as measured by attendance and tardiness)
- Less disciplinary action (e.g., suspension)
Specifically, this set of OST evaluations found statistically significant improvements
in the following areas related to academic achievement:
- Academic involvement
- Achievement motivation
- Achievement test scores
- Attitude toward school or academics
- College attendance
- Competence
- Educational aspirations
- Expulsions
- Grades
- Homework completion
- Lower rates of course failure
- Overall academic performance6
- Reduced suspensions
- School attendance (includes dropout and tardy rates)
While there were some subsamples for which the results were neutral, this set
of evaluations did not report negative academic achievement results associated
with overall participation.
^ Back to Top
| |
In Focus: An Experimental Evaluation of Prevention Outcomes
The Children’s Aid Society Carrera-Model Teen Pregnancy
Prevention Program (CASCM) was launched in Harlem in 1984 and there
are currently 21 replication sites nationwide. Twenty-nine other sites
maintain program variations. Five main activity components and two main
service components constitute CASCM. The five activity components are:
(1) a work-related intervention called job club that includes stipends,
development of an individual bank account, graduated employment experiences,
and career awareness; (2) an educational component that includes individual
academic assessment, tutoring, homework help, PSAT and SAT preparation,
and assistance with college entrance; (3) family life and sex education;
(4) self expression through the arts; and (5) lifetime individual sports.
Its experimental evaluation of 12 sites used random assignment
to develop treatment groups (589 adolescents) and control groups (474
adolescents). Annual surveys were collected from both groups as well as
annual pretests and posttests of knowledge related to sexuality topics.
Results from these surveys indicate that program participation among boys
resulted in significantly higher levels of knowledge about sexuality and
reproductive outcomes. At the third-year follow-up program girls had significantly
lower pregnancy rates and births than did control group girls.
For the full profile of this evaluation see the Harvard
Family Research Project Out-of-School Time Program Evaluation Database
at www.gse.harvard.edu/hfrp/
projects/afterschool/evaldatabase.html.
|
Results From Programs That Assess Prevention Outcomes
Outcomes that fall into the prevention category include changes in sexual behavior,
feelings of personal safety, changes in drug and alcohol use and abuse, and
overall improvements in physical health. While fewer out-of-school time programs—and
therefore fewer evaluations—focus on prevention, results from evaluations
of programs that do articulate prevention as a program component indicate that
OST programs can have a positive prevention impact. Specifically, results from
the 12 evaluations that assessed prevention outcomes in this sample indicate
that participation in OST programs is linked to:
- Avoidance of drug and alcohol use
- Decreases in delinquency and violent behaviors
- Increased knowledge of safe sex and avoidance of sexual activity and pregnancy
- Increased skills for coping with peer pressure
This set of OST evaluations found improvements in the following areas of prevention:
- Avoidance of delinquency (including criminal arrest)
- Avoidance of drug and alcohol use (including cigarette smoking)
- Avoidance of sexual activity
- Avoidance of violence
- Knowledge about drug and alcohol use (including perceived social benefits)
- Knowledge of sexuality issues (including attitudes toward sex)
- Reduced pregnancy rates
- Use of safe sex practices
Looking beyond these positive results, one program evaluation, the 21st Century
Community Learning Centers evaluation, reported a negative relationship between
overall program participation and drug abuse. It is not clear how to interpret
this result, but it suggests that the relationship between program participation
and prevention is complicated and needs further analysis to determine the specific
impacts that OST programs can have in the category of prevention.
^ Back to Top
Results From Programs That Assess Youth Development
Outcomes
Many OST evaluations assess youth development outcomes, which are broadly defined
as those outcomes that assess the social and emotional development of program
participants. Outcomes that fall into this category range from standardized
measures of self-esteem, participant behavior, and interpersonal skills to decision
making, goal setting, leadership, and career development. Results from the set
of 15 evaluations that have assessed positive youth development results indicate
that OST program participation is linked to:
| |
In Focus: A Quasi-Experimental Evaluation of Youth Development
Outcomes
The San Francisco Beacons Initiative aims to help
youth develop competencies that will help them become responsible adults
through out-of-school time programs that focus on five areas: leadership,
career development, arts and recreation, health, and education. This citywide
program operates Beacons Centers at eight sites and in 1999–2000
served 7,500 youth and adults.
Its quasi-experimental evaluation included all youth (Beacon
Center participants and nonparticipants) in the sixth and seventh grades
at each of the three middle schools hosting Beacon Centers. Survey data
from middle school students indicate that youth participating at the Beacons
Centers reported significantly greater opportunities to assume a range
of formal, informal, and representation-type leadership roles than did
nonparticipant youth. Further, middle school participants reported spending
approximately two and a half hours more per week in productive leisure
activities—art, music, dance, drama, and tutoring—than youth
who attended the schools but not the Centers.
For the full profile of this evaluation see the Harvard
Family Research Project Out-of-School Time Program Evaluation Database
at www.gse.harvard.edu/hfrp/
projects/afterschool/evaldatabase.html. This profile will be updated
soon. To be notified when it is available sign up for our OST website
change notification email at www.gse.harvard.edu/hfrp/
subscribe.html.
|
- Decreased behavioral problems
- Improved social and communication skills and/or relationships with others
(peers, parents, and/or teachers)
- Increased community involvement and broadened world view
- Increased self-confidence and self-esteem
Specifically, this set of OST evaluations found improvements in the following
positive youth development areas:
- Communication skills
- Community involvement
- Computer skills
- Confidence/self-esteem
- Conflict resolution
- Decision making
- Decreased aggression
- Desire to help others
- Exposure to new experiences
- General well-being
- Goal setting
- Interactions/relationships with adults
- Interactions/relationships with peers
- Job experience/skills
- Leadership skills
- Maturity
- Money management skills
- Opportunities for leadership roles
- Overall happiness/well-being
- Performance skills (e.g., music)
- Planning/organizing
- Positive attitude about the future
- Positive behavior
- Problem solving
- Productive use of leisure time
- Projected success in career/the future/college
- Public speaking skills
- Respect for diversity
- Respect for others
- Social/interpersonal skills
- Task orientation
- Understanding of a value system
- World view broadened
While there were some subsamples for which the results were neutral, this set
of evaluations did not report negative youth development results.
^ Back to Top
| |
Harvard Family Research Project Out-of-School Time Program
Evaluation Database
The Harvard Family Research Project (HFRP) Out-of-School
Time Program Evaluation Database contains profiles of out-of-school time
(OST) program evaluations. Its purpose is to provide accessible information
about previous and current evaluations to support the development of high
quality evaluations and programs in the OST field.
Types of Programs Included in the Database
Evaluations in the database meet the following three criteria:
- The evaluated program/initiative operates during out-of-school time.
- The evaluation(s) aim to answer a specific evaluation question or
set of questions about a specific program/initiative.
- The evaluated program/initiative serves children between the ages
of 5 and 19.
Types of Information Included in the Database
Each profile contains detailed information about the evaluations as well
as an overview of the OST program/initiative itself. Web links to actual
evaluation reports, where available, are also provided, as are program
and evaluation contacts.
How to Use the Database
The database is located in the OST section of the HFRP website at: www.gse.harvard.edu/hfrp/
projects/afterschool/evaldatabase.html. The search mechanism allows
users to refine their scan of the profiles to specific program and evaluation
characteristics and findings information.
The Scan for This Snapshot
For this review, we conducted two scans. First, we checked off the box
on the search page marked “Experimental” and obtained a list
of all the experimentally designed evaluations in the database. Then,
we went back to the search page and checked off the box marked “Quasi-Experimental”
for all the quasi-experimentally designed evaluations.
|
Participation Rates Related to Outcomes
OST programs have been characterized by mixed patterns of program participation
by young people, both in the frequency and duration of their program attendance.
However, with the increased public investments in OST programs, we are seeing
increased expectations and accountability for how young people will be impacted
as a result of their participation in these programs. This has raised important
research questions for evaluators of OST programs with significant implications
for program expectations and design. Specifically, How much participation is
enough? What level of participation—times per week and duration of involvement
over time—is required to predict positive program effects on youth outcomes?
A growing number of OST evaluations have included research questions that help
them assess the important link between duration and intensity of participation
and participant outcomes. Nine of the studies in this review examined their
outcomes findings in light of program participation rates and eight found statistically
significant positive relationships between time spent in the program and academic
and positive youth development outcomes. For example, the 4-H Youth Development
program (Cornell Cooperative Extension) evaluation found that the longer youth
participate in 4-H (as measured in years spent in the program), the more likely
they were to have learned a specific skill from the program. Further, duration
of participation was linked to higher scores on a developmental assets assessment.
Examining the set of results that related frequency of participation to academic
performance, there is a similar pattern. While the 21st Century Community Learning
Centers evaluation did not find a relationship between frequency of participation
and academic achievement, the other seven studies that analyzed outcomes in
relation to participation reported that greater frequency of participation was
associated with better school attendance rates, lower rates of course failure,
and higher measures of academic achievement. The latest The After-School Corporation
(TASC) evaluation indicates that students who participated in TASC the most
consistently and for the longest period of time experienced the greatest gains
in math as assessed by standardized achievement tests. Further, the Maryland
After School Community Grants Program found that shorter programs—those
meeting fewer than 9.5 hours per week—actually had significant negative
effects on academic performance.
While there is still much work to be done to answer the question of how much
is enough, it is clear that programs are beginning to collect evaluation data
that can help program leaders better understand how participation rates affect
participant outcomes.
^ Back to Top
Conclusion
It is important to underscore that out-of-school time programming is just emerging
as a field. Currently, OST programs vary greatly in program quality and consistency
of participation. And, until recently, there was relatively little attention
given to OST programs regarding their actual impact on young people. The high
level of expectations that we now hold for OST programming is a recent phenomenon
and provides an opportunity to redefine the field and its evaluation. As such,
the field is now working to set realistic outcomes for OST programs and to implement
evaluations that can help identify best practices and standards to guide program
design and implementation in the service of achieving positive outcomes.
Just as the OST field is in its fledgling stage of development, so is the knowledge
as to how to best evaluate program efforts. Some “flagship” evaluations
(such as those of TASC and Los Angeles Better Educated Students for Tomorrow)
have been underway for several years, but for the most part, evaluating OST
programs is unfamiliar territory for many program leaders. However, the new
context of scientifically based research means that now, more than ever, it
is important that OST programs use evaluation to build the case for continued
support of their programs—and, if possible, that they do so using scientifically
based research practices.7
While there is no substitute for rigorous experimentally designed evaluations,
the reality of the context of OST programming makes this type of design challenging
and not applicable to many OST settings. Alternatively, many OST programs included
in our review have chosen to conduct quasi-experimental evaluations that use
comparison groups to make statements about program effectiveness. Although subject
to selection bias, quasi-experimental evaluations can provide a reasonable assessment
of program impact. And, combined with results from experimental studies, they
can be used to examine the range of OST program impacts so that decision makers
can better understand the benefits and limitations of OST programs.
Finally, programs need to continue to collect implementation information that
can provide useful feedback to programs for quality improvement. Moving forward,
there may always be a tension between collecting data for program improvement
and collecting data to satisfy stakeholder accountability requirements. Ultimately,
programs need to do both, using implementation information to create the context
in which to interpret program impacts.
Acknowledgements
Preparation of this Out-of-School Time Evaluation Snapshot was made possible
through the support of the Charles Stewart Mott Foundation. The authors wish
to thank HFRP Consultants Julia Coffman,
Sherri Lauver, and Suzanne Bouffard for their review. We also appreciate the
thoughtful review from Sam Piha from Community Network for Youth Development
and Heather B. Weiss, Director
of HFRP.
1 Our database contains
profiles of out-of-school time (OST) program evaluations, which are searchable
on a wide range of criteria. It is available in the OST section of the HFRP
website at www.gse.harvard.edu/hfrp/projects/afterschool/evaldatabase.html.
2 This and future
Snapshots in the series will be available in the OST section of the HFRP website
at www.gse.harvard.edu/hfrp/projects/afterschool/resources/index.html.
(To be notified when Snapshots become available online sign up for our OST website
change notification email at www.gse.harvard.edu/hfrp/subscribe.html.)
3 While some OST evaluation
reviews have imposed evaluation quality criteria to define their review’s
sample size, we chose to take a more comprehensive approach to look across a
range of evaluations. For reviews that applied stringent evaluation criteria
to identify their samples, in the Recent Reviews of OST Evaluations box, see
the Hollister and the Scott-Little, Hamann, and Jurs publications.
4 P-values indicate
levels of significance of statistical tests and indicate the probability that
the result obtained would occur by chance. Lower p-values are associated with
stronger statements of significance. This review uses a p<.10 value because
that is a cutoff that many OST evaluators employ.
5 We plan to post
a listing of academic performance measures and their data sources compiled from
the HFRP OST Program Evaluation Database in the OST section of our website in
summer 2003. (To be notified when this resource is available sign up for our
OST website change notification email at www.gse.harvard.edu/hfrp/subscribe.html.)
6 This measure was
generally assessed by teacher and student reports of overall academic gains.
As such, it differs from more specific measures of academic achievement such
as grades and test scores.
7 For an overview
of scientifically based research, see Bouffard, S. (2003). Doing what works:
Scientifically based research in education. The Evaluation Exchange, 9(1),
15, 17. www.gse.harvard.edu/hfrp/eval/issue21/bbt1.html
^ Back to Top
Appendix A: Program Descriptions and Evaluation Reports
|
Program Name & Description
|
Experimental & Quasi-Experimental Evaluations |
|
The 21st Century Community Learning Centers program provides expanded
learning opportunities for elementary and middle school children in a
supervised environment nationwide.
|
U.S. Department of Education, Office of the Under Secretary. (2003).
When schools stay open late: The national evaluation of the 21st-Century
Community Learning Centers program, first year findings. Washington,
DC: Author. Available at www.ed.gov/pubs/21cent/firstyear.
|
|
The 4-H Youth Development Program – Cornell Cooperative Extension
is an experiential education program for youth ages 5 to 19 in New York
State.
|
Rodriguez, E., Hirschl, T. A., Mead, J. P., & Goggin, S. E. (1999).
Understanding the difference 4-H Clubs make in the lives of New York
youth: How 4-H contributes to positive youth development. Ithaca,
NY: Cornell University. Available at nys4h-staff.cce.cornell.edu/4-HClubStudy.htm.
|
|
The Across Ages program uses older adults (age 55+) as mentors
for at-risk youth in Philadelphia, Pennsylvania to help youth develop
awareness, self-confidence, and skills to resist drugs and overcome obstacles.
|
LoSciuto, L., Rajala, A. K., Townsend, T. N., & Taylor, A. S. (1996).
An outcome evaluation of Across Ages: An intergenerational mentoring approach
to drug prevention. Journal of Adolescent Research, 11(1), 116–129.
|
|
The After School Achievement Program provides a safe, supervised
place for youth in Houston, Texas and aims to reduce delinquency, crime,
and school dropout and enhance academic enrichment and positive citizenship.
|
Smith, D. W., & Zhang, J. J. (2001). Shaping our children’s
future: Keeping a promise in Houston communities year 4 evaluation of
the Mayor’s After School Achievement Program (ASAP). Houston,
TX: University of Houston.
|
|
Bayview Safe Haven is an after school program in the San Francisco
Bayview/Hunter’s Point neighborhood in California for at-risk youth
ages 10 to 17. It is designed to help youth stay in school and out of
the criminal justice system, while positioning them for a responsible
adulthood and improving the quality of life in their families and communities.
|
LaFrance, S., Twersky, F., Latham, N., Foley, E., Bott, C., & Lee,
L. (2001). A safe place for healthy youth development: A comprehensive
evaluation of the Bayview Safe Haven. San Francisco, CA: BTW Consultants
& LaFrance Associates.
|
|
The Big Brothers and Big Sisters of America program has nationwide
affiliates that provide one-on-one mentoring to at-risk youth between
the ages of 10 and 16.
|
Tierney, J. P., Grossman, J. B. (with Resch, N. L.). (2000). Making
a difference: An impact study of Big Brothers Big Sisters. Philadelphia:
Public/Private Ventures.
|
|
The Children’s Aid Society Carrera-Model Teen Pregnancy Prevention
Program is a national program that aims to empower youth, help them
develop a desire for a productive future, and aid them in improving their
sexual literacy and understanding the consequences of sexual activity.
|
Philliber, S., Kaye, J., & Herrling, S. (2001). The national evaluation
of the Children’s Aid Society Carrera-Model Program to prevent teen
pregnancy. Accord, NY: Philliber Research Associates. Available at
www.childrensaidsociety.org/
media/general/cas-full_12-site_report1.pdf.
|
|
From 1995 to 1997 the Extended-Day Tutoring Program provided after
school literacy tutoring based on the Success for All model to elementary
school students in the city of Memphis, Tennessee Title I schools.
|
Ross, S. M., Lewis, T., Smith, L., & Sterbin, A. (1996). Evaluation
of the extended-day tutoring program in Memphis city schools: Final report
to CRESPAR. Memphis, TN: University of Memphis.
|
|
Foundations operates before and after school enrichment programs
for children pre-K through twelfth grade in several urban schools in the
Mid-Atlantic and Northeast of the U.S.
|
Hamilton, L. S., Le, V., & Klein, S. P. (1999). Foundations School-Age
Enrichment Program: Evaluation of student achievement. Santa Monica,
CA: RAND Education.
|
|
The Howard Street Tutoring Program provides after school remedial
reading instruction through one-on-one tutoring to second and third graders
in Chicago, Illinois who have fallen behind their peers in reading.
|
Morris, D., Shaw, B., & Perney, J. (1990). Helping low readers in
grades 2 and 3: An after-school volunteer tutoring program. The Elementary
School Journal, 91(2), 133–150.
|
|
Los Angeles Better Educated Students for Tomorrow aims to provide
youth in Los Angeles, California with: a safe environment, enhanced opportunities
through the integration of an educational support structure, educational
enrichment activities, recreational activities, and interpersonal skills
and self-esteem development.
|
Huang, D., Gribbons, B., Kim, K. S., Lee, C., & Baker, E. L. (2000).
A decade of results: The impact of the LA’s BEST after school
enrichment initiative on subsequent student achievement and performance.
Los Angeles: UCLA Center for the Study of Evaluation, Graduate School
of Education & Information Studies, University of California. Available
at www.lasbest.org/resourcecenter/uclaeval.pdf.
Brooks, P. E., Mojica, C. M., & Land, R. E. (1995). Final evaluation
report: Longitudinal study of LA’s BEST after school education and
enrichment program, 1992–94. Los Angeles: UCLA Center for the
Study of Evaluation, Graduate School of Education & Information Studies,
University of California.
|
|
The Louisiana State Youth Opportunities Unlimited summer program
provides dropout prevention services for at-risk youth on the Louisiana
State University campus.
|
Shapiro, J. Z., Gaston, S. N., Hebert, J. C., & Guillot, D. J. (1986).
The LSYOU project evaluation. Baton Rouge: College of Education
Administrative and Foundational Services, Louisiana State University.
|
|
The Maryland After School Community Grant Program serves to strengthen
youth resiliency and prevent substance abuse, violence, and delinquency
by increasing the availability of high quality, structured after school
programs for youth in Maryland.
|
Weisman, S. A., Soulé, D. A., & Womer, S. C. (under the direction
of Gottfredson, D. C.). (2001). Maryland After School Community Grant
Program: Report on the 1999–2000 school year evaluation of the phase
I after-school programs.
|
|
The New Orleans ADEPT Drug and Alcohol Community Prevention Project
is a primary-level alcohol and other drug use prevention program that
provides after school child care to 24 low-income elementary schools of
the New Orleans, Louisiana public school district.
|
Ross, J. G., Saavedra, P. J., Schur, G. H., Winters, F., & Felner,
R. D. (1992). The effectiveness of an after-school program for primary
grade latchkey students on precursors of substance abuse. Journal of
Community Psychology, OSAP special issue, 22–38.
|
|
North Carolina Cooperative Extension Service provides training
and technical assistance to school-age care providers throughout North
Carolina with the aim of raising the quality of out-of-school time experiences
for elementary through high school students.
|
Locklear, E. L., & Mustian, R. D. (1998). Extension-supported school-age
care programs benefit youth. Journal of Extension, 36(3). Available
at www.joe.org/joe/1998june/rb4.html.
|
|
The Project Learn/Educational Enhancement Program is a community-based
program implemented in local Boys & Girls Clubs across the country
and designed to improve academic achievement of at-risk students.
|
Schinke, S. P., Cole, K. C., & Poulin, S. R. (2000). Enhancing the
educational achievement of at-risk youth. Prevention Science, 1(1),
51–60.
|
|
The Quantum Opportunities Program was a national pilot initiative
from 1989 to 1993 that tested whether youth from families receiving public
assistance could make a “quantum leap” up the ladder of opportunity
if given a comprehensive and multi-year set of supports.
|
Hahn, A., Leavitt, T., & Aaron, P. (1994). Evaluation of the Quantum
Opportunities Program: Did the program work? Waltham, MA: Brandeis
University.
|
|
San Diego’s “6 to 6” Extended School Day Program
provides access to high quality, affordable enrichment programs before
and after school to elementary and middle school students in San Diego,
California.
|
Hoffman, J. (2001). San Diego After School Regional Consortium: Academic
indicator report 1999–2000. San Diego, CA: Hoffman, Clark &
Associates.
|
|
The San Francisco Beacons Initiative aims to help youth in San
Francisco, California through participation in out-of-school time Beacons
Center activities focused on helping youth develop competencies and become
responsible adults.
|
Walker, K. E., & Arbreton, A. J. A. (2001). Working together to
build Beacon Centers in San Francisco: Evaluation findings from 1998–2000.
Philadelphia: Public/Private Ventures. Available at www.ppv.org/ppv/publications/
publications_description.asp?search_id=8&publication_id=118.
|
|
The Santa Ana After School Learning and Safe Neighborhoods Partnerships
Program integrates academics with recreational enrichment to meet
students’ academic and social needs. The program operates in four
urban public middle schools in the Santa Ana Unified School District in
California.
|
Prenovost, J. K. E. (2001). A first-year evaluation of after school
learning programs in four urban middle schools in the Santa Ana Unified
School District. Irvine, California: Author.
|
|
The Stay SMART program is a national prevention program offered
by the Boys & Girls Clubs of America that seeks to teach youth a broad
spectrum of social and personal competence skills and to help them identify
and resist peer and other social pressures to use alcohol, cigarettes,
and marijuana, as well as to engage in early sexual activity. SMART Leaders
I and II are booster programs designed to reinforce the initial program.
|
St. Pierre, T. L., Kaltreider, D. L., Mark, M. M., & Aikin, K. J.
(1992). Drug prevention in a community setting: A longitudinal study of
the relative effectiveness of a three-year primary prevention program
in Boys & Girls Clubs across the nation. American Journal of Community
Psychology, 20(6), 673–706.
St. Pierre, T. L., Mark, M. M., Kaltreider, D. L., & Aikin, K. J.
(1995). A 27-month evaluation of a sexual activity prevention program
in Boys & Girls Clubs across the nation. Family Relations, 44,
69–77.
|
|
The Teen Outreach Program is a nationwide school-based program
involving young people ages 12 to 17 in volunteer service in their communities.
Designed to increase academic success and decrease teen pregnancy, it
helps youth develop positive self-images, learn valuable life skills,
and establish future goals.
|
Allen, J. P., Philliber, S., Herrling, S., & Kuperminc, G. P. (1997).
Preventing teen pregnancy and academic failure: Experimental evaluation
of a developmentally based approach. Child Development, 64(4),
729–742.
Allen, J. P., & Philliber, S. (2001). Who benefits most from a broadly
targeted prevention program? Differential efficacy across populations
in the Teen Outreach Program. Journal of Community Psychology, 29(6),
637–655. Available at www3.interscience.wiley.com/cgi-bin/
abstract/85514589/ABSTRACT.
|
|
The After-School Corporation After-School Program has a two-part
mission: (1) to enhance the quality of after school programs in New York
State by emphasizing program components associated with student success
and program sustainability and (2) to increase the availability of after
school opportunities by providing resources and strategies for establishing
and expanding after school projects.
|
Reisner, E. R., White, R. N., Birmingham, J., & Welsh, M. (2001).
Building quality and supporting expansion of After-School Projects:
Evaluation results from the TASC After-School Program’s second year.
Washington, DC: Policy Studies Associates.
Reisner, E. R., Russell, C. A., Welsh, M. E., Birmingham, J., & White,
R. N. (2002). Supporting quality and scale in after-school services
to urban youth: Evaluation of program implementation and student engagement
in TASC After-School Program’s third year. Washington, DC: Policy
Studies Associates.
|
|
The Thunderbirds Teen Center Program is a multifunctional facility
in North Phoenix, Arizona that aims to promote teens’ positive self-development
by providing a comprehensive service system that focuses on the whole
individual during out-of-school time.
|
Baker, D., Hultsman, J., & Garst, B. A. (1998). Thunderbirds Teen
Center Program evaluation. Phoenix Parks, Recreation and Library &
Arizona State University. Available at rptsweb.tamu.edu/Faculty/Witt/conpubs/thunder.pdf.
|
|
Virtual Y brings YMCA after school programs and staff into 100
New York City public elementary schools. It offers support for classroom
learning by extending the school day and helping children achieve reading
proficiency through literacy-based activities.
|
Foley, E. M., & Eddins, G. (2001). Preliminary analysis of Virtual
Y after-school program participants’ patterns of school attendance
and academic performance. Final evaluation report program year 1999–2000.
New York: National Center for Schools and Communities, Fordham University.
Available at www.ncscatfordham.org/binarydata/files/rpt_academic.pdf.
Foley, E. M., & Eddins, G. (2001). Impact of the Virtual Y on
children’s classroom behavior. Final evaluation report program year
1999–2000. New York: National Center for Schools and Communities,
Fordham University.
|
|
The Voyager Summer Program is a national summer intervention program
with a core curriculum that helps struggling readers. The program aims
to close the achievement gap between white and minority students.
|
Roberts, G. (2000). Technical evaluation report on the impact of Voyager
summer reading interventions. Austin: University of Texas.
|
|
The Woodrock Youth Development Project is a program of intervention
strategies and support systems for youth in the Kensington neighborhood
in Philadelphia, Pennsylvania. It aims to reduce alcohol, tobacco, and
other drug use among adolescents by improving youths’ awareness about
the dangers of substance abuse, problem-solving and coping skills, self-perceptions,
academic achievement, and cultural pride.
|
LoSciuto, L., Freeman, M. A., Harrington, E., Altman, B., & Lanphear,
A. (1997). An outcome evaluation of the Woodrock Youth Development Project.
Journal of Early Adolescence, 17(1), 51–66.
|
^ Back to Top
Appendix B: Outcomes Linked to Participation in OST Programs
While this set of experimental and quasi-experimental evaluations assessed
many outcomes, this table only reports those for which there were statistically
significant findings (p<.10). “E” denotes experimental design;
“QE” denotes quasi-experimental design. Results are grouped as positive,
neutral, or negative and listed in alphabetical order. Parentheses following
an improvement area refer to the subsample for which the result was statistically
significant.
| Program Name |
Outcomes |
|
21st Century Community Learning Centers program, national (E and QE)
|
Academic
Middle School Results
Positive: achievement motivation, grades (black and Hispanic middle school
students, math only), school attendance
Neutral: homework completion, overall academic performance
Elementary School Results
Positive: achievement motivation
Neutral: grades (except for social studies), homework completion
Participation
Students who attended more frequently did not have higher academic outcomes
than students who attended less frequently.
Prevention
Middle School Results
Negative: avoidance of delinquency, avoidance of drug/alcohol use
Youth Development
Middle School Results
Neutral: interactions/relationships with peers
Negative: conflict resolution
Elementary School Results
Positive: productive leisure time (frequent participants only)
Neutral: interactions/relationships with peers, positive behavior
Participation
While elementary school participants were no more or less likely to watch
TV than the control students, frequent elementary school participants
were significantly more likely to be engaged in tutoring and extracurricular
activities such as band, drama, art, etc.
|
|
4-H Youth Development Program – Cornell Cooperative Extension, NY
(QE)
|
Academic
Positive: achievement motivation, educational aspiration, grades
Youth Development
Positive: communication skills, community involvement, confidence/self-esteem,
conflict resolution, decision making, desire to help others, goal setting,
interactions/relationships with adults, interactions/relationships with
peers, leadership skills, planning/organizing, problem solving, projected
success in career/the future/college, public speaking skills, respect
for diversity, understanding of a value system, world view broadened
Participation
The longer youth participate in 4-H, the more likely they are to report
having learned a specific skill from 4-H. Length of time that youth participate
in 4-H was found to have a significant impact on asset development. Longer
participation led to higher scores on the developmental asset areas. Type
of 4-H Club was not found to be associated with developmental asset outcomes.
|
|
Across Ages,* Philadelphia, PA (E)
* Results reported are for the curriculum + mentoring group only.
|
Academic
Positive: attitude toward school/academics, school attendance
Prevention
Positive: avoidance of drug/alcohol use
Youth Development
Positive: community involvement, general well-being, interactions/relationships
with adults, projected success in career/the future/college (most significant
for those with “exceptional mentoring relationships”)
|
|
After School Achievement Program, Houston, TX (QE)
|
Academic
Positive: achievement test scores (in science and fine arts)
Prevention
Neutral: avoidance of delinquency
|
|
Bayview Safe Haven, San Francisco, CA (QE)
|
Academic
Positive: reduced suspensions
Neutral: expulsions, school attendance (all results are for the intervention
period only, not for post-intervention follow-up)
Prevention
Positive: avoidance of delinquency
Participation
Youth who participated voluntarily had fewer arrests than those who participated
as a condition of probation.
|
|
Big Brothers and Big Sisters of America, national (E)
|
Academic
Positive: competence (strongest for minority females), grades (strongest
for minority females), school attendance (strongest for females)
Neutral: homework completion
Prevention
Positive: avoidance of drug/alcohol use (strongest for minority youth),
avoidance of violence
Neutral: avoidance of delinquency
Youth Development
Positive: interactions/relationships with peers
Neutral: confidence/self-esteem, world view broadened
|
|
Children’s Aid Society Carrera-Model Teen Pregnancy Prevention Program,
national (E)
|
Academic
Positive: achievement test scores, educational aspiration, overall academic
performance
Prevention
Positive: avoidance of drug/alcohol use (boys), avoidance of sexual activity
(marginal significance), knowledge of sexuality issues, reduced pregnancy
rates (girls), use of safe sex practices (girls)
Youth Development
Positive: computer skills, job experience/skills, money management skills
|
|
Extended-Day Tutoring Program, Memphis, TN (QE)
|
Academic
Positive: achievement test scores (tutored third grade frequent participants
only)
|
|
Foundations, national (QE)
|
Academic
Positive: achievement test scores (highest gains for fourth graders in
year one and first and second graders in year two; no gains for grade
5), grades (in math for one site only)
|
|
Howard Street Tutoring Program, Chicago, IL (E)
|
Academic
Positive: achievement test scores
|
|
Los Angeles Better Educated Students for Tomorrow, Los Angeles, CA (QE)
|
Academic
Positive: achievement test scores, attitude toward school/academics,
educational aspiration
Participation
Greater participation was significantly related to positive achievement
on standardized tests, better school attendance, and fewer absences.
Youth Development
Positive: interactions/relationships with adults (especially after school
staff)
|
|
Louisiana State Youth Opportunities Unlimited, LA (E)
|
Academic
Positive: achievement test scores, attitude toward school/academics,
competence, educational aspiration, school attendance
Youth Development
Positive: job experience/skills
|
|
Maryland After School Community Grants Program, MD (E and QE)
|
Academic
Neutral: overall academic performance
Participation
Shorter programs—those meeting for less than 9.5 hours a week—appear
to have significant negative effects on academic performance (p<.05).
Prevention
Positive: avoidance of delinquency (only significant for one middle school
site), avoidance of drug/alcohol use (two sites only)
Youth Development
Positive: positive behavior (middle school students only), social/interpersonal
skills
|
|
New Orleans ADEPT Drug and Alcohol Community Prevention Project, New
Orleans, LA (QE)
|
Academic
Positive: achievement test scores for participation in a self-esteem-building
curriculum group
Neutral: achievement test scores overall
Youth Development
Neutral: confidence/self-esteem (and other personality measures), overall
happiness
|
|
North Carolina Cooperative Extension Service, NC (QE)
|
Academic
Positive: achievement test scores, homework completion, school attendance
Youth Development
Positive: communication skills, community involvement, interactions/relationships
with adults, interactions/relationships with peers, maturity, task orientation
(this set of results only reached significance for school-age providers
surveys; surveys of parents and classroom teachers revealed higher, but
not significant results)
|
|
Project Learn/Educational Enhancement Program, national (QE)
|
Academic
Positive: academic involvement, achievement test scores, attitude toward
school/academics, grades, overall academic performance, school attendance
Participation
Increased program attendance led to higher outcomes on measures of academic
achievement and enjoyment of reading.
|
|
Quantum Opportunities Program, national (E)
|
Academic
Positive: achievement test scores, college attendance, school attendance
Prevention
Positive: avoidance of drug/alcohol use, reduced pregnancy rates
Youth Development
Positive: community involvement, projected success in career/the future/college
Neutral: job experience/skills (as assessed by self-reporting of the need
for help in this area), overall happiness (regarding family life)
|
|
San Diego’s “6 to 6” Extended School Day Program, San
Diego, CA (QE)
|
Academic
Positive: achievement test scores
Neutral: school attendance
|
|
San Francisco Beacons Initiative, San Francisco, CA (QE)
|
Youth Development
Positive: opportunities for leadership roles (middle school students),
productive leisure time
|
|
Santa Ana After School Learning and Safe Neighborhoods Partnership Program,
Santa Ana, CA (QE)
|
Academic
Positive: school attendance (significant for high-dosage participants
only)
Neutral: achievement test scores (not significant, but high-dosage participants
tended to score better)
Participation
Increased program attendance led to better academic outcomes overall.
|
|
Stay SMART and SMART Leaders program, national (QE)
|
Prevention
Positive: decrease in perceived benefits of sex (non-virgins only), attitudes
about sexual issues (non-virgins), avoidance of drug/alcohol use, avoidance
of sexual activity (non-virgins only), knowledge about drug/alcohol use,
knowledge of sexual issues
|
|
Teen Outreach Program, national (E and QE)
|
Academic
Positive: reduced suspension, grades (in one of the two evaluations,
this was most significant for females, ethnic minority youth, and youth
with histories of suspension)
Participation
In one of the two evaluations, significantly less course failure was reported
by students who worked more volunteer hours in the program.
Prevention
Positive: reduced pregnancy rates (females; most significant for those
who were already teen parents)
|
|
The After-School Corporation After-School Program, NY (QE)
|
Academic
Positive: achievement test scores (for mathematics; most significant
for low-achieving, special education, English language learner, African-American,
and Hispanic students), school attendance
Neutral: achievement test scores (for language arts)
Participation
Positive academic outcomes were reported for “active participants,”
i.e., those that participated at least three days per week. Those who
participated at least 60 days and at least 60% of possible days gained
four scale-score points in math after two years of participation and six
scale-score points after three years. Those who participated at least
80 days and at least 80% of possible days gained six scale-score points
in math after two years of participation.
Youth Development
Positive: interactions/relationships with peers, leadership skills (middle
school students), performance skills (such as playing a musical instrument,
singing, or public speaking), world view broadened
|
|
Thunderbirds Teen Center Program, Phoenix, AZ (QE)
|
Academic
Neutral: grades, school attendance
|
|
Virtual Y, New York, NY (QE)
|
Academic
Positive: achievement test scores (fourth grade), school attendance (third
and fourth grade)
Youth Development
Positive: interactions/relationships with peers, positive behavior, social/interpersonal
skills, task orientation
|
|
Voyager Summer Program, national (QE)
|
Academic
Positive: achievement test scores
|
|
Woodrock Youth Development Project, Philadelphia, PA (E)
|
Academic
Positive: school attendance
Prevention
Positive: avoidance of drug/alcohol use
Neutral: attitudes about drug/alcohol use
Youth Development
Positive: confidence/self-esteem, positive behavior, decreased aggression,
respect for diversity
|
|
