Out-of-School Time Evaluation Snapshot
A Review of Activity Implementation in Out-of-School Time Programs
Number 2, August 2003
Suzanne Bouffard and Priscilla
M. D. Little
Download this brief:
This Snapshot is also available in hard copy for free. Please go
to our publications list
to learn how to order it.
Harvard Family Research Project’s series of Out-of-School Time
Evaluation Snapshots distill the wealth of information compiled in our Out-of-School
Time Program Evaluation Database1
into a single report. Each Snapshot examines a specific aspect of out-of-school
time (OST) evaluation. This Snapshot examines the range and scope of
activities being implemented in current out-of-school time programs to set a
context for understanding the links between program activities and positive
outcomes for youth.
As the need grows for quality programs during nonschool hours, there is increasing
evidence that out-of-school time (OST) programs are associated with positive
outcomes for youth. However, to date, most evaluators have examined OST programs
as a whole without taking into account the specific activities they offer. As
a result, the processes by which these programs influence child outcomes are
not yet well understood. This Snapshot surveys the range of activities
being implemented in OST settings across the country in order to understand
and promote effective OST programming.
| |
Harvard Family Research Project Out-of-School Time Program
Evaluation Database
The Harvard Family Research Project (HFRP) Out-of-School
Time Program Evaluation Database contains profiles of out-of-school time
(OST) program evaluations. Its purpose is to provide accessible information
about previous and current evaluations to support the development of high
quality evaluations and programs in the OST field.
Types of Programs Included in the Database
Evaluations in the database meet the following three criteria:
- The evaluated program/initiative operates during out-of-school time.
- The evaluation(s) aim to answer a specific evaluation question or
set of questions about a specific program/initiative.
- The evaluated program/initiative serves children between the ages
of 5 and 19.
Types of Information Included in the Database
Each profile contains detailed information about the evaluations as well
as an overview of the OST program/initiative itself. Web links to actual
evaluation reports, where available, are also provided, as are program
and evaluation contacts.
How to Use the Database
The database is located in the OST section of the HFRP website at: www.gse.harvard.edu/hfrp/
projects/afterschool/evaldatabase.html. The search mechanism allows
users to refine their scan of the profiles to specific program and evaluation
characteristics and findings information.
The Scan for This Snapshot
For this review, we conducted two scans. First, we checked off the box
on the search page marked “Activity Implementation” to obtain
a list of all of the evaluations in the database that reported any information
on implementation (including specific program activities, the degree to
which programs conformed to their stated goals, and challenges to implementation).
Next, we conducted a manual scan of these evaluation profiles in order
to find those that reported specific activities offered in the OST programs.
|
Why Examine Activities in OST Programs?
There are multiple reasons for examining the range of specific activities in
OST programs. First, there is an increasing demand for research that links specific
program activities with children’s outcomes. These links can help answer
the question of what defines quality in OST programs and can contribute to the
creation of optimally effective programs. Before such links can be explored
however, it is necessary to understand the components of existing programs.
Secondly, research on voluntary extracurricular activities suggests that the
specific types of activities in which youth engage make a difference for youth
outcomes (Eccles & Barber, 1999; Marsh & Kleitman, 2002).
Which OST Programs Evaluate Activity Implementation?
We culled the data for this review from profiles of evaluations of out-of-school
time programs included in the Harvard Family Research Project Out-of-School
Time Program Evaluation Database (see box). As of summer 2003 when we collected
the data, 27 program evaluations included information about activity offerings
and therefore were included in our analysis. These programs range in size from
small single-site programs to national multisite programs. This Snapshot
only includes information on program activities, but many programs also assessed
youths’ developmental outcomes and we address these findings in the first
Snapshot in this series (available at www.gse.harvard.edu/hfrp/projects/
afterschool/resources/index.html). Furthermore, this analysis reports only
on activities for children and adolescents (although some of these programs
also offered services for adults). We provide a list of all the programs we
reviewed in the appendix.
How Do Evaluators Assess Activity Implementation?
Evaluators have used a variety of methods to assess the spectrum of OST activities.
Of the 27 programs in this review, 23 reported the methods they used to collect
activity data. Among these programs, observations conducted by researchers
constituted the most frequent strategy for evaluating activity implementation
(14 programs). Reports from OST program staff were also common (9 programs
used staff surveys and 7 programs used staff interviews). Some programs collected
reports from youth participants (9 used surveys and 4 used interviews)
and reports from parents (2 used surveys and 1 used interviews). In addition,
eight programs used reviews of program documents, such as activity logs
and attendance records. Many program evaluators employed multiple data sources
to report on the types of activities being provided.
What Is the Scope of Activities Provided by OST Programs?
Not surprisingly, this review confirms that OST programs range in scope from
targeted, single-activity programs to programs offering a diverse array of activities.
The majority of programs reviewed (19 programs, approximately 70%) offer multiple
activities, while eight programs (approximately 30%) focus on only one type
of activity. Of the single-activity programs, six (75%) focus on academic activities,
one on fostering positive adult relationships, and one on the development of
positive self-concept. Thus it appears that while most OST programs are multicomponent,
the single-component programs tend to provide academic enrichment activities.
The majority of the multicomponent programs give youth a choice of activities
and youth often participate in more than one activity per day. In the single-component
programs, activities are more likely to be mandatory. In these programs, the
broad activity category (e.g., academics, self-concept) remains constant, but
slight variations in activities do occur. Sometimes this variation occurs within
one day. For example, in the Extended-Day Tutoring Program, which aims to improve
literacy skills in elementary school children, each session begins with a listening
and reading comprehension task, proceeds to a workbook exercise, and concludes
with time for a variety of other activities, including partner reading, book
reports, and computer skills.2
In other programs, activity variation occurs from one session to the next. For
example, in the SECME Raising Interest in Science and Engineering project, which
is designed to promote girls’ self-confidence in math and science, each
session has a specific focus, ranging from technology to engineering to the
process of applying to magnet schools. These examples illustrate an important
point about OST program activities—even in the single-component programs,
like those that provide academic enrichment, there can be variation in the types
of activities offered.
What Are the Specific Activities Offered by OST Programs?
Table 1 summarizes the specific types of activities offered across the OST programs
in this review and the number of programs that offer each activity, listed in
descending order of frequency. While some programs operate at only one site,
others operate at multiple sites. In many of the multisite programs, activities
vary slightly from one site to another. For this analysis, a program is considered
to include the activity if any of its sites offers the activity.
| Table 1: OST Program Activities and Their
Frequency |
| Activity |
Number of Programs Offering Activity |
Academics*
Arts
Recreational sports/exercise
Computer skills
Community service
Field trips
Positive youth development*
Career skills/exploration
Unstructured play
One-to-one adult mentoring |
21
14
14
13
8
7
7
6
4
2 |
| * Specific examples are given in the box
titled Spotlight on Academic and Positive Youth Development Activities below. |
It is clear from Table 1 that this set of OST programs offers a wide range
of activities that target several goals. While the majority of programs offer
some sort of academic enrichment, many programs also offer arts, sports, and
community service as well as services that may be otherwise unavailable to program
participants, including academic tutoring, career counseling, and computer skills
training. A few programs in this set are primarily focused on prevention of
substance abuse and teen pregnancy (for example, Owensboro Public Schools 21st
Century Community Learning Centers and Children’s Aid Society Carrera-Model
Teen Pregnancy Prevention Program). These findings demonstrate that many OST
programs serve a larger need than basic childcare or recreation; for many families
and communities these programs can also provide important services and skills
training.
| |
Spotlight on Academic and Positive Youth Development Activities
Academic Activities
The widespread public interest in academic outcomes led us to further
examine the content of the 21 programs in Table 1 that include academic
enrichment. Most programs provide multiple types of academic enrichment.
Specifically, 13 programs include homework help, 11 provide tutoring services,
2 offer standardized testing skills/tutoring, and 14 include other types
of academic enrichment. It is important for future evaluations to note
the variation within this general academic category because each activity
targets a specific academic purpose and therefore necessitates corresponding
performance measures to accurately assess its impact. For example, a program
that offers homework help may want to track participants’ homework
completion rates, while a program that provides general academic tutoring
may want to track overall performance in school as reported by teachers,
parents, and report cards. A program that offers multiple activities may
need to assess the outcomes of its participants in many different ways.
Positive Youth Development Activities
Similarly, due to increased attention to the value of OST programs in
developing nonacademic youth skills, we also explored the single-focus
programs with an articulated youth development focus. Like other single-focus
programs, those that primarily provide positive youth development activities
do so in a variety of ways, including leadership skills (3 programs),
social and communication skills (2), decision making and life management
(2), conflict management (1), and self-concept development (1). Many of
these skills are often thought to be inherent in traditional extracurricular
activities such as team sports and community service, however, several
of the OST programs described here go further and offer explicit training
in these skills.
|
What Are the Challenges to Implementing Activities?
Only a few programs reported the major challenges they faced in implementing
activities. Program staff and evaluators cited poor time management, inadequate
staff training, lack of resources (including space, materials, and adult volunteers),
and the need to provide after school snacks in order to sustain participant
energy and interest. Uncovering these implementation challenges is a necessary
component of documenting OST program activities and improving OST programming.
How Does Activity Implementation Information Lead to Better Programs?
Collecting activity implementation data (also known as service documentation)
is a critical first step in evaluation and program improvement for all OST programs.3
First, monitoring a program by systematically documenting the services and activities
it provides enables programs to gather information for multiple stakeholders.
Most funders, including the U.S. Department of Education through its 21st Century
Community Learning Centers grants program, require activity implementation data
as part of their accountability requirements.
Second, all programs need to understand what services they are offering and
to whom in order to build organizational capacity for self-assessment and program
modification. They can then use this information to begin a process of self-reflection,
comparing their original program goals and objectives with the activities they
offer as part of their continuous improvement efforts.
Finally, service documentation is a necessary precursor to determining which
aspects of a program lead to positive youth outcomes. Currently, most of our
research and evaluation has examined the relationship between program participation
(i.e., whether the individual was in the program) and outcomes.4
However, these studies fall short of defining the elements or services within
the programs that brought about individual change. Moving forward, it is essential
that programs conducting outcomes studies collect activity implementation information
that will help identify more specifically which elements of the program are
or are not leading to positive outcomes for its participants.
1 Our database contains
profiles of out-of-school time (OST) program evaluations, which are searchable
on a wide range of criteria. It is available in the OST section of the HFRP
website at www.gse.harvard.edu/hfrp/projects/afterschool/evaldatabase.html.
2 For more information
on this and other program evaluations included in this review, see the HFRP
Out-of-School Time Program Evaluation Database at www.gse.harvard.edu/hfrp/projects/afterschool/evaldatabase.html.
3 For information
on service documentation as a phase of the evaluation process, see Little, P.,
DuPree, S., & Deich, S. (2002). Documenting progress and demonstrating
results: Evaluating local out-of-school time programs. Cambridge, MA and
Washington, DC: Harvard Family Research Project and The Finance Project. Available
at www.gse.harvard.edu/hfrp/projects/afterschool/resources/index.html#local.
4 See for example
Simpkins, S. (2003). Does youth participation in out-of-school time activities
make a difference? The Evaluation Exchange, 9(1) 2–3, 21.
Available at www.gse.harvard.edu/hfrp/eval/issue21/theory.html.
References
Eccles, J. S., & Barber, B. L. (1999). Student council, volunteering, basketball,
or marching band: What kind of extracurricular involvement matters? Journal
of Adolescent Research, 14, 10–43.
Marsh, H. W., & Kleitman, S. (2002). Extracurricular school activities:
The good, the bad, and the nonlinear. Harvard Educational Review, 72,
464–514.
Acknowledgements
This Snapshot is based on a review of the Harvard Family Research Project
Out-of-School Time Program Evaluation Database,
which is supported by a grant from the C. S. Mott Foundation. The principal
investigator for this study is Dr.
Heather B. Weiss. The authors wish to thank Julia
Coffman, Consultant, for her review.
Appendix: Out-of-School Time Programs Included in the
Review
- 4-H Youth Development Program – Cornell Cooperative Extension
- After School Achievement Program
- After School Learning and Safe Neighborhoods Partnership Program
- Baltimore’s After School Strategy – YouthPlaces Initiative
- Big Brothers and Big Sisters of America
- Cap City Kids
- Children’s Aid Society Carrera-Model Teen Pregnancy Prevention Program
- District of Columbia 21st Century Community Learning Centers Program
- Extended-Day Tutoring Program
- Fifth Dimension/University-Community Links
- Fort Worth After School Program
- Juvenile Mentoring Program
- Los Angeles Better Educated Students for Tomorrow Program
- Louisiana State Youth Opportunities Unlimited Summer Program
- Maryland After School Community Grant Program
- New York City Beacons Initiative
- North Carolina Support Our Students Initiative
- Ohio Urban School Initiative School Age Child Care Project
- Owensboro Public Schools 21st Century Community Learning Centers
- Project for Neighborhood Aftercare Program
- Quest for Excellence
- San Diego’s “6 to 6” Extended School Day Program
- San Francisco Beacons Initiative
- School-to-Jobs Programme
- SECME Raising Interest in Science & Engineering
- The After-School Corporation After-School Program
- Virtual Y
|
Additional Resources on Evaluating Out-of-School Time Activities
The 2003 report by the U.S. Conference of Mayors, After-School
Programs in Cities Across the United States, includes information
about programs’ activity offerings. www.usmayors.org/uscm/uscm_projects_services/education/afterschool.asp
Afterschool Alliance’s Action Kit provides information on
creating and implementing age-appropriate activities in OST. (See Section
4: Lend a Hand, on page 9.) www.afterschoolalliance.org/action_kit.cfm
The Afterschool.gov website’s Planning Activities section
offers resources for designing OST activities relating to a wide range
of topics. www.afterschool.gov
Beyond the Bell™: A Toolkit for Creating Effective After-School
Programs is a practical and easy-to-use reference for those designing
after school programs. The toolkit is designed around key decision points
in six areas: management, collaboration, programming, linkages with the
traditional school day, evaluation, and communication. www.ncrel.org/after/bellkit.htm
|
|
