Out-of-School Time Evaluation Snapshot

Learning From Small-Scale Experimental Evaluations of After School Programs

Number 8, May 2006

Christopher Wimer, Harvard Family Research Project

145KB Acrobat file

Help for Acrobat file

This Snapshot is also available in hard copy for free. Please go to our publications list to learn how to order it.

Harvard Family Research Project’s series of Out-of-School Time Evaluation Snapshots distills the wealth of information compiled in our Out-of-School Time Program Evaluation Database and Bibliography into a single report. Each Snapshot examines a specific aspect of out-of-school time (OST) evaluation. This Snapshot reviews small-scale experimental evaluations of after school programs, highlighting these studies' evaluation strategies and results.

Experimental studies—studies that randomly assign participants to either a group receiving some intervention (the treatment group) or a group receiving no intervention (the control group)—are still relatively rare in both the out-of-school time (OST) field and other child- and youth-related fields, such as education.1 The rarity of such studies may be due to logistical and financial challenges and/or to the fact that random assignment can sometimes create ethical dilemmas. However, because they are less biased than many other types of studies, experimental studies are valuable for justifying public investment—financial, political, and social—in after school programs and other human services.

The evaluation of the federal 21st Century Community Learning Centers (21st CCLC) initiative is one of the largest and most publicized experimental studies of after school programs.2 Now, a growing body of experimental studies of smaller after school programs and initiatives is beginning to surface, as stakeholders in the field seek to learn when, whether, and how individual after school programs can make a difference in the lives of youth. This Snapshot examines the programs, evaluation methods, and evaluation findings of these smaller scale experimental evaluations. Although they are often overlooked, small-scale studies can inform our thinking about how to overcome the methodological challenges of conducting rigorous experiments and can provide evidence of after school programs' effectiveness in achieving results.

A Picture of Program Diversity

One of the defining characteristics of the after school field is the sheer diversity of program goals, activities, and components. The seven programs with experimental evaluations reviewed here were no exception. The brief program descriptions below give an overview of the characteristics and diversity of the sample.

Cooke Middle School After School Recreation Program (CASP) provides youth in an inner-city Philadelphia middle school with activities that promote physical, emotional, and social well-being. The program operates Monday through Thursday evenings from 5 p.m. to 7 p.m. in order to give all youth at the school, including those who attend academic programs immediately after school, the opportunity to participate. Activities, which include sports/physical fitness, art, dance, board games, reading, and homework, are designed to provide a safe, structured, adult-supervised environment that encourages positive adolescent development; provide opportunities to develop fitness skills and wellness; offer extracurricular activities that encourage students' social, emotional, intellectual, and physical skills; and teach children how to get along in a nonthreatening environment.

Gevirtz Homework Project (GHP) is an after school program in three public elementary schools in Santa Barbara, California. GHP's goal is to provide students with academic support and to improve their academic achievement through assistance with homework and study skills. Students enter GHP in fourth grade and are expected to continue through sixth grade. The program provides specific homework assistance on a regular basis in order to build a strong academic foundation and establish study skills and to address student homework needs without parental involvement. Sessions run for approximately 45 minutes per day three to four times per week. The principal at each school selects a credentialed K–6 teacher to conduct homework sessions. The teacher, the teacher's aide, or both are bilingual (Spanish and English).

Girlfriends for KEEPS (Keys to Eating, Exercising, Playing, and Sharing) (GFK) is an obesity prevention program for low-income 8- to 10-year-old African American girls in Minneapolis, Minnesota. Girls participate in the program for 12 weeks. Club meetings, held twice a week for 1 hour after school, consist of fun, culturally appropriate, interactive, hands-on activities that emphasize skill building and the practice of that week's health behavior message (e.g., selecting low-fat foods for snacks, eating smaller portions of snacks). A healthy snack, sometimes prepared by the girls, and water are offered at each meeting. The intervention seeks to increase participants' physical activity with a choice of activities such as dancing (ethnic, hip hop, aerobic) and double-dutch jump rope. Family activities, including weekly family packets sent to parents, family night events, organized neighborhood walks, and telephone calls by staff to encourage parents and to check their progress on meeting family goals around healthy eating and activity behaviors, reinforce the intervention's messages.

Go Grrrls (GG) is a preventive program that focuses on the promotion of Tucson, Arizona, middle school girls' positive psychosocial development to help them navigate through early adolescence. Based on the idea that early adolescence is a time when many youth begin experimentation with risky behaviors such as cigarette use, drug use, and sex and that this time period is characterized by many obstacles and barriers to healthy development, GG is delivered in small groups and consists of a detailed curriculum of 12 sessions built around tasks considered critical for the healthy psychosocial development of early adolescent girls in contemporary society, including being a girl in today's society, establishing a positive self-image, establishing independence, making and keeping friends, obtaining help and gaining access to resources, and planning for the future.

Hispanic After School Program (HASP) promotes positive ethno-cultural identity, school adjustment, and self-concepts of Puerto Rican children in a semirural town in Massachusetts. A community mental health center, which sponsors HASP, and the local public school system engage in cooperative mutual planning to identify and treat potentially at-risk children. The three basic expected behavioral outcomes are acceptance by teachers and students of cultural uniqueness and differences, Latino students' active bilingual speech in the presence of teachers and other children, and receptiveness of the school's students and staff to cultural differences. The program components designed to produce these outcomes include participation in Spanish/Puerto Rican singing and arts and crafts; discussion of ethnicity, ethnicity-related challenges, morals, values, sex roles, and skin color; and role-modeling by Latino/a professionals.

Howard Street Tutoring Program (HSTP), no longer in existence, provided after school remedial reading instruction through one-on-one tutoring to second and third grade children in Chicago, Illinois, who had fallen behind their peers. The program operated 4 days per week from 2:30 p.m. to 4 p.m. from early October to late May. Two groups of approximately 10 second and third graders received tutoring 2 afternoons a week. Participants were identified by their classroom teachers and paired with volunteer reading tutors who each worked with an individual child for the duration of the program. After eating a snack and listening to a story or playing a game, children were paired with their tutors and engaged in the daily activities: 15–20 minutes of contextual reading at the child's instructional level, 10–12 minutes of word study, 15 minutes of writing, 10–15 minutes of easy contextual reading, and 5–10 minutes of the adult reading to the child.

Siblings of Children with Developmental Disabilities After School Support Program (SCDDASSP) serves children from an East Coast inner city who have siblings with developmental disabilities, such as mild or moderate mental retardation. This program grew out of parents' concerns about their children without developmental disabilities, given the time and energy needed for meetings, appointments, and activities for their children with disabilities. By providing services for the nondisabled siblings, the program attempts to alleviate families' stress and improve participants' socioemotional adjustment, family functioning, and sibling relationships. The 15-week program includes group discussions about developmental disabilities, recreational activities, and homework assistance. At meetings every weekday from 3 p.m. to 5:30 p.m. at a community center, the children are separated into teams of 15 for group discussions and homework assistance. The whole group participates in the recreational activities together.

Emerging Patterns of Findings

While seven evaluations clearly do not provide enough data to conduct a systematic, empirical meta-analysis of patterns of outcomes across studies, this sample provides enough information to look descriptively at the emerging patterns of outcomes. In this Snapshot, we summarize evaluation results within each of four outcome domains: academics, family, prevention, and youth development. Here, “positive effects” or “negative effects” refer to estimated effects that reached conventional levels of statistical significance (i.e., p < .05). The Appendix shows all program impact findings estimated by comparing treatment group and control group youth, arranged by program.

Academics
Four of the studies (CASP, GHP, HASP, and HSTP) examined academic outcomes. Three of these four found at least some positive effects of the program on academic outcomes. The evaluation of HSTP found positive effects on five measures of academic achievement (e.g., word recognition, spelling) immediately after the program ended and on four of these five measures 1 year after the program ended. HASP found positive effects for reductions in poor classroom behavior, the only academic outcome examined. CASP, primarily a sports and recreation program, demonstrated improvements in educational aspirations and time spent on homework, though it found no effects on GPA, achievement scores, or school attendance.

In contrast, GHP found no overall effects on academic outcomes (academic performance, perception of self as a student, or homework completion). It did, however, find a general pattern of positive effects on these outcomes for English Language Learners (though it also found some negative effects for English Proficient students).

Family
Three programs (GFK, GHP, and SCDDASSP) examined family outcomes and were generally less successful in affecting these outcomes. The evaluation of GFK found some positive effects of the program on three measures of parents' utilization of healthy food practices, but it found no results for a host of other outcomes around healthy eating and physical activity motivations, beliefs, and practices (though the evaluation relied on a small sample and many of the outcomes were in the desired direction). Neither GHP nor SCDDASSP found benefits for their respective family indicators, which were parent–child involvement for GHP and family functioning, sibling relationship quality, and family social support for SCDDASSP.

Prevention
Only one program (GFK) examined prevention-related outcomes, and these outcomes were related to obesity prevention. The outcomes were similar to the program's family findings. Some positive effects were found for girls' healthy choice behavioral intentions, diet knowledge, and preferences for physical activity, but on other measures (e.g., dietary intake, calories from fat, physical activity), no positive effects were found. On two measures (unhealthy weight concern and preference for large body size) the program found negative effects (i.e., program girls were more likely to exhibit unhealthy weight concerns and preferences for larger body sizes than control group girls).

Youth Development
Four of the five programs that studied youth development outcomes (CASP, GG, HASP, and SCDDASSP) reported positive effects. CASP found positive effects on time spent in strength training activities. GG found improvements in body image, assertiveness, self-efficacy, self-liking/competence, and healthy attitudes toward physical attractiveness (e.g., disagreement with statements such as “The way I look is more important than the way I act”). HASP found improvements in self-concept, which was the one youth development indicator that it examined. SCDDASP found positive effects on various types of self-esteem, lower depression, lower anxiety, and various types of social support.

Four of the five programs (CASP, GG, GHP, and SCDDASSP) found some null results as well. CASP found no effects for time in sports and artistic activities, behavior, television watching, or time spent in self-care. GG found no effects for hopelessness, availability of help sources, or friendship esteem. GHP found no effects on their youth development indicators, which were measures of social skills and social support. Lastly, SCDDASSP found no effects on body image self-esteem or sports/athletic self-esteem.

Addressing the Challenges

Small-scale randomized experiments face a number of methodological challenges, some particular to their smaller scale and others common to all randomized experiments. When programs are small in nature—that is, when they operate at the local level and/or at few sites—sample sizes used to estimate impacts are consequently smaller as well, which may compromise the ability of these studies to detect meaningful effects. When randomization is conducted on smaller samples of participants, the likelihood that randomization will produce two groups that are exactly alike on relevant dimensions decreases.

In addition, as in any randomized study, evaluators must confront the potential ethical complications of denying services to respondents assigned to the control group. Finally, while random assignment is useful in estimating patterns of outcomes, it is not always useful in understanding why the program did or did not lead to a certain pattern of outcomes. This section highlights how the evaluations in this review carried out their experimental designs, with particular attention to evaluators' strategies for confronting the above-mentioned challenges.

Dealing with small sample sizes. The total sample sizes used to estimate program impacts in this sample of studies ranged from a maximum of 227 to as small as 34. The average sample of initial respondents was 116 youth, who were then randomly assigned to either a treatment group or a control group, usually on a one-to-one basis. This may have limited the ability of these evaluations to detect moderate-sized effects. For example, evaluators in the GFK study reported that because of the small sample size, they were unable to test for differences in what would have been the primary outcome of interest, body mass index (BMI).

One helpful strategy for assessing the potential problems stemming from these sample sizes was employed in the CASP evaluation. Here, the evaluator reported the results of a power analysis, which is designed to reveal the minimum size of effects that the research design would be able to detect. For example, this study would not have been able to detect effects of less than .3 GPA units (on a 4.0 scale) or less than 3 days of school attendance.

Ensuring true randomization. When randomizing subjects in smaller scale studies, another challenge is ensuring that the randomization “works”—that is, ensuring that the randomization leads to two equal groups before the treatment group undergoes their “treatment.” One strategy that three of the studies employed was using some form of stratification before randomly assigning youth to their group. This means that evaluators would first break down their sample into different groups (e.g., males and females) and then randomly assign treatment within these groups. This was done to ensure that the random assignment process would lead the two groups to be balanced or equivalent before the beginning of the treatment.

Another useful evaluation strategy was testing for equivalency between the two groups after randomization. Five of the seven studies reported testing for equivalency between the treatment and comparison groups to ensure that the two groups were similar on relevant pretreatment factors.

Managing the ethical challenges of control groups. Randomized studies offering potentially valuable services to participants in the treatment group inevitably confront the question of how to be fair to participants in the control group. The majority of studies reported some effort to grapple with this issue. One study (CASP) pointed out that it was ethical to simply deny services to the control group because there were a limited number of available slots in the program and the randomization process was therefore a fair method of allocating services.

Two other studies (GG and SCDDASSP) indicated that control group youth were placed on a wait list to enroll in the program at a later date, while a third (GFK) offered the control group alternative after school services unrelated to the program's goals. One or more of these strategies may have also been employed in the other three studies, but this information was not available in the reports. One of the studies (GHP), however, did examine what control group youth were doing with their after school time, finding that over a third were engaged in some other kind of after school activity similar to the program. This strategy is useful in understanding program impacts, as it allows the evaluators to see how groups truly differ on the treatment variable.

Understanding the story behind the results. Though experimental designs are quite useful in confidently estimating program impacts, they are less useful in understanding why the program may or may not have produced the outcomes that it did. One useful strategy for dealing with this challenge was collecting supplemental data, such as qualitative interview and observation data, to flesh out the understanding of outcomes. Three of the studies collected some sort of qualitative data on the programs' implementation to help understand the pattern of outcomes that they found in the experimental analysis. For example, the CASP program conducted site observations and interviews to understand what parts of the program were working well, why youth participated or stopped participating, and what weaknesses and challenges kept the program from serving youth better.

Implications for the Field

Small-scale randomized experiments can be crucial building blocks in understanding whether and how after school programs can achieve results for their participants and their families. While not always feasible or appropriate,3 randomized trials can provide important evidence for the potential effectiveness of after school programs in improving youth's lives. This Snapshot has examined the types of small-scale after school programs that have been evaluated with experimental designs, the challenges and evaluation strategies associated with these small-scale experimental evaluations, and the results that have emerged across this set of evaluations.

Several key findings and lessons emerge from the seven small-scale evaluations reviewed in this Snapshot. These lessons can be useful both for those conducting after school program evaluations and for the field in general, as the evidence base for the importance of nonschool hour opportunities for youth continue to grow.

• First, there was great diversity in the types of programs evaluated, evaluation strategies, and the types of outcomes examined. This highlights the fact that it is important for evaluators of after school programs to pay close attention to programs' underlying theories of change4 and to collect information on those outcomes that programs are intended to affect and for which they are held responsible.

• Second, small-scale randomized studies entail a number of challenges for evaluators, including small samples, services provided to control groups, effective randomization, and understanding of the mechanisms that produce outcomes. This set of evaluations adopted a number of creative strategies for dealing with challenges that can be useful to others in the field.

• Third, the evaluations used rigorous methods to demonstrate that after school programs can produce benefits for participants. This set of programs was most successful in producing positive youth development and academic outcomes but less successful in family- and prevention-related outcomes.

Experimental evaluations will be important in the continuing development of the after school field in order to build the knowledge base about effective programming and to cement the case for continued financial and resource investments. As this research accumulates, it is critical to synthesize whether, how, and why programs of diverse sizes, characteristics, and goals are producing the types of outcomes that they target. This Snapshot provides an initial review of the small body of small-scale experimental studies that are currently available, with an eye toward laying a foundation for synthesizing future research and evaluation.

Notes

1 Cook, T. D. (2002). Randomized experiments in educational policy research: A critical examination of the reasons the educational evaluation community has offered for not doing them. Educational Evaluation and Policy Analysis, 24(3), 175–199.
2 U.S. Department of Education, Office of the Under Secretary. (2003). When schools stay open late: The national evaluation of the 21st-Century Learning Centers program, first year findings. Washington, DC: Author.
3 Weiss, A. R. (2005). What is the Campbell Collaboration and how is it helping to identify “what works”? The Evaluation Exchange, 11(2), 9.
4 C. S. Mott Foundation Committee on After-School Research and Practice. Moving Towards Success: Framework for After-School Programs. Washington, DC: Collaborative Communications Group, 2005. Available at http://www.publicengagement.
com/Framework.

Acknowledgements

This Snapshot is based on a review of the Harvard Family Research Project Out-of-School Time Program Evaluation Bibliography, which is supported by grants from the C. S. Mott Foundation and the W. K. Kellogg Foundation. The principal investigator for this study is Dr. Heather B. Weiss. The author wishes to thank Priscilla Little, Suzanne Bouffard, and Julia Coffman for their thoughtful review.

Appendix: Programs and Findings Included in this Snapshot

Girlfriends for KEEPS (Keys to Eating, Exercising, Playing, and Sharing)
Initiated in 2000, this is an obesity prevention program for low-income African American elementary school girls in Minneapolis, Minnesota.

Story, M., Sherwood, N. E., Himes, J. H., Davis, M., Jacobs, Jr., D. R., Cartwright, Y., et al. (2003). An after-school obesity prevention program for African-American girls: The Minnesota GEMS Pilot Study [Supplement 1]. Ethnicity & Disease, 13(1), 54–64.

Family Findings At the 12-week follow-up, parents of treatment group girls reported significantly less availability of higher fat foods in their homes (p = .001), more low-fat food practices (p = .009), and lower energy intake from fat in their own diets (p = .03), compared to parents of control group girls. No significant between-group differences were observed for the other parent-reported diet and activity measures, although most measures changed in the direction expected for the treatment group parents.

Prevention Findings There were no significant differences in BMI at the 12-week follow-up between the treatment and control groups. There was a trend for waist circumference to be 1.4 cm higher in the treatment, compared to the control, group (p = .08) at posttest.

For dietary intake measures at posttest, treatment group girls had lower caloric intake, lower percent of calories derived from fat, and more servings of water per day compared to control group girls. Treatment group girls, however, had lower fruit and vegetable servings per day and higher sweetened beverage servings per day than control group girls. None of these differences were statistically significant.
Physical activity measures demonstrated consistently greater activity levels in the treatment group compared to the control group at posttest. Electronically monitored physical activity levels, minutes of moderate to vigorous physical activity between noon and 6 p.m., and self-report of usual physical activity all increased more among girls in the treatment group, though none of these differences reached statistical significance.

At posttest, treatment girls reported significantly higher scores on the healthy choice behavioral intentions (p = .001), diet knowledge (p = .001), and preferences for physical activity (p = .04) scales than did control girls. Treatment girls were also significantly more likely than control girls to report a preference for larger body size (p = .01) and were more likely to report engaging in both moderate (p = .004) and unhealthy behaviors related to weight concern (p = .04). No between-group differences in the prevalence of dieting were observed.

Cooke Middle School After School Recreation Program (CASP)
Founded in 1999 in an inner-city Philadelphia, Pennsylvania, middle school, this program offers activities designed to promote students' physical, emotional, and social well-being during the evenings.

Lauver, S. C. (2002). Assessing the benefits of an after-school program for urban youth: An impact and process evaluation. Philadelphia: Author.

Academic Findings Participation in CASP had no measurable impacts on academic performance or standardized test scores as measured by GPA and SAT-9 standardized test scores.

CASP participation showed no measurable impact on youth's school attendance. Treatment group youth demonstrated significantly higher levels of time spent on homework each week (p < .05) than control group youth.

Participation in CASP had a significant impact on youth's aspirations for further education (p < .01), as program students were significantly more likely to want to attend college or job training after high school.

Youth Development Findings There were no statistically significant program impacts found for time spent on artistic activities or in exercise or sports. There was a statistically significant program impact for time spent on strength training activities at least one hour per week (p < .01), with 80% of the treatment group spending time in these activities as compared to 61% of the control group.

There were no measurable program impacts found for treatment youth's in-school behavior. CASP participation was not found to be related to youth's time spent watching television or the amount of time they spent in self-care (time spent at home without an adult present).

The Siblings of Children with Developmental Disabilities After School Support Program (SCDDASSP)
Initiated in 1996, this after school program serves children from an East Coast inner city who have siblings with developmental disabilities such as mild or moderate mental retardation. By providing services for the nondisabled siblings, the program attempts to alleviate families' stress and improve participants' socioemotional adjustment, family functioning, and sibling relationships.

Phillips, R. S. C. (1999). Intervention with siblings of children with developmental disabilities from economically disadvantaged families. Families in Society: The Journal of Contemporary Human Services, 80(6), 569–577.

Family Findings No significant treatment effects were found for family functioning, family social support, or the quality of sibling relationships.

Youth Development Findings Results showed that children who participated in the program showed increased socioemotional adjustment compared to children in the control group (the latter group showed no improvements). Specifically, there were significant treatment effects found for the following measures of socioemotional adjustment: depression (p < .05), anxiety (p < .05), self-esteem/peers (p < .05), self-esteem/school (p < .01), self-esteem/family (p < .05), and self-esteem/global (p < .01). No significant treatment effects were found for self-esteem/body image or self-esteem/sports/athletics.

Results revealed a significant treatment effect for decreased sibling-related stress (p < .01), but no effects for parent-related or home life-related stress. There were significant treatment effects found for increased peer social support, school social support, and center staff social support (p < .01 for all).

The Gevirtz Homework Project (GHP)
Initiated in 1997, this project is an after school program in three public elementary schools in Santa Barbara, California. Its goal is to provide students with academic support and improve their academic achievement through assistance with homework and study skills.

Cosden, M., Morrison, G., Albanese, A. L., & Macias, S. (2001). Evaluation of the Gevirtz Homework Project: Final report. Santa Barbara, CA: Gevirtz Research Center.

Academic Findings No significant differences were found between the treatment and control groups for academic performance, monthly homework completion, study skills, or perception of self as a student.

There was a significant interaction between treatment/control group status and English Language Learner (ELL)/English Proficient (EP) status for homeroom teacher ratings of mean percentage of homework completion (p < .05). ELL treatment youth completed more homework than ELL control youth (86.9% vs. 80%), while EP treatment youth completed less homework than EP control youth (78.9% vs. 88.2%).

There was a significant interaction between treatment/control group status and ELL/EP status for report card ratings of mean effort in reading (p < .01) and math (p < .05). ELL treatment youth demonstrated more effort than ELL control youth. This pattern was also true for report card ratings of study skills (p < .05).

There was a significant interaction between treatment/control group status and ELL/EP status for teacher ratings of mean school functioning (p < .05). ELL treatment youth demonstrated better school functioning than ELL control youth, while EP control youth demonstrated better school functioning than EP treatment youth.

Family Findings No significant differences were found between the treatment and control group for parent–child involvement measures.

There was a significant interaction between treatment/control group status and ELL/EP status for parent report of providing parental supervision (p < .01). ELL treatment parents reported more parental supervision than ELL control parents, while EP control parents reported more parental supervision than EP treatment parents.

Youth Development Findings No significant differences were found between the treatment and control group for social skills or social support.

There was a significant interaction between treatment/control group status and ELL/EP status for report card ratings of social skills (p < .05) and teacher ratings of interpersonal skills (p < .05) and acting-out behavior (p<.01). ELL treatment youth demonstrated more positive scores on these ratings than ELL control youth, while EP control youth demonstrated more positive scores than EP treatment youth.

Go Grrrls (GG)
Developed in 1995 in Tucson, Arizona, this preventive after school intervention program focuses on the promotion of middle school girls' positive psychosocial development to help them navigate through early adolescence.

LeCroy, C. W. (2003). Experimental evaluation of “Go Grrrls.” Tucson, AZ: Author.

Youth Development Findings The intervention group reported significantly greater increases in body image (effect size = . 05, p < .008), assertiveness (effect size = .04, p < .01), positive attitudes regarding attractiveness (effect size = .08, p < .002), self-efficacy (effect size = .03, p < .03), and self-liking and competence (effect size = .06, p < .006) than the control group.

Hopelessness and help sources outcomes showed positive effects of the intervention (effect sizes ~ .02), but only at the .10 level of significance. No effect of the program was found for friendship self-esteem.

The Hispanic After School Program (HASP)
Begun circa 1979, this program is designed to promote positive ethno-cultural identity, school adjustment, and self-concepts of Puerto Rican children attending two elementary schools in a semirural town in Massachusetts.

Garza Fuentes, E., & LeCapitaine, J. E. (1990). The effects of a primary prevention program on Hispanic children. Education, 110(3), 298–303.

Academic Findings From pretest to posttest, the program group demonstrated a decrease in the mean number of maladaptive behaviors whereas the control group manifested an increase. Analysis indicated a significant (p < .05) difference between the program and control groups' posttest adjusted means. In particular, the program group made significant gains in improving their classroom behavior, including decreases in fighting, classroom disruption, restlessness, unhappiness, impulsivity, sickness, moodiness, and difficulties with learning. There were no significant gains on the other three behaviors in the scale.

Youth Development Findings Although both groups demonstrated increased mean global self-concept scores, the program group made significantly greater gains in self-concept than the control group (p < .05). Specifically, the program group manifested improved self-perceptions in the following areas: academic status, physical attributes, happiness, anxiety, and popularity.

Howard Street Tutoring Program (HSTP)
Begun in 1979 on the North Side of Chicago, Illinois, this program provided after school remedial reading instruction through one-on-one tutoring to second and third grade children who had fallen behind their peers.

Morris, D., Shaw, B., & Perney, J. (1990). Helping low readers in grades 2 and 3: An after-school volunteer tutoring program. The Elementary School Journal, 91(2), 133–150.

Academic Findings Significant (p < .05) differences were found between the treatment and control group gains on all measures except for timed and untimed word recognition. On all achievement measures during both school years, including timed word recognition, untimed word recognition, basal word recognition, basal passages, and two measures of spelling, the treatment group had greater gains from the pretest to the posttest than the control group.

When all 30 children who received treatment during 1986–1987 and 1987–1988 are compared to the 30 children who comprised the control groups over both years, the score gains from the pretest to the posttest favored the treatment group and are significant (p < .01) on four of the five achievement measures.

The basal passage reading score, a measure of oral reading success on a set of graded passages, provides the best assessment of children's reading “instructional level” with every 10-point gain corresponding to a grade level. On average, the 30 students who received tutoring had a gain on the basal passage reading score of 12.2 while the 30 students in the control group had an average gain of 6.6. In other words, tutored students advanced more than one grade level in reading during the school year while students who did not receive tutoring only advance two thirds of a grade level. This difference was statistically significant (p < .01).

There was greater variation in the basal passage reading scores of the tutored children than in the basal passage reading scores of the control group children. Not all tutored children advanced a full grade level during the school year, but 50% did as compared to only 20% of the control group. Also, only 23% of the tutored group made “limited progress,” defined as less than a five-point or half grade level gain in basal reading passage score, while a full 47% of the control group made “limited progress."” Finally, 34% of the tutored group made large gains in reading, more than 15 points on the basal passage reading score, while only 3% of control group students made similar gains.

All rights reserved. Information on this site may not be reproduced in whole or in part without written permission from Harvard Family Research Project.

© 2006 President and Fellows of Harvard College
Published by Harvard Family Research Project
3 Garden Street
Cambridge, MA 02138
Tel: 617-495-9108
Fax: 617-495-8594
Email: hfrp@gse.harvard.edu