Out-of-School Time Evaluation Snapshot

Engaging With Families in Out-of-School Time Learning

Number 4, April 2004

Erin Harris, Research Assistant, and Chris Wimer, Research Assistant

75KB Acrobat file

Help for Acrobat file

This Snapshot is also available in hard copy for free. Please go to our publications list to learn how to order it.

Harvard Family Research Project’s series of Out-of-School Time Evaluation Snapshots distills the wealth of information compiled in our Out-of-School Time Program Evaluation Database¹ into a single report. Each Snapshot examines a specific aspect of out-of-school time (OST) evaluation. This Snapshot provides an overview of how researchers are evaluating OST programs’ engagement with families.

Engaging with families is one of the many strategies that out-of-school time (OST) programs use to create quality, adult-supervised experiences for youth during nonschool hours. According to Weiss and Brigham (2003), strategies for engaging with families can embody three types of program goals:²

1. Support for children’s learning – This includes strategies such as helping families increase students’ positive development and academic performance; increasing parents’ engagement in their children’s education; and facilitating communication between families, schools, and OST programs concerning students’ learning and development in and out of the classroom.

2. Support to families – This includes both direct supports and services to family members (e.g., English as a second language classes, job skills training) as well as opportunities and efforts to improve family relationships (e.g., recreation and social events that include both children and families). This also includes more general efforts by programs to encourage both positive interactions between parents and children and parental engagement in activities with their children, within and outside the program.

3. General parent involvement – This includes enlisting parental support of program staff and goals. This could also entail active parental assistance with the program’s operation (e.g., acting as volunteers, assuming leadership or governance roles).

As these varied goals show, OST programs take a number of approaches to engaging with families. Many of these do not involve actual parent presence in the program. This is important because the rationale for many OST programs is to provide safe and enriching experiences for children while their parents are at work and cannot provide supervision. In fact, many evaluations note that programs provide benefits to families stemming from the provision of convenient child care that allow them to better balance work and family life. Despite this rationale, it remains important for OST programs to consider both how they bring families into the program setting and how they reach out to families to improve the lives of youth.

The remainder of this Snapshot uses information posted in our Out-of-School Time Program Evaluation Database as of March 2004 to examine how programs are collecting data on engaging with families. (See Appendix A for a list of the programs included in our review.) It identifies common evaluation questions and corresponding performance measures that programs use to assess their strategies for engaging with families.

How Do Evaluators Assess OST Programs’ Engagement With Families?
Evaluators of these programs rely on a number of data sources to measure engagement with families, the most common of which is survey and questionnaire data, followed closely by interview and focus group data. Data are most often collected from the parents themselves. School staff, program staff, and youth are also frequently interviewed and surveyed regarding programs’ engagement with families. Program documents are a data source for a number of evaluations, allowing evaluators to examine related policies and parental outreach materials. Finally, a few evaluations make use of site-visit observations to measure programs’ engagement with families.

How Do Programs Measure Engagement With Families?
The performance measures that programs use to measure their engagement with families fall into three categories: (1) type and frequency—how programs engage with families, (2) contextual factors—what factors affect programs’ engagement with families, and (3) effectiveness—how successful programs are in engaging with families, and whether this engagement leads to positive results. (See Appendix B for a complete list of evaluation findings by category.)

1. Type and Frequency – This set of measures describes the ways in which programs engage with students’ families and how frequently. There are three primary ways that programs engage with families:

   a. Activities or services that programs provide to support families – These include cultural and recreational events; classes in parenting, English as a second language, basic literacy, computer skills, and GED exam preparation; parent counseling and support groups; intergenerational activities; social service referrals; and food sent home with children.

   b. Activities or strategies to support communication and relationship building with parents – These include talking individually with parents about their child’s needs or interests; orienting parents to the program; initiating general information phone calls to parents; face-to-face contact when parents pick up children; and sending home flyers, newsletters, and bulletins.

   c. Services parents offer to programs – These include program and activity planning assistance and serving as paid staff, activity assistants, advisory board members, tutors, translators or interpreters, and volunteers.

2. Contextual Factors – Contextual factors provide background information that might help set the stage for family engagement in programs. Our review suggests that programs collect information on several contextual factors:

   a. Parental attitudes toward their children’s education or school – This includes how interested or concerned parents are with their child’s education, how much contact they have with teachers and schools, and their expectations for their child’s education.

   b. Program factors that strengthen or limit programs’ engagement with families – These include parental work requirements in the program and mandatory monthly parent meetings, inadequate family- or parent-oriented activities, and lack of support from the host school.

   c. Family factors that strengthen or limit programs’ engagement with families – These include parental interest or availability, lack of engagement by parents of adolescents, parents’ work schedules, parents’ language and cultural barriers, transportation, and families’ residence outside the school neighborhood.

3. Effectiveness – These measures include both the effectiveness of programs in engaging with families and the extent to which this engagement leads to positive results:

   a. Levels of family participation in programs – A slight majority of findings indicate generally low family involvement in the actual OST programming, although a number of findings suggest high involvement, and several reveal a level somewhere in between.

   b. Program success in communicating and building relationships with families – Evaluations generally indicate more success in this area, with the majority of findings indicating positive results, only a few mixed results, and no entirely negative findings. Many program evaluations note that the majority of parents are satisfied with the programs’ communication and accessibility, and parents often report feeling comfortable approaching program staff.

   c. Improvement in families’ engagement with their children’s school or education and/or improvement in their relationship with their children – Evaluation findings in these categories show a general pattern of positive results, with no completely negative findings, and several indicating no change. For example, program involvement is associated with better relationships between parents and children (fewer disagreements, less lying to parents, increased parent-child trust); increased family involvement in school related events, such as school open houses and parent-teacher meetings; and increased parental assistance with children’s homework and discussions about schoolwork.

   d. Value added to programs by engaging families – This type of finding is not commonly cited, but when mentioned generally indicates that this engagement contributes to better program implementation and outcomes. For example, in the evaluation of the Virtual Y programs in New York City, careful communication with families in certain programs is associated with better program outcomes.

Conclusion
Many OST program leaders believe that engaging with families can add value to their programming, and are interested in finding ways to improve such efforts. As this review indicates, programs are taking a variety of approaches to understanding how families fit into their programming, from the types of data they are collecting to what they hope to learn from the data. Examining strategies to engage with families and the effectiveness of these strategies will continue to be important for understanding OST programs’ implementation, quality, and outcomes.

Notes
1 Our database contains profiles of out-of-school time (OST) program evaluations, which are searchable on a wide range of criteria. It is available in the OST section of the HFRP website at www.gse.harvard.edu/hfrp/projects/afterschool/evaldatabase.html.

2 These goals are based on responses from OST programs (a cohort of 21st Century Community Learning Centers grantees) to an open-ended survey question about their primary family involvement goals.

Reference
Weiss, A. R., & Brigham, R. A. (2003). The family participation in after-school study. Boston, MA: Institute for Responsive Education.

Acknowledgements
This Snapshot is based on a review of the Harvard Family Research Project Out-of-School Time Program Evaluation Database, which is supported by grants from the C. S. Mott Foundation and the W. K. Kellogg Foundation. The principal investigator for this study is Dr. Heather B. Weiss. The authors wish to thank Julia Coffman, Holly Kreider, Priscilla Little, and M. Elena Lopez for their thoughtful review.

Appendix A: Out-of-School Time Programs Included in the Review

21st Century Community Learning Centers – District of Columbia
21st Century Community Learning Centers – national
21st Century Community Learning Centers – Owensboro, Kentucky Public Schools
The 3:00 Project®
After School Achievement Program
After School Education and Safety Program – Santa Ana, California
Austin Eastside Story After-School Program
Baltimore’s After School Strategy – YouthPlaces Initiative
Beacons Initiative – New York, New York
Big Brothers Big Sisters of America
Cap City Kids
Cooke Middle School After School Recreation Program
Cooperative Extension Service Youth-at-Risk School-Age Child Care Initiative
Extended-Service Schools Initiative
Fifth Dimension/University-Community Links
Fort Worth After School Program
Juvenile Mentoring Program
Los Angeles’ Better Educated Students for Tomorrow Program
New Orleans ADEPT Drug and Alcohol Community Prevention Project
Ohio Urban School Initiative School Age Child Care Project
San Diego’s “6 to 6” Extended School Day Program
Say Yes First
School-to-Jobs Programme
SECME Raising Interest in Science & Engineering
The After-School Corporation (TASC)
Virtual Y

Appendix B: Evaluation Findings That Assess Engagement With Families

This appendix provides a complete listing of all findings related to engaging with families from evaluations in the Out-of-School Time Program Evaluation Database as of March 2004. These findings are from 34 evaluations of the 26 programs listed in Appendix A. The findings are organized into the same categories outlined in the Snapshot. Each bullet point describes a separate finding and, because this list is a compilation of findings from multiple evaluations, some findings may be repetitive or appear to be incongruous with each other.

1. Type and Frequency: How programs engage with families

a. Activities or services that programs provide to support families

Support Services
• Site coordinators reach out to parents through referrals to local agencies or organizations for assistance or information.
• Some sites send food home with children for families.
• Some sites provide referrals to parents and families for other social services.
• 67% of program sites provide specific support to parents, either through parent counseling or parent support groups.
• In 16 sites (40%), programs provide support to families struggling with social and emotional problems through the inclusion of preventive service programs.
• Program sites in neighborhoods with high concentrations of immigrants develop services and activities specifically tailored to these new residents.
• Efforts to involve families include referring parents to other services offered by their sponsoring agency.

Events
• Site coordinators reach out to parents by sponsoring cultural or recreational events and holding meetings where representatives of local agencies present information.
• More than half of site coordinators report holding meetings or events to which parents are invited at least a few times per month. Staff report engaging in this activity less often, with 66% reporting rarely or never holding such meetings or events.
• Three areas that attract substantial numbers of adult participants are educational activities, sports and recreational activities, and culturally specific programming.
• Sites conduct outreach to some parents and families through family nights, concerts, and dances.
• 76% of program sites report holding intergenerational activities at least several times a year, and 24% report including these activities on a continuing basis.
• Methods for increasing parent involvement include inviting parents to after school events.
• Efforts to involve families include hosting special events for families and holding monthly workshops for parents.

Classes
• Site coordinators reach out to parents by offering classes in parenting, English as a second language, computer skills, or GED exam preparation.
• Adults report participating in sports and physical fitness activities, basic education, English-language instruction, GED preparation, and computer instruction at the programs.
• Adult education opportunities, available at 87% of the sites, are offered in cooperation with the city Board of Education, including GED preparatory classes and basic literacy and English-language classes.
• Methods for increasing parent involvement include offering adult education classes.

b. Activities or strategies to support communication and relationship building with parents

Verbal Communication and Meetings
• Site coordinators reach out to parents by talking individually with parents about their child’s needs or interests.
• When asked about reaching out to parents, 48% of coordinators say they talked to at least 16 parents individually in the last month about their child’s needs or interests.
• More than half of site coordinators report that they spend time talking on the phone with parents daily. More than two-thirds of staff report doing this rarely or never.
• Almost two-thirds of site coordinators report holding informal conversations with parents on-site most days. Project staff report that this is the most common form of communication with parents: 32% of staff say that they hold such conversations daily, 21% hold them 1–2 times weekly, 25% hold them a few times a month, and 22% hold them rarely or never.
• Site coordinators’ contact with parents seems to be regular, frequent, and conducted through varied strategies, although parental contact in the first year of a program seems to consist primarily of one-time or occasional activities that occur on an informal basis, rather than a more strategic approach.
• 13% of site coordinators report holding conferences with one or more parents almost daily, while approximately another 20% hold them once or twice weekly. About 20% of site coordinators and 76% of staff report rarely or never holding conferences with parents.
• Parent involvement mostly consists of coordinators orienting parents to the program, initiating general information phone calls to parents, and having some face-to-face contact when parents pick up their children.

Written Information
• The in-depth study sites reveal a range of communication strategies with parents, from a written system for documenting children’s daily experiences, to a checklist on which staff indicate homework completion and any behavior problems, to no policies at all for parental communication.
• Site coordinators’ efforts to involve families include sending home flyers, newsletters, and monthly calendars.
• About two-thirds of site coordinators report sending materials home a few times a month. Staff report doing this less often, with 43% sending home materials rarely or never.

c. Services parents offer to programs

Paid Staff
• Parents work as staff in the programs.
• 59% of site coordinators report that parents serve as paid staff.
• Parents who directly assist the projects most often serve as paid staff.
• Efforts to involve families include hiring parents as assistants or counselors.
• In exceptional situations, parents in specialty professions serve as instructors for some activities.

Volunteers and Unpaid Staff
• Some parents volunteer in the programs.
• Efforts to involve families include recruiting parents as program volunteers.
• Parents who directly assist the projects were activity assistants, tutors, and language translators or interpreters.
• 34% of site coordinators report that parents serve as activity assistants.

Decision-Making or Organizing Roles
• Some sites have parents planning what activities would be offered.
• Efforts to involve families include having parents as members of project leadership committees and recruiting parents to organize year-end culminating events and family nights.
• Parents who directly assist the projects are members of the site’s advisory board.

2. Contextual Factors: What factors affect programs’ engagement with families

a. Parental attitudes toward their children’s education or school

Concern or Interest in Their Child’s Education
• 25% of the participants’ parents have no concern for their child’s education.
• Teachers view participants’ parents as more concerned than nonparticipants’ parents before the program began—45% of participants’ parents are very concerned compared to only 26% of nonparticipants’ parents.
• 27% of participants’ parents say that they would like to have little or no contact with their child’s school.
• 21% of participants’ parents say they check their child’s homework once a week or less.

Contact and Communication With the School
• 34% teachers say they have “quite a bit” of contact with participants’ parents compared to 22% for nonparticipants’ parents. Similarly, 16% of teachers say they have a “great deal of contact” with participants’ parents compared to 7% for nonparticipants’ parents.
• Teachers report that 18% of participants’ parents have little to no contact with the school.

Expectations for Child’s Education
• In response to a question about the highest grade level parents believed their child would complete in school, 20% estimated twelfth grade, 51% estimated four years of college, and 19% estimated four or more years past a four-year college degree.

b. Program factors that strengthen or limit programs’ engagement with families

• Site coordinators perceive lack of support from the host school as a factor limiting outreach to families.
• Informants partially attribute parents’ sparse activity participation to a lack of adequate family- and parent-oriented activities.
• Parental work requirements and mandatory monthly parent meetings seem, according to the evaluator, to create a sense of responsibility for and ownership of the program. 
• Lack of parental involvement in joint program activities with their children is attributed to several reasons, including offering few parent activities.

c. Family factors that strengthen or limit programs’ engagement with families

• Site coordinators perceive the following factors to limit outreach to families: lack of engagement by parents of adolescents, parents’ work schedules, parents’ language and cultural barriers, and families’ residence outside the school neighborhood.
• Informants partially attributed parents’ sparse activity participation to a lack of parental interest or availability.
• Barriers to parent involvement noted by key informants are: financial, work, and transportation struggles; communication barriers (e.g., no phone); and issues related to low parent interest.
• Lack of parental involvement in joint program activities with their children is attributed to several reasons, including parents’ work schedules interfering with after school programming and sometimes transportation barriers.
• The majority of parents state that they wish to be involved in planning and other after school activities, though three-fourths have never been involved in such activities.
• Significantly more parents than not say that they would be willing to help out with the program the following year.

3. Effectiveness: How successful programs are in engaging with families, and whether this engagement leads to positive results

a. Levels of family participation in programs

High Involvement
• One center initiated a relatively successful weekly family night.
• Special program-sponsored family and community events attract large groups across generations and cultures.
• Parent participation at Family Days, held at the end of each four-week summer session, was almost 100%, i.e., virtually all of the participants attended, along with parents and other family members.
• According to survey data, projects achieve the highest levels of parent participation at special events hosted by the after school project, with 54% of site coordinators reporting that at least half of the parents typically attend special events.
• The independent assessments of three centers reveal extremely positive ratings in the area of family involvement.

Medium or Neutral Involvement
• About one-quarter of parents report that they have been involved in other program-sponsored activities.
• Some sites have more involvement in family activities than others.
• The program is rated 3 on a scale of 0 (program does not exhibit characteristic) to 4 (program exhibits the characteristic a great deal) in the involvement of parents in the design, operation, and improvement of the program.

Low Involvement
• Parent involvement at many in-depth study sites continues to be low; however, sites are increasingly exerting efforts to involve and inform parents.
• Parent involvement is relatively low compared to other program components. Most parents report peripheral involvement.
• All coordinators describe parent and family involvement as somewhat limited and an area for improvement.
• Parent participation in Engineering Days (a forum for parents to ask questions related to promoting their daughters’ interest in science, mathematics, and technology, while also working together on hands-on design challenges) is low.
• Parental input is sought through advisory committees, but the evaluators recommend that more effort be made to engage parents in the program planning process at each site.
• Assistant principals and students report that parents are not yet very involved in the after school programs, but several assistant principals report having plans to engage more parents.
• When asked how the program tried to involve them, parents most often respond that their involvement is peripheral. Parents are typically kept informed (via phone calls, materials sent home, and face-to-face contact) of the program’s activities and their child’s progress, but are less often involved in joint activities with their children.

b. Program success in communicating and building relationships with families

Generally Successful
• Several parents note that the program has an open-door policy and all parents feel comfortable talking with program staff.
• All but one of the parents interviewed had met the coordinator or other staff in person.
• Parents and staff communicate regularly.
• All parents interviewed are satisfied with their levels of interaction, describing the program as “very accessible.”
• Staff and administrators are sensitive to the children and families with whom they work—they try not to judge and they make efforts to build strong relationships with parents.
• Lines of communication with parents are clear and parents are informed of all accidents and incidents.
• Site supervisors state that they know most, if not all, parents.
• Staff engage in developing relationships with parents.
• Of youth survey respondents, 68% indicate their parents would go to a program staff member for help if they needed to.
• Of parent survey respondents, 93% state they would go to a program staff member for help with their child.
• Parent complaints appear to be dealt with promptly, documented if they are unresolved after the initial contact, and treated as important.
• There are few parent complaints and prior complaints appear to have been resolved according to documented procedures.
• When asked what they think the purpose of the program is, most parents give responses that indicate that they have a realistic awareness of the program’s purpose.
• More than 95% of caregivers agree that they are well informed about program activities.
• Of the parents surveyed, 80% agree that the program staff keep them informed about their children’s successes and difficulties.
• 92% of parents surveyed feel free to visit the program any time.
• 82% of parents surveyed feel free to share their ideas with program staff.
• Communication between the program staff and the parents, teachers, and advisory council members improved compared to the previous year.
• The majority of parents are satisfied with the flow of information from the programs.
• Nearly 95% of parents report that program staff keep them informed of their child’s successes and difficulties.
• Approximately 90% of parents report that they feel free to visit the project whenever they can and to share their ideas about the program and its policies and they feel they are kept informed about schedules and activities and about their child’s successes and difficulties.

Mixed
• Parents describe different levels of program engagement by coordinators and center staff.
• Most parents (71%) rate their communication with the program as “excellent,” although some report that it is weak.
• A quarter of parents say that they do not know much about the program’s activities. Organizational support for inclusion of youth and parents in program planning is mixed.

c. Improvement in families’ engagement with their children’s school or education and/or improvement in their relationship with their children

Parent Engagement With Their Child’s School or Education
Improvements
• At the middle school level, program participation is associated with increased parent involvement at their child’s school. Parents of program participants are more likely to volunteer at their child’s school and attend open houses or parent-teacher organization meetings three or more times per year.
• For middle school students, increases in involvement for parents in two-parent families are larger than for parents in single-parent families. Participation led to a 14% increase in parents from two-parent households attending open houses, but only a 6% increase for single parents (p < .01). High-attendance programs (programs in which the average participant attended more than 40 days during the school year) have a larger impact than low-attendance programs (fewer than 20 days) and medium-attendance programs (20–40 days) on parent attendance at parent-teacher organization meetings (14.6 percentage points vs. 0 to 4 percentage points, p < .05).
• Centers serving elementary students increased the percentage of parents helping their child with homework at least three times in the last week, asking their child about classwork, and attending after school events.
• 28% of the surveyed principals note that they have parents whose involvement in the school had increased as a result of their child’s participation in the program. The principals collectively list 40 such parents.
• 36% of principals report that the program increased parents’ attendance at parent-teacher conferences.
• 45% of principals report that the program increased parents’ attendance at school events by a “great extent” or to “some extent.”
• Almost two-thirds of parents say that their contact with their children’s school increased as a result of the after school program.
• Participants’ parents are more likely than nonparticipants’ parents to be pleased with their child’s school—26% of participants’ parents are only somewhat or not very pleased compared to 41% of nonparticipants’ parents.
• Parents cite the program’s workshops and counseling for helping them learn to communicate better with their children and their children’s teachers.

Neutral or Mixed Results
• Elementary school centers did not affect several indicators of parent involvement, such as whether parents check that their child had completed homework, attend school events such as open houses and parent-teacher organization meetings, or volunteer in the school.

Parents’ Relationships With Their Children
Improvements
• Parent volunteers report that the opportunity to volunteer contributed to their feeling closer to their children and more connected to the neighborhood.
• Participant students score significantly higher than comparison students on measures of communication and involvement with family members and other adults (e.g., talking with parents, enjoying doing things with their family, helping at home; means = 3.60 for participant students and 3.41 for comparison; p < .05, effect size = .26).
• Participants report better relationships with parents than members of the control group by the end of the study period, due primarily to a higher level of trust in their parents. This effect was strongest for white males in the treatment group.
• At the conclusion of the study, the treatment group report lying to their parents 37% less than the control group.
• 61.6% of mentees and 40.2% of mentors indicate that the relationship helps the mentee get along with his or her family.
• Two participants were given their own computers by their mothers during the year in recognition of their intense interest in computers and their dedication to the after school activities.
• Parents participating in interviews or completing surveys feel the programs have positive impacts on their families.
• Participant students have significantly fewer disagreements or arguments with their parents than comparison students (means = 3.04 for participant students and 3.25 for comparison; p < .08, effect size = –.22).
• Of parents surveyed, 74% say the program helps their child get along with family members.
• 86% of parents surveyed say the program helps them to better appreciate their child’s talents.
• 95% of parents indicate that the food that the program sends home with their children helps them support their families.

Neutral or Mixed
• No statistically significant differences were found between the treatment and control groups on measures of communication with their parents.
• There were no significant differences between program and comparison students in parental monitoring.
• There were no significant relationships between program attendance and parents’ involvement with their children.

d. Value added to programs by engaging families

• Evaluators identify parental involvement as a “best practice.”
• Careful communication with both the home and host school (as rated by parents) is associated with better program outcomes.
• Many key informants note that lack of parental involvement is a program implementation barrier.

All rights reserved. Information on this site may not be reproduced in whole or in part without written permission from Harvard Family Research Project.

© 2006 President and Fellows of Harvard College
Published by Harvard Family Research Project
3 Garden Street
Cambridge, MA 02138
Tel: 617-495-9108
Fax: 617-495-8594
Email: hfrp@gse.harvard.edu