 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
 |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
There are no more hard copies of this brief available, but feel free to print out the Acrobat version through the link above. Heading Index:
Harvard Family Research Project’s (HFRP) Issues and Opportunities in Out-of-School Time Evaluation briefs are short, user-friendly documents that highlight current research and evaluation work in the out-of-school time field. These documents draw on HFRP’s research work in out-of-school time in order to provide practitioners, funders, evaluators, and policymakers with information to help inform their work. This fourth brief, Beyond the Head Count: Evaluating Family Involvement in Out-of-School Time, offers an overview of how out-of-school time programs involve families and how programs can evaluate family involvement. Why Consider Family Involvement in Out-of-School Time Programming?Engaging families is one of the many strategies that out-of-school time (OST) programs use to create quality, adult-supervised experiences for children ages five through nineteen during non-school hours. Therefore, like many OST program components, it is critical that the field build a knowledge base to understand how families are involved in their children’s out-of-school time, and how that involvement influences children’s development. Evaluating family involvement in OST is an effective means to build this knowledge base, thus enabling OST programs to understand and improve their family involvement strategies and services, while expanding opportunities for families to be together. Both the potential gains from evaluating family involvement and the increasing demand from the U.S. Department of Education, other funders, and the public to assess program impact and to use data for program improvement, render evaluation of OST programming, including family involvement, essential to ensure quality programming and sustainability of public and financial support.1 This Issues and Opportunities in Out-of-School Time Evaluation brief draws on evaluation findings from the Harvard Family Research Project’s Out-of-School Time Evaluation Database2 and key informant interviews and email correspondence to define and propose a framework for understanding family involvement in OST. The brief then examines ways for OST programs to evaluate their own family involvement strategies and practices. HFRP’s Out-of-School Time Evaluation Database is a collection of evaluations of both large and small out-of-school time programs and initiatives. Evaluations included in the database meet the following three criteria: 1) the evaluated program/initiative operates during out-of-school time, 2) the evaluation aims to answer a specific evaluation question or set of questions about a specific program/initiative, 3) and the evaluated program/initiative serves children between the ages of five and nineteen. Of the 21 profiles currently in the database, 6 reported family involvement findings. To augment the database information, HFRP conducted 15 key informant interviews with OST program directors and evaluators across the nation. Additionally, HFRP staff monitored the School Aged Child Care Issues and Concerns listserv (SAC-L)3 and Promising Practices in Afterschool listserv (PPAS)4 and elicited responses from over 40 OST professionals regarding their family involvement practices (see Appendix A for the list of questions). This brief brings together information from these sources to answer the following questions:
How Do Out-of-School Time Programs Involve Families?Out-of-school time programs that involve families tend to do so in a variety of ways. Our exploratory research suggests that these programs may create opportunities for families to:
These four dimensions comprise our definition of family involvement in OST. The following section of this brief establishes these four conceptual dimensions and illustrates their implementation with examples from OST programs from across the nation. These four dimensions can be viewed as a dynamic template for OST programs to use to implement family involvement. Further, these four dimensions can serve as a basis for evaluating family involvement in OST programs.
Definition From the Field
2. Shared Out-of-School Time Experiences Definition From the Field
Definition From the Field
Through input and collaboration in the planning process, families ensure quality programming for their children and serve as leaders in the larger community.
Definition From the Field As part of the partnership’s efforts to work more closely with schools, the Wyman Center family coordinator has developed a working relationship with a neighborhood elementary school. Touching base with the school frequently, she advises the school leaders on how to execute outreach efforts and what to do to engage parents more actively in their children’s education. Conversely, she reports back to the community with explanations of school policies and reform efforts. While this role is largely informal, it is evident that the partnership is leveraging its position in the community to serve as a link between families and schools.
How Are Out-of-School Time Programs Evaluating Family Involvement?Few resources have systematically compiled information about family involvement in OST programs. However, HFRP’s Out-of-School Time Evaluation Database provides information, in an accessible way, about evaluations of both large and small out-of-school time programs and initiatives (available on the Web at www.gse.harvard.edu/hfrp/projects/afterschool/evaldatabase.html). Each profile in the database contains an overview of the out-of-school time program or initiative that was evaluated, as well as detailed information about each evaluation report produced about that program. Six of the OST programs included in the database looked at family involvement in OST as part of their overall evaluations. The table below highlights these six programs and the performance measures and data sources employed to evaluate family involvement practices. Each of these OST program evaluations also examined many other aspects of program functioning and impact.
Types of Programs Evaluating Family Involvement The six programs in the Harvard Family Research Project Out-of-School Time Evaluation Database that evaluated family involvement are:
These six programs, with varied size, scope, and program mission, represent a range of OST approaches. Some initiatives have just 350 participants per year, as in the Austin Eastside Afterschool Program, while others are large, such as New York City Beacons Initiative, which serves 76,000 youth and 33,000 adults annually. Four of the programs are local in scope, but TASC operates throughout the state of New York and the Juvenile Mentoring Program serves youth nationwide. LA’s BEST has been in operation since 1988 while YouthPlaces was founded in 1999. The six programs have a variety of missions, including increasing the quality and quantity of OST opportunities (YouthPlaces and TASC), broadening supports for youth and families (NYC Beacons), and reducing juvenile delinquency and enhancing academic achievement (JUMP). All of the OST evaluations that examined family involvement had a non-experimental design6 component. Additionally, two of the six employed a quasi-experimental design, in which program participants were compared to non-participants who attended the same schools. However, the two programs that used a quasi-experimental design did not use it for an analysis of the impacts of family involvement, but rather focused on other OST program impacts. This distribution of designs is similar to all OST program evaluations: most evaluations focus on how the program is being implemented and examine participants’ experiences (formative evaluation) rather than on the impacts that the program is having on participants (summative evaluation).
Performance measures allow a program to quantify the services or interventions offered in the areas of activities, efficiency, capacity, or quality. A program acting alone can affect performance measures, of which there are two types. Measures of effort are the direct outputs of program strategies and activities. Measures of effect are changes in the target population that come about as a result of program strategies and activities. For example, a measure of effort for family involvement may be the number of families attending a family picnic; a measure of effect may be feelings of closeness between parents and children as a result of the picnic.7 It is important that program objectives drive performance measure development so that the data will be relevant to program stakeholders. The evaluations that examined family involvement used a wide array of performance measures to do so. Performance measures fell into the following categories:
As this broad selection of performance measures indicates, each evaluation included in the HFRP database that assessed family involvement did so in a unique way. The performance measures that evaluators used to evaluate family involvement were a reflection of the mission of the OST program, the program’s philosophy on family involvement, and the strategies and activities the program employed to engage families. The majority of the performance measures that OST evaluations relied on to measure family involvement were descriptive. In other words, they measured what the programs were doing to engage families and how this involvement was implemented. These six programs relied on a number of data sources to measure family involvement, but the most common source was interview data. In particular, evaluators interviewed parents, program staff, youth, and principals, in that order of frequency, to gain a better understanding of family involvement practices and experiences. Program documents were also a data source for some evaluations, allowing evaluators to examine family involvement policies and parental outreach materials. Some evaluations made use of survey data to gauge family involvement, but far fewer than the OST evaluations in the database that did not assess family involvement.9 What Can Out-of-School Time Programs Do to Evaluate Family Involvement?Evaluation allows for a systematic assessment of family involvement practices that will benefit all programs striving to engage families. OST programs can use evaluation to learn what family involvement activities they are currently engaged in, whether those activities meet the needs of program stakeholders, what can be done to improve family involvement practice, and what the outcomes are for families and children of the activities in each of the four dimensions of family involvement. However, there is no one best way to evaluate family involvement in OST. This review of the evaluation of family involvement in out-of-school time programs demonstrates that programs are using a wide array of evaluation designs, methods, and data sources to document and assess the level and type of involvement that families have in their children’s OST programming. OST programs select the evaluation approach that enables them to answer the most pressing questions, based on the information needed to make program improvements and meet funders’ requirements. For example, an OST program whose mission includes a goal to involve families in program governance might assess the degree to which parents participate in decision-making meetings, provide feedback about program implementation, or help interview staff. Similarly, a program aiming to strengthen family-school-youth relationships might interview school staff, family, and youth to determine the degree of continuity these stakeholders perceive among the home, school, and OST program venues. Despite the variation in how OST programs are evaluating family involvement practices, there are some basic evaluation approaches that can help programs collect information to better understand how to serve and engage the families of the youth that participate in their programs. While a program’s information needs will determine the evaluation approach it chooses, thinking carefully about each of the four dimensions of family involvement allows the program to broaden its scope of inquiry and study those aspects of family involvement that are most relevant to its programming. “Need” can be defined as “the gap between the problem and the existing efforts, resources, and programs to deal with the need” (Linney & Wandersman, 1991). When assessing needs, a program must acknowledge the services already available and identify those that could potentially be of help if provided to the youth and families participating in the OST program. It is equally important to identify the strengths of the families and communities the program serves, and assess the ways in which these strengths can be capitalized on for maximum benefit to program participants. A needs assessment to better understand the needs of families could inform four areas:
OST programs can use several methods to carry out a needs assessment, including interviewing and surveying families about how they want to be involved in their youth’s non-school time. Programs can also interview and survey other community members, businesses, cultural organizations, and religious associations. Further, other youth-serving organizations may provide valuable information about how they engage families and what they perceive the community needs. Although needs assessments are commonly associated with programs just starting to serve youth and their families, they can be conducted at any time in a program’s development when staff members want to step back and assess the evolving needs and capabilities of participants and their families. Families can also be involved in continuous assessment and improvement of program offerings. OST programs may document their services to gather valuable information about the frequency, content, and quality of program activities offered in each of the four dimensions of family involvement. Programs can then report this data to multiple stakeholders, such as program staff, funders, parents, and community members. Many funders, including the U.S. Department of Education through the 21st Century Community Learning Centers grants, require that OST programs provide documentation of services. However, even programs whose funders have no such requirement can benefit from systematically assessing and describing the services they offer. Family involvement services and activities may be documented and measured as part of a larger effort to document all OST services. The questions that OST programs use to guide the documentation of their family involvement services relate to the kind(s) of family involvement that the program practices. For example, programs that offer adult educational development may want to track the number and types of adult enrichment opportunities available. Similarly, OST programs that provide links between home and school may want to track how many times staff have participated in family-school meetings, or how frequently they distribute information about the participants’ school(s). Following are some service documentation questions to guide OST programs in documenting their family involvement services in each of the four out-of-school time family involvement dimensions. 1. Adult Development 2. Shared Out-of-School Time Experiences 3. Governance and Leadership 4. Linking to Schools Regardless of the type(s) of the family involvement strategies that OST programs pursue, there are some basic questions that all programs can ask to get a better sense of their service delivery:
A variety of methods can be used to collect data for service documentation. These include, but are not limited to:
Program impact evaluation is used to explore a program’s effectiveness in producing intended results. However, evaluating the impact of family involvement on the children and families who participate in an OST program is something that few, if any, OST programs have attempted. OST programs that have conducted impact evaluations of their programming tend to look at the effect of all program activities on the children and families they serve. Disaggregating particular program strategies or activities, such as family involvement practices, to see the differential impact of one area of service is more complex and beyond the scope of most evaluations. Nonetheless, including family involvement practices in the evaluation of overall program impact on participants is a viable strategy. This will allow programs who do value family involvement to prove to stakeholders that their comprehensive program, of which family involvement is one component, is making a difference. As OST evaluation designs become more nuanced, programs can use program impact evaluation to answer the following questions about family involvement. Clearly, each OST program can focus on the outcomes most closely related to their mission, and design research questions accordingly.
Impact evaluations strive to demonstrate a causal relationship between the services provided and the outcomes experienced by participants. The design of the evaluation determines the strength of the evidence for that causal relationship. An experimentally designed evaluation, which enables evaluators to assume that the only difference between the participants and non-participants is their participation in the evaluated program, provides the most powerful statistical support for a causal relationship. However, experimental studies require the identification of a randomly assigned control group, which may present evaluation challenges that are beyond the scope of many OST programs. Therefore, a quasi-experimental design, where OST program participants’ outcomes are compared to outcomes from demographically similar non-participants at a different school or from a national data set, may be a more practical solution. However, both experimental and quasi-experimental designs may be more resource-intensive than many local OST programs can manage by themselves. Further, many OST evaluation questions do not necessitate the use of experimental and quasi-experimental designs, and thus most OST programs prefer to assess their family involvement strategies using non-experimental approaches.
As an OST program begins the evaluation process, there are two possibilities as to who will conduct the evaluation. The first is the program itself—a director, staff member, or other program stakeholder. The second option is an outside evaluator. Which of these two possibilities is appropriate for any given OST program depends on the type of evaluation conducted and the resources that the program has at its disposal, including time, money, and evaluation expertise. Conducting a needs assessment or documenting services are two types of evaluation that an OST program might be able to conduct without the assistance of a professional program evaluator. An impact evaluation, due to the more complex design and methods required, may be better suited to an outside evaluator’s expertise. Outside evaluators also bring an objectivity to the evaluation that may be valuable in identifying areas for program improvement and inspiring credibility in program stakeholders, particularly funders and the public. Hiring a consultant is not the only way to bring in an outside evaluator. Following are two examples of how OST programs can partner with others to evaluate their family involvement practices. Partnering With Universities This university-OST program evaluation partnership is a promising model for a number of reasons. To begin with, cash-strapped programs often do not have the funds to hire an independent evaluator. Expertise at the university level is a good alternative. Also, using student researchers provides programs with multiple evaluators who each bring different perspectives. Finally, by engaging students in evaluation early in their career, the OST field is training a new generation of evaluators sensitized to the importance of family involvement in OST. Partnering With Parents Using parents as evaluators in this way generates many positive outcomes for the program and the families. First, relationships are formed among families as well as between families and OST program staff. The process of observing also brings parents into the daily activities of the program and enables them to better understand the program in action. Usually parents only see a snapshot at pick-up time, if that. These parents are in a unique position to both connect with their children’s daily experience and also to market the program to other families. As a result of the process, four of these families have formed a Parent Involvement Committee in response to their findings that other families are looking for ways to engage in the program.
Research suggests that family involvement in children’s learning and development supports children’s school success (see Appendix B). This involvement is not only focused on what parents do to support their children’s learning in school, but also what they do to engage in children’s learning and development during the non-school hours. As an increasing number of children and youth take advantage of OST programs, providers must consider innovative ways to expand the range of opportunities for families to participate in out-of-school time learning so that these benefits are not lost. Four dimensions of family involvement seem particularly salient to out-of-school time programs: adult development, shared experiences for children and families, program governance, and linking to schools. This last dimension, which is unique to OST programs, holds high potential for future evaluation of family involvement impact on student outcomes because research has found that a predictor of student achievement is family involvement in their child’s education at school and in the community (Henderson & Berla, 1994). To date, little evaluation work has been conducted to examine the nature and scope of family involvement, much less its impact on youth development. This review of OST program evaluations that do include family involvement reveals that most programs are conducting formative evaluations to learn about families’ experiences and program practices along the four dimensions of family involvement in OST. Moving forward, especially in the current policy context which emphasizes a connection between OST experiences and academic achievement, evaluation of family involvement in OST programs—whether through needs assessment, service documentation, or impact evaluation—is key to improving family involvement practices and, ultimately, to fostering improved child outcomes. Appendix AQuestions Sent to SAC-L and PPAS Listservs To inform this brief, HFRP staff monitored the School Aged Child Care Issues and Concerns listserv (SAC-L) and Promising Practices in Afterschool listserv (PPAS) and elicited responses from over 40 OST professionals regarding their family involvement practices. HFRP staff asked the following questions:
Appendix BResearch on the Benefits of Family Involvement in Education Research over the last 30 years has shown the benefits that family involvement in education can confer on students, their families, and schools. The positive influence of family involvement in education on children’s achievement has been established (Chavkin, 1993; Eccles & Harold, 1993; Epstein, 1991; Henderson & Berla, 1994; Hoover-Dempsey & Sandler, 1997). This positive effect endures throughout the pre-k through 12th-grade years and manifests itself in areas such as literacy and mathematics (Izzo, Weissberg, Kasprow & Fendrich, 1999; Lee & Croninger, 1994; U.S. Department of Education, 2001). Beyond increases in learning and higher test scores, research has also demonstrated that family involvement improves student school attendance, fosters higher aspirations for post-secondary education, facilitates successful transitions between elementary, middle, and high school, reduces drop-out rates, and increases enrollment in higher level high school classes (Kraft-Sayre & Pianta, 2000; Ma, 1999; Scribner, Young & Pedroza, 1999). Students’ social functioning can also be improved by family involvement in education in such areas as student behavior, motivation, social competence, and relationships between students and teachers as well as between students and their peers (Adams & Christenson, 2000; Marcon, 1999; Palenchar, Vondra & Wilson, 2001). In addition to supporting the successes of individual students, family involvement in education has been shown to foster more positive outcomes for both schools and families. Research suggests that schools may enjoy an improved school climate, increased resources, and richer curriculum capacity when families are actively engaged in their children’s learning (Smrekar & Cohen-Vogel, 2001; Moll & Gonzales, 1997). Outcomes for families may include: changes in skills, knowledge, and beliefs (Chrispeels & Rivero, 2001); access to more resources; and increased capacity to advocate for children’s rights and school reform (Lopez, 2002). Notes1 Little, P., Traub, F., & Horsch, K. (2002, April). Evaluation of 21st Century Community Learning Center programs: A guide for state education agencies. Cambridge, MA: Harvard Family Research Project. [Available at: www.gse.harvard.edu/hfrp/projects/afterschool/resources/issuebrief2.html.] 2 The Harvard Family Research Project Out-of-School Time Evaluation Database is available at: www.gse.harvard.edu/hfrp/projects/afterschool/evaldatabase.html. 3 For more information about the SAC-L listserv go to: ecap.crc.uiuc.edu/listserv/sac-l.html. 4 For more information about the PPAS listserv go to: www.afterschool.org/communicating.cfm#listserve. 5 For the complete text of the No Child Left Behind Act, see www.ed.gov/policy/elsec/leg/esea02/index.html. 6 Non-experimental evaluation designs use purposeful sampling techniques to get “information-rich” cases and include: case studies, data collection and reporting for accountability, participatory approaches, theory-based/grounded-theory approaches, ethnographic approaches, and mixed method studies. In experimental evaluation design, participants are randomly assigned to treatment and control groups. Quasi-experimental designs are similar to experimental designs except that the subjects are not randomly assigned to either the experimental or the control group, or the researcher cannot control which group will get the treatment. 7 For more information on performance measures in out-of-school time see: Coffman, J. (2002). Learning from logic models in out-of-school time. Cambridge, MA: Harvard Family Research Project. [Available at: www.gse.harvard.edu/hfrp/projects/afterschool/resources/learning_logic_models.html.] 8 In other words, 4 out of the 24 performance measures included in this analysis fell into this category. 9 Based on information from the Harvard Family Research Project Out-of-School Time Evaluation Database, available at www.gse.harvard.edu/hfrp/projects/afterschool/evaldatabase.html. ReferencesAdams, K. C., & Christenson, S. L. (2000). Trust and the family-school relationship. Examination of parent-teacher differences in elementary and secondary grades. Journal of School Psychology, 38, 477-497. Chavkin, N. (1993). Families and schools in a pluralistic society. Albany, NY: State University of New York Press. Chrispeels, J. H., & Rivero, E. (2001). Engaging Latino families for student success: How parent education can reshape parents’ sense of place in the education of their children. Peabody Journal of Education, 76, 119-169. Eccles, J. S., & Harold, R. D. (1993). Parent-school involvement during the early adolescent years. Teachers College Record, 94, 568-587. Epstein, J. (1991). Effects on student achievement of teacher’s practices of parent involvement. Advances in Reading/Language Research, 5, 261-276. Hendersen, A. & Berla, N. (Eds.). (1994). A new generation of evidence: The family is critical to student achievement. Columbia, MD: National Committee for Citizens in Education. Hoover-Dempsey, K. V., & Sandler, H. M. (1997). Why do parents become involved in their children’s education? Review of Educational Research, 67, 3-42. Izzo, C., Weissberg, R., Kasprow, W., & Fendrich, M. (1999). A longitudinal assessment of teacher perceptions of parent involvement in children’s education and school performance. American Journal of Community Psychology, 27(6), 817-839. Kraft-Sayre, M. E., & Pianta, R. C. (2000). Enhancing the transition to kindergarten: Linking children, families, & schools. Charlottesville, VA: University of Virginia, National Center for Early Development & Learning. Lee, V., & Croninger, R. (1994). The relative importance of home and school in the development of literacy skills for middle-grade students. American Journal of Education, 102, 286-329. Linney, J. A., & Wandersman, A. (1991). Prevention plus III: Assessing alcohol and other drug prevention programs at the school and community level. Rockville, MD: U.S. Department of Health and Human Services. Lopez, M. E. (2002, Spring). When parents assess schools. The Evaluation Exchange, 8(1), 16-17. [Available at: www.gse.harvard.edu/hfrp/eval/issue18/bbt1.html.] Ma, X. (1999). Dropping out of advanced mathematics: The effects of parental involvement. Teachers College Record, 101, 60-81. Marcon, R. A. (1999). Positive relationships between parent school involvement and public school inner-city preschoolers’ development and academic performance. School Psychology Review, 28(3), 395-412. Moll, L. C., & Gozalez, N. (1997). Teachers as social scientists: Learning about culture from household research. In P. M. Hall (Ed.), Race, ethnicity, and multiculturalism: Policy and practice (pp. 89-114). New York: Garland. Palenchar, D. R., Vondra, J. I., & Wilson, J. A. (2001). Parental involvement in the home and at school as predictors of early school functioning in an urban, low-income sample. Paper presented at the annual meeting of the American Educational Research Association, Seattle, WA. Roman, J. (Ed.). (1998). The NSACA standards for school-age care. Boston: National School Age Care Alliance. Ruiz-de-Velasco, J., & Fix, M. (2001). Overlooked & underserved: Immigrant students in U.S. secondary school. Washington, DC: Urban Institute. [Available at www.urbaninstitute.org/pdfs/overlooked.pdf (downloads as an Acrobat file).] Schorr, L. (1988). Within our reach: Breaking the cycle of disadvantage. New York: Doubleday. Scribner, J. D., Young, M. D., & Pedroza, A. (1999). Building collaborative relationships with parents. In P. Reyes, J. D. Scribner & A. Paredes-Scribner (Eds.), Lessons from high-performing Hispanic schools: Creating learning communities (pp. 36-60). New York: Teachers College Press. Shartrand, A. M., Weiss, H. B., Kreider, H. M., & Lopez, M. E. (1997). New skills for new schools: Preparing teachers in family involvement. Cambridge, MA: Harvard Family Research Project. Smrekar, C., & Cohen-Vogel, L. (2001). The voices of parents: Rethinking the intersection of family and school. Peabody Journal of Education, 76, 75-100. U. S. Department of Education, Office of the Deputy Secretary, Planning and Evaluation Service. (2001). The longitudinal evaluation of school change and performance (LESPC) in Title I schools, volume 1: Executive summary. Washington, DC: Author. [Available at: www.ed.gov/offices/OUS/PES/esed/lescp_highlights.html.] AcknowledgementsPreparation of this brief was made possible through the support of the Charles Stewart Mott Foundation. M. Elena Lopez at HFRP was an invaluable advisor and reviewer. We also want to acknowledge the professionals from the field who reviewed earlier drafts and provided valuable feedback: Tania Buck, Donna Walker James, and Christina Russell. In addition, we are grateful to Jill Anthony, Deanna Armstrong, Phyllis Berger, Sydney Bialo, Carol Bines, Jane Feinberg, Rachel Fix, Robin Granger, Laurie Beth Hartzell, Sharon Misplay, Claudette Reichert, Kelly Reinhardt, Joan Ricci, Jane Voorhees, Morton Weeks, John Werner, and Allison Williams for speaking to us about family involvement practices and evaluation in their OST programs and the numerous OST providers who shared their insights with us by email. ^ Back to top |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 |
 |
 |
 |
 |
 |
 |
 |