Out-of-School Time Evaluation Snapshot

Detangling Data Collection: Methods for Gathering Data

Number 5, August 2004

Suzanne Bouffard and Priscilla M. D. Little, Harvard Family Research Project

95KB Acrobat file

Help for Acrobat file

This Snapshot is also available in hard copy for free. Please go to our publications list to learn how to order it.

Harvard Family Research Project’s series of Out-of-School Time Evaluation Snapshots distills the wealth of information compiled in our Out-of-School Time Program Evaluation Database¹ into a single report. Each Snapshot examines a specific aspect of out-of-school time (OST) evaluation. This Snapshot describes the common data collection methods used by current out-of-school time programs to evaluate their implementation and outcomes.

Once the domain of researchers, evaluation has become an integral part of daily life—and a key accountability requirement—for out-of-school time (OST) programs and professionals. Many programs use a variety of methods to collect information on community needs, program operations, and youth outcomes. This diversity of data collection options is one important reason for all OST professionals to understand the what, why, and how of data collection methods. This knowledge will help them make informed decisions about the best strategies for collecting data appropriate for their unique program characteristics and for the evaluation questions they wish to address.

This Snapshot describes common data collection methods and offers guidelines for their use. It provides detailed information about using surveys and questionnaires, interviews and focus groups, observations, tests and assessments, and secondary sources and data reviews. In describing each method, this Snapshot provides real-world examples from program evaluations represented in Harvard Family Research Project’s Out-of-School Time Program Evaluation Database.

Evaluation Terminology Used in This Snapshot

When it comes to the language of evaluating programs, stakeholders use a variety of terms. The following definitions provide a common language for this Snapshot. These terms are important for understanding how data collection methods fit into the context of program evaluation.

Evaluation – an overall assessment of whether a program achieves its goals, how it works, and what adjustments may improve its results

Outcome – a bottom-line condition of well-being for children, families, or communities. It is a broadly defined, fundamental condition that government and citizens consider essential. An outcome may also be referred to as a result.

Sample – the group of individuals or other units (e.g., programs, communities, etc.) studied. A sample is often a subset of the total population served by a program.

Data collection method – the strategy and system used to gather information on participants, programs, and other elements of the evaluation

Data source – the individual or institution from which the evaluation data are collected (e.g., participants, parents, school records, etc.)

Quantitative data – numeric information that is subject to statistical analysis

Qualitative data – text-based information that provides descriptive details, often collected from interviews, focus groups, or observations

Data Collection Methods

The following six data collection methods are used by OST programs across the country to collect both formative and summative evaluation information. The following information is provided below about each data collection method: its primary purpose, the type of information it can collect (based on our review of the evaluations posted in our OST Program Evaluation Database), its advantages and challenges, and any relevant additional information.

Method 1: Surveys and Questionnaires

Purpose: Surveys and questionnaires are collected by evaluators to gather specific information from participants, families, staff and administrators, teachers, community members, and other stakeholders. Data collected often include demographic information, satisfaction levels, and opinions of the program.

Surveys and questionnaires are usually administered on paper, in a structured or semi-structured format. Respondents often choose from among a set of forced-choice, or provided, responses. These can include yes/no or scaled responses. Surveys and questionnaires can be administered in person, by mail, over the phone, or via email/Internet.

Type of Information Collected

• Youth and family demographics
• Program characteristics (activities, staffing, level of parental involvement)
• Youth’s program participation (how often, for how long, in which program activities)
• Youth’s and families’ perceptions of and satisfaction with the program
• Youth outcomes (academic adjustment and achievement, social skills, risk-taking behavior, attitudes, self-concept, employability, etc.)

Advantages

• Less time consuming and expensive to administer than other methods
• Can be administered to large groups of individuals
• Effective for assessing program satisfaction

Challenges

• Data entry and analysis can be time consuming
• May be difficult to receive completed surveys from stakeholders. A range of incentives can be offered on return of completed surveys to boost the likelihood of response, from financial compensation for providers and families to parties for youth participants.

Additional Information: Surveys and questionnaires are the most commonly used method for collecting data by programs in the OST Program Evaluation Database.

Method 2: Interviews and Focus Groups

Purpose: Interviews and focus groups are most often used to gather detailed, qualitative descriptions of how programs operate and how stakeholders perceive them. Interviews are conducted one-on-one, while focus groups are conducted in small groups. Both are usually conducted with targeted samples of stakeholders, such as staff, administrators, youth, families, funders, and community members.

Interviews and focus groups can be conducted in person or by phone. Questions are generally open-ended and responses are documented in thorough, detailed notes or transcription. However, some interviews use structured quantitative response categories.

Type of Information Collected

• Student, family, and community background (demographics, alternative or prior OST arrangements)
• Program characteristics (goals, needs, implementation, staff hiring and training, program accessibility, community involvement)
• Youth’s program participation (how often, youth’s goals in the program)
• Youth’s and families’ perceptions of and satisfaction with the program
• Youth outcomes (academic adjustment, social skills, attitudes, self-concept)

Advantages

• Provide rich data that paint a broad picture
• May highlight issues not previously considered or information that is useful for interpreting quantitative data collected through other methods
• Small focus groups may increase the comfort level of participants

Challenges

• Can be difficult to elicit participation from individuals who have time constraints
• Interviews and large focus groups may intimidate some participants
• Documentation and analysis can be time consuming and may require the help of someone versed in qualitative analysis

Method 3: Observations

Purpose: Observations are a generally unobtrusive method for gathering information about how the program or initiative operates. They are usually conducted by external evaluators or researchers and are often used to verify and supplement information gathered through other methods. This is often a method used to assess program quality standards such as those in the School-Age Care Environmental Rating Scale (SACERS).²

Observations can be highly structured, with protocols for recording specific behaviors at specific times, or unstructured, taking a “look-and-see” approach. They are most reliable when they are conducted over a period of time to minimize the chances of the observation day(s) being atypical.

Type of Information Collected

• Program characteristics (implementation, activities, interpersonal interactions, administration and management, health and safety)

Advantages

• Provide highly detailed information from an external perspective on what actually occurs in programs
• Trained evaluators may provide less biased descriptions than program staff or stakeholders

Challenges

• Can be time consuming, labor intensive, and expensive
• Observers must be trained and be consistent with one another
• Observations conducted on a sample of days may not represent the range of program practices and experiences over time

Method 4: Tests and Assessments

Purpose: Tests and assessments are developed or used specifically for the program evaluation to quantify characteristics of the program, participants, or outcomes. Examples include achievement tests and psychological tests (e.g., assessments of depressive symptoms and self-esteem). They may be standardized or created by program evaluators for the specific program.

Type of Information Collected

• Program implementation (staffing models, activity offerings, management and organizational strategies)
• Youth outcomes (academic achievement, self-esteem, mental health and well-being, social skills and development, risk-taking behavior, career maturity, fitness)

Advantages

• Often more valid and reliable than perceptions or opinions
• Comparing scores before and after the program is a strong method for assessing whether outcomes actually changed over time

Challenges

• Can be costly and time intensive
• May require scoring by an external source

Additional Information: For more information on standardized assessments, see HFRP’s Snapshot 6, due to be available in fall 2004. (To be notified when it is available sign up for our out-of-school time updates email at www.gse.harvard.edu/hfrp/subscribe.html.)

Method 5: Document Reviews

Purpose: Document reviews analyze existing program records and other documents not gathered or developed specifically for the evaluation. Examples include recruitment and attendance records, budget, staff records, and annual reports. They are particularly useful for documenting implementation.

Type of Information Collected

• Youth, family, and community demographics
• Program characteristics (implementation, activities and curricula, budget)
• Youth’s program participation (how often and for how long)
• Youth characteristics before and after the program (academic achievement, school attendance, limited English proficiency status)

Advantages

• Records are tailored to programs
• Save on evaluation time and costs
• May elicit a high degree of accuracy from staff if they are also used for accountability purposes

Challenges

• May not be available or applicable for some indicators
• May be incomplete due to staff time constraints

Method 6: Secondary Sources and Data Reviews

Purpose: Secondary sources and data reviews use existing documents or data that were originally collected for purposes other than the program evaluation or documentation, but which are useful for the evaluation. Examples include achievement data, standardized test scores, court records, and community demographic data.

They can also include previous similar studies, which are used to compare program youth with other American youth (e.g., National Education Longitudinal Survey).

Type of Information Collected

• Youth, family, and community background (demographics, neighborhood income, criminal history, mental health history)
• Program characteristics (implementation, program-school connections)
• Youth academic outcomes (grades, test scores, attendance, disciplinary actions, grade promotion, course enrollment)
• Community outcomes (crime rates, creation of community partnerships)

Advantages

• May be less biased than perceptions or opinions
• Can save administrative time and costs

Challenges

• Obtaining records often requires special permission from parents and school officials
• Some national datasets charge access fees

Additional Information: Programs in the OST Program Evaluation Database primarily use secondary sources to collect academic achievement data and, to a lesser extent, information on program implementation.

Data Collection Tips: Issues to Consider

Using Single Versus Multiple Methods
No one data collection method is ideal for every situation. For this reason, it is preferable to use multiple methods whenever possible. Using multiple methods to assess the same outcomes (e.g., using surveys and document review to assess program management) provides a richer, more detailed picture. It also illuminates inconsistencies between methods and reduces the chance of bias caused by a particular method. According to the scan of the OST Program Evaluation Database, multiple methods are more often used by large-scale evaluations of multisite programs, such as the After School Achievement Program and California’s Communities Organizing Resources to Advance Learning initiative. This may reflect the fact that using multiple methods requires more resources, which tend to be more available in large programs.

Selecting Data Sources
Equally important to selecting a data collection method is selecting a data source, or information provider. Data can be collected from youth, families, staff, funders, educators, and other stakeholders. Certain data sources lend themselves more easily to certain data collection methods; for example, to assess parent satisfaction, surveys are popular because they can be mailed to hard-to-reach parents. As with data collection methods, it is recommended that evaluations use several data sources. However, programs may already have some of the evaluation information they need. Conducting an inventory of what programs already have and what is already available in the community can reduce data burden as well as data collection costs.

Selecting a Sample of Individuals
The choice of data collection methods is also affected by the sample to be studied. Some methods are well suited to collecting data from all participants (e.g., surveys), while others are better suited to a smaller group that represents the diversity of all participants (e.g., focus groups). The choice of sample is in turn affected by the size of the program to be evaluated. In general, including all participants produces the most reliable results but may limit the type and amount of data collected because of the cost implications of doing so.

Collecting Data Before and After Program Participation and Implementation
Using the same data collection method to gather information before the start of the program and after its completion (also known as a pre/posttest design) provides the opportunity to determine whether some characteristic changed during the course of the program. This can suggest that the program played a role in effecting the change; however, unless a program uses random assignment as part of its pre/posttest design, this method cannot establish that the program caused the change, because other unmeasured factors may have been responsible.

Cost Considerations
Selecting data collection methods has cost implications. While conducting observations and reviewing program documents can be done with little additional funding, designing and administering a survey and analyzing its results may require the assistance of an outside evaluator. At minimum, it will require additional staff time for training. Similarly, using standardized tests and assessments will likely necessitate external expertise to analyze results. An additional cost consideration is the use of management information systems (MIS) that can be used to record and store data collected from numerous methods. Like most other components of evaluation, it is important to start small and build data collection tools over time, as appropriate to the program's evolving evaluation needs.

Acknowledgements

This Snapshot is based on a review of the Harvard Family Research Project Out-of-School Time Program Evaluation Database, which is supported by grants from the C. S. Mott Foundation and the W. K. Kellogg Foundation. The principal investigator for this study is Dr. Heather B. Weiss. The authors wish to thank Julia Coffman, HFRP consultant, for her thoughtful review.

Notes

¹ Our database contains profiles of out-of-school time (OST) program evaluations, which are searchable on a wide range of criteria. It is available in the OST section of the HFRP website at www.gse.harvard.edu/hfrp/projects/afterschool/evaldatabase.html.
² For information on scales used to assess program quality, see Yohalem, N., Pittman, K., & Wilson-Ahlstrom, A. (2004). Getting inside the “black box” to measure program quality. The Evaluation Exchange, 10(1), 6–7. Available at www.gse.harvard.edu/hfrp/eval/issue25/spotlight.html.

All rights reserved. Information on this site may not be reproduced in whole or in part without written permission from Harvard Family Research Project.

© 2006 President and Fellows of Harvard College
Published by Harvard Family Research Project
3 Garden Street
Cambridge, MA 02138
Tel: 617-495-9108
Fax: 617-495-8594
Email: hfrp@gse.harvard.edu