Out-of-School Time Evaluation Snapshot
Performance Measures in Out-of-School Time Evaluation
Number 3, March 2004
Priscilla M. D. Little,
Project Manager; Erin Harris, Research
Assistant; and Suzanne Bouffard, HFRP Consultant
Download this brief:
Harvard Family Research Project's series of Out-of-School Time Evaluation
Snapshots distills the wealth of information compiled in our Out-of-School Time
Program Evaluation Database1
into a single report. Each Snapshot examines a specific aspect of out-of-school
time (OST) evaluation. This Snapshot provides the academic, youth development,
and prevention performance measures currently being used by out-of-school time
programs to assess their progress. It also includes the corresponding data sources
for these measures.2
A future Snapshot will focus on the specific standardized assessment
tools and tests that programs use to demonstrate impact.3
Increasingly, out-of-school time (OST) programs are being asked to provide
data that documents their progress and demonstrates their results. But for many
OST programs and their over-worked staff this added responsibility is the “straw
that broke the camel's back.” Across the country there is a cry from programs
for help in deciding what to measure and how to measure it. Although there is
no formal consensus regarding realistic outcomes for all OST programs,
one of the unintended benefits of the release of the first national 21st Century
Community Learning Centers impact evaluation4
has been increased attention to the question “What is realistic for OST
programs to be held accountable for?” Proceedings from a June 2003 After
School Summit hosted by the U.S. Department of Education and Arnold Schwarzenegger
indicate that researchers and policymakers alike are interested and invested
in helping programs measure and improve their performance.
This Snapshot contributes to that conversation by providing a comprehensive
listing of the performance measures that OST programs nationwide are currently
using as well as the sources that they use to collect data on each measure.
This Snapshot is based on a review of the outcomes reported in the evaluations
posted in the Harvard Family Research Project Out-of-School Time Program Evaluation
Database.5 While not
an exhaustive listing of the universe of performance measures currently being
used, this Snapshot provides information for programs making evaluation
decisions as well as for researchers, evaluators, and others helping programs
build evaluation capacity and improve performance.
|
What Is a Performance Measure?
Performance measures assess your program's progress on the implementation
of your strategies and activities. They assess the results of your
out-of-school time (OST) program's service delivery. Ask yourself: In
the work that my program does, what do we hope to directly affect? What
results are we willing to be directly accountable for producing? What
can we realistically accomplish?
There are two types of performance measures:
Measures of effort - Also commonly known as outputs, these are
measures of the products and services generated by program strategies
and activities. Ask yourself: What does my program generate (e.g., publications,
training materials), What levels of activity do we produce (e.g., the
number of children served or products developed), and What will measure
the quality of our services (e.g., parent and child satisfaction rates)?
Measures of effort assess how much you did, but do little in terms of
explaining how well you did it or how well your program ultimately worked
for the target population with whom you are working. These are the easiest
of all the evaluation measures to identify and track (e.g., number of
children served in the OST program and participant demographics, number
of classes/sessions/trainings held, etc.).
Measures of effect - These are changes in knowledge, skills, attitudes,
or behaviors in your target population. Ask yourself: How will I know
that the children or families I work with in my OST program are better
off? What changes do I expect to result from the strategies and activities
my program provides? Remember that measures of effect reflect changes
that your program acting alone expects to produce (e.g., increased social
competence, higher self-esteem and confidence, improved study habits).
Adapted from: Watson, S. (2000). Using results to improve the lives
of children and families: A guide for public-private child care partnerships.
Vienna, VA: Child Care Partnership Project. Available at nccic.org/ccpartnerships/results.pdf
(Acrobat file) and Coffman, J. (2002). Learning from logic models in
out-of-school time. Cambridge, MA: Harvard Family Research Project.
Available at www.gse.harvard.edu/hfrp/projects/afterschool/resources/
learning_logic_models.html.
|
Performance Measures and Data Sources for Academic Outcomes
Evaluators use a broad range of performance measures to assess participants'
academic outcomes in OST programs. They range from grades to standardized testing
to participants' homework completion. The list below shows the broad range of
possibilities for OST academic performance measures. Obviously a single program
would not use all these measures and may develop some not on this list. Data
sources for these performance measures include: parent, teacher, principal,
OST staff, and participant interviews and surveys; school records; and standardized
testing results.
|
Data Sources for Academic Performance Measures
|
| Performance Measure |
Data Source(s) |
| Ability to get along with others in school |
Parent |
| Academic knowledge in specific content areas |
Participant |
| Academic performance in general |
Participant, parent, principal, program staff |
| Attendance/absenteeism |
Participant, school records, teacher, parent, principal |
| Attitude toward school |
Participant |
| Behavior in school* |
Principal, teacher, standardized behavior scales |
| Communication skills in school |
Parent |
| Cognitive improvement |
Teacher |
| College attendance |
Participant youth survey, school records |
| College preparation/plans/information |
Participant |
| College retention/continuation |
Participant |
| Comfort with taking tests |
Participant |
| Disciplinary action/suspension/expulsion |
School records |
| Dropout |
Participant |
| Effectiveness of school overall |
Principal |
| Effort (including grades for effort) |
Participant, school records |
| English language development |
Participant |
| Expectations of achievement and success |
Participant, teacher |
| Family involvement in school events |
Principal |
| General academic knowledge |
Participant |
| Grade point average |
School records |
| Grade retention |
Participant, principal, school records |
| Grades – overall |
Participant, teacher |
| Grades in content areas (math, reading, etc.) |
School records, parent |
| Grades in OST program academic courses |
Participant |
| Help seeking |
Participant |
| Homework performance |
Participant, teacher, parent, principal |
| Intention to graduate |
Participant |
| Intention to remain in school |
Participant |
| Interest in recreational reading |
Program staff |
| Learning skills development |
Teacher |
| Liking school |
Participant, parent |
| Motivation to learn |
Participant, principal |
| Overall happiness in school |
Parent |
| Perceived competence |
Participant |
| Quality of schoolwork |
Participant |
| Recreational reading |
Parent, teacher |
| Safety – viewing school as a safe place |
Parent |
| Scholastic achievement assessed by knowledge about specific
subjects |
Standardized behavior scales |
| Scholastic competence |
Participant |
| School vandalism |
Principal |
| Special education placement |
Principal |
| Standardized test scores |
SAT-9, PSAT, Comprehensive Test of Basic Skills, Stanford
Diagnostic Reading Test (SDRT-IV), Terra Nova Reading/Language Arts and
Mathematics Computation Test, CTB/McGraw-Hill CAT-5 Math and Reading Comprehension
Tests, Informal Reading Inventory, state assessments, school-wide assessments,
program-specific tests |
| Study skills |
Participant |
| * School behaviors included in the scales
are: frustration, tolerance, distraction, ignoring teasing, nervousness,
sadness, aggression, acting out, shyness, and anxiety. |
Performance Measures and Data Sources for Positive Youth Development Outcomes
Many OST evaluations assess youth development outcomes, which are broadly defined
as those outcomes that assess the social and emotional development of program
participants. Outcomes that fall into this category range from standardized
measures of self-esteem, participant behavior, and interpersonal skills to decision
making and goal setting, leadership, and career development. The list below
shows the broad range of possibilities for OST youth development performance
measures. Obviously a single program would not use all these measures and may
develop some not on this list. Like the data sources for academic performance
measures, sources for youth development performance measures include: parent,
teacher, principal, OST staff, and participant interviews and surveys; school
records; and standardized assessments.
|
Data Sources for Youth Development Performance Measures
|
| Performance Measure |
Data Source(s) |
| Adult relationships |
Participant |
| Awareness of community resources |
Participant |
| Banking skills/financial skills |
Participant |
| Behavior change toward new program component |
Parent, participant, program staff |
| Communication skills |
Participant |
| Computer skills |
Participant |
| Conflict resolution |
Program staff |
| Cultural awareness |
Participant |
| Depression prevention |
Parent, teacher |
| Developmental Assets* |
Participant youth survey |
| Exposure to new activities |
Program staff, principal |
| Facing issues outside of OST program |
Participant |
| Independence |
Participant |
| Interaction with adults |
Teacher, principal, program staff |
| Interaction with other students in OST |
Participant |
| Interaction with peers |
Participant, parent, program staff |
| Interest in nonacademic subjects (art, music, etc.)
|
Participant, program staff |
| Leadership development/opportunities |
Participant, program staff |
| Life skills |
Participant youth survey |
| Opportunities to volunteer |
Participant |
| Peer relationships |
Standardized assessment, program staff |
| Problem-solving skills |
Participant |
| Problem-solving skills |
Participant |
| Productive use of leisure time |
Participant |
| Professional/workforce skills and development |
Participant |
| Respect for others |
Participant |
| Self-confidence |
Parent, provider, standardized tests, employer |
| Self-esteem |
Participant youth survey, standardized assessment, teacher |
| Sense of belonging |
Participant |
| Sense of community |
Participant |
| Sense of safety |
Participant |
| Social skills |
Parent |
| Sources of support for youth |
Participant |
| * Search Institute 40 Developmental Assets.
For more information on this assessment framework, see www.search-institute.org/assets. |
Performance Measures and Data Sources for Prevention Outcomes
Performance measures that fall into the prevention category include participants'
changes in sexual behavior, feelings of personal safety, changes in drug and
alcohol use and abuse, and overall improvements in physical health. The list
below shows the broad range of possibilities for OST prevention performance
measures. Obviously a single program would not use all these measures and may
develop some not on this list. Data sources for this information include: parent,
participant, mentor, and teacher interviews and surveys and police reporting
and records.
|
Data Sources for Prevention Performance Measures
|
| Performance Measure |
Data Source(s) |
| Aggression |
Participant, mentor |
| Birth control use |
Participant |
| Crime rate – youth perpetrators |
Police records |
| Crime rate – youth victims |
Police records |
| Delinquency |
Participant, mentor |
| Delinquency – attitudes about it |
Participant |
| Delinquent peer group membership |
Participant, mentor |
| Gang membership |
Participant, mentor |
| “Kept out of trouble” |
Participant |
| Physical fitness |
Parent |
| Pregnancy and birth rates |
Participant |
| Race relations |
Participant |
| “Risk taking” |
Parent, teacher |
| Sex – attitudes about it |
Participant |
| Sexual behavior/initiation of intercourse |
Participant |
| Sexual decision making |
Participant |
| Sexual knowledge |
Participant |
| Substance use – attitudes about it |
Participant |
| Substance use decision-making skills |
Participant |
| Substance use/drug use/alcohol use |
Participant, mentor |
| Weapons possession/use |
Participant, mentor |
Which Performance Measures Are Best?
There may never be a single set of performance measures that all OST
programs can be held accountable for, but there are at least four important
points that all OST programs must take into consideration when selecting performance
measures for evaluation:
| |
Harvard Family Research Project Out-of-School Time Program
Evaluation Database
The Harvard Family Research Project (HFRP) Out-of-School
Time Program Evaluation Database contains profiles of out-of-school time
(OST) program evaluations. Its purpose is to provide accessible information
about previous and current evaluations to support the development of high
quality evaluations and programs in the OST field.
Types of Programs Included in the Database
Evaluations in the database meet the following three criteria:
- The evaluated program or initiative operates during out-of-school
time.
- The evaluations aim to answer a specific evaluation question or set
of questions about a specific program or initiative.
- The evaluated program or initiative serves children between the ages
of 5 and 19.
Types of Information Included in the Database
Each profile contains detailed information about the evaluations as well
as an overview of the OST program or initiative itself. Web links to actual
evaluation reports, where available, are also provided, as are program
and evaluation contacts.
How to Use the Database
The database is located in the OST section of the HFRP website at: www.gse.harvard.edu/hfrp/
projects/afterschool/evaldatabase.html. The search mechanism allows
users to refine their scan of the profiles to specific program and evaluation
characteristics and findings information.
|
1. The range of performance measures currently used to assess program OST
outcomes reflects the diversity of OST programming. The selection of which
performance measures are best suited to any single program or initiative should
be inextricably tied to the program's goals, strategies, and activities. Being
intentional about a theory of change—a way of articulating the program's
primary goals, strategies, and activities—can help to determine what measures
to use to assess progress toward achieving program goals.
2. Performance measures are not the same as performance indicators.
Indicators, such as “increasing the percentage of students graduating from
high school,” are measures for which data exist that quantify and track
community-wide progress toward results. They require community-wide effort
to move and reflect substantial changes across a community. The selection of
performance measures should reflect the context in which the OST program operates,
recognizing the strengths and limitations of the program to affect overall change
given that context. And they should contribute to understanding progress on
the program's performance indicators.
3. Availability of data sources is a consideration when selecting and
developing performance measures. Many programs rely on parent, participant,
and staff reporting as data sources, using program-generated surveys and questionnaires
to collect data. This is a less costly option than using standardized academic
and behavioral assessments that may require training to administer, but has
less validity than standardized testing and assessment tools.
4. Performance measures should, in part, be selected because they will yield
useful information for program improvement as well as to fulfill accountability
requirements. A litmus test for a good evaluation, and consequently the list
of performance measures selected, is to ask the question, “Will the information
collected be useful to the program and its stakeholders?” The answer should
be a resounding “yes.”
1 Our database contains
profiles of out-of-school time (OST) program evaluations, which are searchable
on a wide range of criteria. It is available in the OST section of the HFRP
website at www.gse.harvard.edu/hfrp/projects/afterschool/evaldatabase.html.
2 Performance measure
information for this Snapshot was adapted from Documenting Progress
and Demonstrating Results: Evaluating Local Out-of-School Time Programs,
a joint publication of the Harvard Family Research Project and the Finance Project,
available at www.gse.harvard.edu/hfrp/projects/afterschool/resources/index.html#local.
3 This and future Snapshots
in the series will be available in the OST section of the HFRP website at www.gse.harvard.edu/hfrp/projects/afterschool/resources/index.html.
(To be notified when Snapshots become available sign up for our OST website
change notification email at www.gse.harvard.edu/hfrp/subscribe.html.)
4 U.S. Department of
Education, Office of the Under Secretary. (2003). When schools stay open
late: The national evaluation of the 21st-Century Community Learning Centers
program, first year findings. Washington, DC: Author. Available at www.ed.gov/pubs/21cent/firstyear.
5 Our scan for this
Snapshot was conducted in October 2003, at which time there were 54 profile
posted in our database.
Acknowledgements
This Snapshot is based on a review of the Harvard Family Research Project
Out-of-School Time Program Evaluation Database, which is supported by grants
from the C. S. Mott Foundation and the W. K. Kellogg Foundation. The principal
investigator for this study is Dr. Heather B. Weiss. The authors wish to thank
Julia Coffman and Sherri Lauver, HFRP Consultants, for their review.
Related Resources
Collaborative Communications Group. (2003, June 5–6). After School
Summit summary report. The After School Summit was hosted by the U.S.
Department of Education and Arnold Schwarzenegger in Washington, DC. The summary
report available at www.publicengagement.com/afterschoolsummit.
The Finance Project. Out-of-school time project: Measuring and using results.
Available at www.financeprojectinfo.org/OST/ostevaluation.asp.
Harvard Family Research Project. Out-of-school time program evaluation database.
Available at www.gse.harvard.edu/hfrp/projects/afterschool/evaldatabase.html.
U.S. Department of Education, Office of the Under Secretary. (2003). When
schools stay open late: The national evaluation of the 21st-Century Community
Learning Centers program, first year findings. Washington, DC: Author.
Available at www.ed.gov/pubs/21cent/firstyear.
Little, P., Dupree, S., & Deich, S. (2002). Documenting progress
and demonstrating results: Evaluating local out-of-school time programs.
Cambridge, MA and Washington, DC: Harvard Family Research Project and The Finance
Project. Available at www.gse.harvard.edu/hfrp/projects/afterschool/resources/index.html#local.
Little, P., & Harris, E. (2003). A review of out-of-school time quasi-experimental
and experimental evaluation results. Cambridge, MA: Harvard Family Research
Project. Available at www.gse.harvard.edu/hfrp/projects/afterschool/resources/snapshot1.html.
Watson, S. (2000). Using results to improve the lives of children and
families: A guide for public-private child care partnerships. Vienna,
VA: Child Care Partnership Project. Available at nccic.org/ccpartnerships/results.pdf
(Acrobat file).
|
