Face Value

by Samantha Cleaver

In an episode of the PBS Kids show WordGirl, the spunky, vocabulary-savvy character dressed in red superhero suit and cap, yellow cape waving behind her, takes on evil businessman Mr. Big. Big is plotting to hypnotize people with a marketing campaign that would convince them to buy his signature product, “the machine that makes the stuff.” Not to worry, by the end of the 11-minute episode, WordGirl has triumphed and Mr. Big is defeated — for now. But the bigger questions for Lecturer Ilona Holland’s class remained: Does WordGirl (which recently won an Emmy) resonate with the target audience of early elementary school students? And will watching her and monkey sidekick Captain Huggy Face on TV help students learn vocabulary?

Last fall, students in T523: Formative Evaluation for Educational Product Development, were charged with finding out whether an educational product, in this case, a children’s show, lived up to its goals. This concept of formative evaluation — testing the product or program with its intended audience — says Lecturer Joe Blatt, Ed.M.’77, director of the Technology, Innovation, and Education Program, is a powerful idea for anyone who is designing materials or experiences. For students who will be creating the next wave of educational programs and materials, he says, learning evaluation is critical.

The challenge for Holland’s students was to translate the goal of the WordGirl show — teach vocabulary through entertainment — into age-appropriate questions for six- and seven-year-olds. They collected data from the target audience, analyzed it, and then presented the findings to the show’s creator and producer, Dorothea Gillim, Ed.M.’91. In the end, the students found that kids loved the show and their vocabulary improved after watching two episodes.

“The kids thought it was fun,” says Regina Husa, Ed.M.’08. “They liked WordGirl, and they loved Captain Huggy Face.”

For Gillim, the results were encouraging. “It was gratifying in the sense that we were being effective in teaching vocabulary,” she says. Gillim is now developing a new season that incorporates the results from the T523 evaluation — namely that kids responded most to action and to Captain Huggy Face. From the student perspective, Husa, who evaluates medical programs for a hospital in Montreal, learned how to more effectively write specific questions. “I’ll be working with people in their 20s and 30s, and that’s still an issue,” she says. “If you don’t ask the right questions, you won’t get the right data.”

Theory on the Ground

Like T523, T506: Evaluation for Informed Decision Making, another class that Holland teaches in the spring, takes the theory of program evaluation to its logical conclusion — handson projects in the community where students can work with clients and produce real results. (T523 evaluates a product or service; T506 evaluates an organization.) Blatt says this type of approach reflects the school’s changing role in the community.

“There was a time when the school was, in my impression, a partner of arts and sciences,” he says. But now, the role of the school is about contributing to the actual experience of children’s learning around the world, he says. And in order for students to make a difference in what happens with children, they need to have grounded practice in the application of theory.

In both T523 and T506, the students get to practice their skills while community organizations receive access to evaluation tools that they might not have otherwise because of small nonprofit budgets or limited time. Holland’s students have also expanded the Harvard Graduate School of Education’s reach by producing evaluation reports for national organizations like the Detroit Institute of Art and NASA, as well as local groups.

One local group was Cambridge School Volunteers, an organization that trains and supervises volunteers in classrooms across the city. When Harriet Finck, executive director of the group, came to T506 last spring, she was ready to tell the class about her organization, “warts and all.” She had wanted to do an evaluation of her program for a while, but without money in the budget to pay a consultant, she was having trouble getting started. As she looked out at the class, Finck thought the graduate students seemed young, but she was impressed with the questions they asked. (Down the road, she would also be impressed with the work they put into the project and the eventual 69-page report that detailed how to evaluate the organization’s literacy program.) At first, she remembers thinking, “How are they going to make sense out of all I’m telling them?” But they did. “I’d had all these thoughts in my head, and they heard those, found the commonalities, and put it all together.”

The students did the same for the other two local organizations they evaluated in the spring: the Science Club for Girls in Cambridge and the Boston Children’s Museum. During the first half of the semester, the students learned about the organizations with which they would be working and the program evaluation process. “The point is to determine what, exactly, the organization needs and how the evaluation process can be applied to that organization’s needs,” says Holland, Ed.M.’85, Ed.D.’91. For students who will go into public and nonprofit sector careers, it’s important to work with what’s available, she says. “We really work hard to anchor the plan in the realities of the organizations” in terms of time, money, and staff, she says. “You can think of a very rigorous and extensive evaluation, but the organization could never carry it out because of limited resources and expertise.”

The projects are two-way streets, and “the process is very synergistic,” Holland says. The community organizations benefit from the process and the final plan while the students learn firsthand how to create a needs assessment, a program theory chart, and other materials. For Finck, working with the students helped her small staff get on the same page.

“Everybody has a slightly different perspective on [his or her] role,” she says. The report allowed her to present information objectively and to help her employees feel good about what they are doing as they move forward.

Connie Chow, executive director of the Science Club for Girls, was eager to learn more about measuring outcomes. But, she says, working with Holland’s class helped her slow down and consider what really goes into an evaluation in a way that makes sense for her organization.

Hannah Sin, Ed.M.’08, took T506 to learn how to evaluate public and nonprofit organizations that often lack a clear bottom line. Sin worked on a needs assessment for Chow that evaluated the attrition of middle school students after the program. In the process, she saw firsthand the challenges that many organizations face in funding and implementing the exact strategies she was proposing.

“The organizations we worked with are doing a lot of highimpact work with communities and families,” she says. “In terms of financial and staff resources, we knew that we didn’t want to overburden them with plans that would overstretch the staff” or tax them financially. Sin proposed working with similar area groups, an idea that was well received by the organization.

At the end of the semester, the students presented their work to the organizations. Chow received seven evaluations that she plans to start implementing in the next school year. “We had these ideas percolating in the back of our minds,” she says, “but as a small organization, we haven’t had the time to articulate them, nor to spend enough time with the literature. So what this experience has done for us is get us more up to date on evaluation . . . and provide us with manageable chunks that we can work with in terms of programming.” Finck is working the report into an evaluation plan that she thinks will take several years to implement but will bring her organization to a new level of functioning.

A New Way of Thinking

Holland wants her students to leave her class with a new way of thinking. “You need to question the assumptions of any organization or project you’re working on,” she says, “and doublecheck them against data.” She also wants students to learn to turn to their target audiences for feedback, whether they’re kindergartners or senior citizens.

Students appreciate putting theory into practice. “Rather than feeding us textbooks,” says Wendy Valentine, Ed.M.’08, “Holland really cares more about what it takes to do a good evaluation, and how truly difficult, but how powerful, it is to organizations.”

“What struck me is the high level of emotional intelligence that it required,” says Sin of evaluating, a skill that is usually considered technical. In working with the organizations, “people are so passionate about what they do in these organizations, you don’t want to offend them or undervalue their work.”

At the end of each semester, the students and organizations see the results. Holland says, “I just love the fact that we are really making a difference.”

— Samantha Cleaver is a freelance writer from Chicago. This is her first piece for Ed.

Illustration by Jeff Hopkins, Ed.M.’05 (Wordgirl and Captain Huggy Face characters courtesy of Scholastic Entertainment Inc. and Soup2Nuts Inc. TM and © 2008 Scholastic Entertainment Inc.)

About the Article

A version of this article originally appeared in the Fall 2008 issue of Ed., the magazine of the Harvard Graduate School of Education.

Respond to this story with an e-mail to the editor.