HGSE Students Evaluate NASA

It is not often that master's students find their class research projects being used by major organizations such as NASA. But, that is exactly what has happened for students in Lecturer Ilona Holland's spring course, Evaluation for Informed Decision-Making. For the past two years, Holland's students have been developing evaluation plans for parts of two NASA programs, Mars Public Engagement and the Robotics Alliance Project. The program leaders have been so impressed with the final plans that they have begun implementing some of the students' recommendations.

It is a mutually beneficial arrangement, says Holland. NASA benefits from the hard work the students put into creating detailed plans for ways to evaluate NASA's educational and public outreach offerings, and the students benefit from communicating with a high-level organization about how such an organization functions. "For the Ed School, this transforms the theoretical into a concrete experience that has practical applications," Holland says.

Holland's course is designed to familiarize students with planning, reading, and applying program evaluations. Students are introduced to such topics as types of evaluation, the identification of stakeholders, the definition of goals for program evaluation, the documentation and development of program theory, the interaction between evaluation and policy and program development, the selection of appropriate measures, and the interplay of these variables when designing evaluations.

Part of Holland's course involves splitting the class into three groups and creating evaluations for an organization. Students work closely with partner organizations like NASA, the Milwaukee Public Schools, and the Arnold Arboretum in determining their evaluation needs. NASA has remained a bright spot in the course's partnerships.

Two years ago, Holland approached HGSE doctoral student Cassie Bowman, who works for a NASA contractor, and inquired about whether any NASA programs might be interested in collaboration. As it happened, HGSE offered something that two of NASA's programs had been looking for: insight into ways educationally-focused evaluation might be incorporated into their work.

While it may sound surprising that NASA would benefit from working with graduate students, Bowman explains that it's not that unusual. "I think in particular the [two NASA] programs involved are open to making improvements to what they're doing," Bowman says. "In any organization, even here at the Ed School, you can get so focused on what you're doing that it's hard to step back and take stock."

NASA's educational programs strive to enlighten and entice more students in areas of math and science. In the first year of the collaboration, Holland's students worked exclusively on elements within NASA's Mars Public Engagement, including the Mars Student Imaging Program (MSIP) and a mentoring program that partners high school students with scientists. In 2005, some of the HGSE students visited the MSIP site in Arizona to see the program in action. According to Bowman, the NASA programs are interested in better understanding what students and the public gain from their offerings and identifying changes that might improve products, activities, and opportunities. This is why the evaluations proposed by Holland's class are so important.

This past year, eight students focused on the Robotics Alliance Project, which engages students in math, science, engineering, and technology by providing online courses, Webcasts, access to competitions, and a Web-based clearinghouse of robotics-related educational material and activities. After a general interview with the program at the start of the class, the group conducted a second, focused interview with members of the Robotics Alliance Project staff, focusing on two components of the program: the robotics curriculum clearinghouse and online courses.

"We spent a good deal of time and a portion of the semester on the components and the rationale of the program," said John Bickar, Ed.M.'06. "The final piece was taking a look at how the program was actually implemented and doing a critical analysis to determine what type of evaluation would be best."

While students knew they would work with this NASA program as part of the class, it wasn't clear from the beginning that their work could actually be implemented. "At the beginning, many students didn't believe that our program would make use of it," Bowman says. "In both cases, we went in [to this collaboration] expecting to get something useable out of it, but we didn't have an idea of how much. We have been impressed by the quality and depth of ideas [of the students], so we've been really satisfied."

The idea that students' work is going to actually be used by the organization they are studying creates an interesting dynamic in the classroom. "It was very helpful in the learning process to have this hands-on work and see the reality that program evaluation is a lot messier than academic literature describes it," Bickar says. "It helps to drive and keep motivation up throughout all this work. You want to produce something of high quality because it's going to be used and not just put on a shelf."