Better Feedback with AI?

The rise of artificial intelligence (AI) has made an immediate and lasting impact on the educational landscape.

Concerns about academic resilience amid the proliferation of large language models has been a primary concern for educators, with some worried that AI will become a tool to circumvent — not enhance — learning. But researchers are also exploring how AI can be used by instructors to improve their own performance in certain circumstances.

In a recent study, researchers explored the impact the large language model GPT-3 could have on educators giving feedback to struggling students. Led by Associate Professor Bertrand Schneider and Ph.D. student Gahyun Sung, researchers explored “what tasks AI could and could not do well in an educational context” and provided lessons to support future AI-human partnerships.

The research, based in a graduate school environment, focused on a specific classroom setting: a makerspace — a physical, project-based environment in which providing feedback to all students in real time can be challenging.

“A makerspace is a place where students need constant feedback, and often struggle to feel like they belong,” says Schneider. 

Unlike past forms of automated feedback that’s “formulaic and closed,” the goal of the writing platform developed by the researchers was to use GPT-3 to “write humanlike feedback that can deliver information and be encouraging at the same time,” says Schneider. In addition to crafting encouraging feedback for struggling students, it also summarized the students’ blog posts detailing their projects.

What AI does well:

Researchers found that GPT-3 was able to do three things well within the context of teacher feedback:

1. Use supportive language to appreciate projects
2. Recognize the work put into these projects
3. Value student strategies used in makerspaces

The report notes “clear value” in summarizing student reflections on their own work, giving instructors a “concise and accurate” overview of student progress. Those summaries were strong enough to add directly to emails providing feedback to the students.

What AI does not do well:

GPT-3 failed, however, to offer positive feedback for struggling students, according to the study. Students self-reporting high stress levels or low levels of enjoyment sometimes perceived the feedback to be unhelpfully short and insensitive.

Lessons Learned

Schneider, Sung, and co-author Lenore Guillian of Taskbase, noted they learned important lessons regarding AI use moving forward. 

1. Human input when using AI to create feedback is an “indispensable, limited resource,” say the researchers. The more thorough the student responses — and larger the overall class size — the better ChatGPT could gather information and create better overall feedback.
2. An instructor must understand not just what AI can do well, but also where it lacks. Writing better prompts and focusing on ways AI automation may not replace instructor interaction can help utilize the tool’s overall impact.
3. Feedback quality depends on how much information the model possesses about each student. The researchers hope to increase the accuracy and relevance of AI-generated feedback by using camera-based location data from the makerspace to better understand how students work. They also aim to create better prompts with the input of students and instructors to better develop the platform’s effectiveness and, according to Schneider, “create a fluid collaboration between humans and AI.”

“As educators, we want to think about how the future generation can work alongside AI, and develop capacities that allow the critical usage of AI technologies,” says Schneider. “Embracing AI in the classroom is a way for teachers to gain firsthand understanding, and model its use for learners.”