Study finds AI tutor helps college students learn more than active learning classes

A new study published in Scientific Reports reports that college students learned significantly more in less time when using an AI-powered tutor compared to participating in an in-class active learning session. The researchers also found that students felt more engaged and motivated while using the AI tool. This finding comes from a randomized, controlled trial that measured both learning outcomes and student perceptions in an authentic educational setting.

The study involved 316 students who were divided into two groups. Each group experienced two lessons with distinct teaching methodologies in consecutive weeks. In the initial week, one cohort worked on an AI-assisted lesson from their homes, whereas the second group took part in an active learning session within the classroom. The conditions were reversed the following week. The custom AI tutor was designed using the same pedagogical best practices as the in-class lessons. To ensure accuracy, the researchers did not rely solely on GPT-4 to generate solutions; instead, they enriched the prompts with comprehensive, step-by-step answers to guide the AI tutor in delivering high-quality explanations. The platform was built to guide students sequentially through each part of a problem because a system prompt alone could not reliably provide enough structure for multi-part problems.

Students in the AI group achieved a higher median post-score of 4.5, compared to a median of 3.5 for the in-class active learning group. The median learning gains for the AI-tutored group were over double those of the in-class group. A statistical test confirmed a significant difference in post-scores between the two groups. In terms of time, the median time on task for the AI group was 49 minutes, with 70% of students spending less than 60 minutes on the task. A linear regression model suggested an effect size of 0.63, which the authors note is an underestimation due to a ceiling effect; a quantile regression provided an effect size ranging from 0.73 to 1.3 standard deviations. There was no correlation between the time spent on the task and the students’ post-test scores. On the topic of views, learners assigned to the AI condition showed stronger agreement with claims concerning motivation and engagement compared to peers in the in-class setting. However, there was no statistically significant difference between the groups in their agreement with statements relating to enjoyment and growth mindset. Furthermore, 83% of the class indicated that the AI tutor’s clarifications matched or surpassed the quality of explanations given by human teachers.

While this study shows positive results for college students, other reports highlight different experiences. Rice told a University of Maryland publication: “AI tools could exacerbate inequities in learning outcomes,” Rice told a University of Maryland publication . “That is particularly troubling to me.” In a different setting, merely 15 percent of pupils in AI-enhanced courses ever utilized the assistant; among these users, the majority sought direct solutions instead of leveraging its instructional capabilities. One recent report noted that one AI tutor engaged middle schoolers, while another backfired with college students. Diane Tavenner stated: "Specifically RCT." Michael Horn stated: "I think another little bugaboo people had was like, there’s no academic study or RCT or something like that." Tavenner also mentioned: "you have 400 schools, many different goals, and many different ways of implementing or starting or spreading".