Students Using Macmillan Learning AI Tutor Show Improved Problem Solving Skills, Confidence and Engagement

[MarisaBluestone]

New York, NY; Dec 19 2024 — Macmillan Learning, a privately-held, family-owned digital learning company today shared early findings from two semesters of testing its AI Tutor within Achieve, their digital platform. The data from more than two million messages and surveys supported improved student confidence, learning habits, problem-solving skills, and classroom engagement. Macmillan Learning developed the AI Tutor as a purpose-driven AI study tool that deepens students’ understanding of their coursework rather than simply providing answers. 

The data represents a full year of research (Fall 2023 - Spring 2024) involving approximately 8,000 students, examining the Tutor’s impact on learning behavior and classroom engagement. Ongoing analysis for an additional 100,000 current students is showing early results that replicate the trends of the company’s earlier studies.

The tutor is an AI-powered personalized homework tool that supports students whenever they are doing work within the platform, guiding them through thought-provoking questions and dialogue, encouraging them to think critically. Designed with research-based guardrails, it helps students make connections and build the problem-solving skills that support persistence in their learning journey. The tutor is like an extension of an instructor or teaching assistant that guides them without judgment, or the stigma of asking basic questions, whenever they need support.

“We believe that students benefit from structured support that is available in the moment they need it and that respects their learning journey, encouraging them to persist when the material is difficult. Receiving customized support with appropriate guardrails to encourage development of problem-solving skills supports learning and confidence,” said Marcy Baughman, Vice President, Learning Science and Research.

The Data and Trends

Enhanced Student Confidence and Persistence

Many students reported feeling more confident after using the AI Tutor; in fact, data from a midpoint survey found that 41.5% of students reported increased confidence in their problem-solving abilities, describing it as a supportive, judgment-free space that encouraged persistence through challenging content. Instructors agree, and 41% attributed their students’ improved confidence or exam performance to the AI Tutor. “The AI Tutor is a welcome feature in my Calc II class,” said Oakton College Mathematics Professor Safa Hamed. “Calc II has a reputation that it is a very tough class. Knowing that the AI Tutor is available, getting stuck on an assignment is no longer as scary.”

“Our primary goal in developing the AI Tutor was to ensure it truly supported learning,” said Jaclyn Wiggins Camacho, Product Director. “We designed it to address actual student needs in a way that is constructive and intentional, so they develop confidence and persistence in problem-solving. It won’t give the answers, but will help direct students to where they can themselves discover them.”

Positive Shifts in Learning Habits and Classroom Dynamics

Designed to guide students through questions rather than give direct answers, the Tutor encourages students to work through problems critically and develop confidence in their understanding. Many students expressed appreciation for the Tutor’s patient, step-by-step guidance. 

Nearly half of instructors reported that students have been seeking help from them less with the use of the AI tutor, indicating that it supplements rather than replaces traditional resources.  One student noted, "After using it once or twice, I feel I better understand the concept and would no longer need help." 

Instructors noticed that, with the AI Tutor's support, students are arriving in class with a stronger foundational understanding, allowing for more advanced and enriching discussions during class and office hours. They believe the AI Tutor’s real-time guidance helps students tackle basic questions independently, and reduces homework-related emails and fewer requests for extensions. “It’s remarkable to see how students are engaging more deeply with content,” one instructor noted. “With fewer ‘quick-fix’ questions, I’m able to focus on higher-level discussions that enrich our class time.”

Better Preparation and Exam Readiness

Students using the AI Tutor engaged with it for an average session duration of 6.3 minutes, with some sessions lasting up to 30 minutes.  Approximately 52% of students reported that the AI Tutor helped them organize and clarify complex problems, contributing to their exam readiness. Many students indicated that the tutor's Socratic approach played a crucial role in helping them develop a structured approach to studying. 

Student Usage Insights 

Available around the clock, the AI Tutor is a reliable tool for students that need support outside of traditional class hours. Many students reach out to the AI Tutor during late-night study sessions and near assignment deadlines, with data showing that students use it the most from 6 p.m. to 3 a.m. on weeknights. One student remarked, “I feel like if this was a real person, I would be annoying them with so many questions, but the AI tutor is very patient and helps to go step by step."

Homework is consistently identified as the primary context in which students use the AI Tutor, particularly when they struggle with specific problems. Many instructors believe students are most likely to use the AI Tutor when preparing for quizzes or exams, as they want to ensure understanding before tests. 

“Knowing the Tutor is always available is reassuring for students, especially when they face complex assignments on their own time,” shared an instructor. “I’ve seen students grow more comfortable approaching challenging topics because they know they can get quality help right when they need it.”

The Tutor’s conversational, encouraging tone fosters a sense of comfort, with students sometimes addressing it informally: “bae i dont get this.” This reflects the safe, non-judgmental environment the Tutor creates, encouraging persistence in learning.

Interactions with the Tutor vary across disciplines. In Economics, students tend to write longer, in-depth questions, while in Chemistry, messages are shorter and more frequent, often focusing on specific calculations or quick checks (“multiply by 3?”). This pattern highlights the Tutor’s adaptability, providing personalized support that matches the needs of each student.

Nearly all students (90%) and 75% of instructors indicated they would be disappointed if the AI Tutor were no longer available. Instructors who initially had concerns about AI tools expressed that the Tutor’s thoughtful, guided approach alleviated their worries, making it a trusted resource in their teaching toolkit​.

About the Data:

The research from Fall 2023 - Spring 2024 was collected by analyzing students’ interactions with the AI Tutor. The data and insights come from product usage, user experience perceptions, and early insights from an impact study. This includes more than two million messages from more than 8,000 students, across over 80 unique courses, as well as surveys several times throughout each semester. To protect privacy, students' questions and surveys were collected anonymously. Approximately 30 instructors also provided feedback on their and their students' experiences.

The data and insights didn’t vary widely from fall to spring, but did help the company's product and research teams better refine the AI Tutor to improve usability, accuracy and support. Additional insights from millions of messages from around 100,000 students will be shared in 2025. 

The impact study took place over one semester and was conducted by the company’s Learning Science and Research team. The team is focused on data- driven insights that analyze course outcomes coupled with demographic student information and platform usage data, enabling a more nuanced understanding of educational impact. Committed to co-design, the team works closely with educators, administrators, and students to ensure their research aligns with real-world educational needs and contexts​​. Their rigorous research is approved by a third-party Institutional Review Board (IRB) and then approved at individual institutions. 

The AI Tutor is available for courses that use STEM and Economics course materials within Achieve. 

About Macmillan Learning

Macmillan Learning is a privately-held, family-owned company that inspires what’s possible for every learner. We envision a world in which every learner succeeds. Through our content, tools and services, we aim to make that a reality. To learn more, please visit macmillanlearning.com or join our Community.