Minneapolis & New York, November 13, 2019 -- Macmillan Learning, a privately-held, family-owned education publishing company, announced today during the American Evaluation Association (AEA) Annual Conference that student final exam scores were more than half-letter grade higher when instructors assigned pre-class activities in Achieve, the company’s new digital learning platform. The company partnered with instructors at 38 two- and four-year institutions to research the efficacy of Achieve, evaluating 2,251 students during the spring 2019 semester. The results revealed today represent one finding from a multi-year efficacy study; other efficacy results are expected to be published later this year. The Achieve platform launches in January 2020 and will be widely available in Fall.
“Macmillan Learning is deeply committed to using evidence-based research to build products that help more students to succeed. To best accomplish this, the company took the unusual step of embedding efficacy research for Achieve into the process of designing, developing and improving the new platform,” said Adam Black, PhD, Macmillan Learning’s Chief Learning Officer.
Key Findings
Use of pre-class activities in Achieve positively influenced summative assessment scores in Achieve and also final exam scores—even when controlling for students’ prior academic performance, baseline level of motivation to succeed in the course, and instructor characteristics. Among the findings:
-
Students who completed pre-class activities earned, on average, 8.7 percentage points higher on assessments. There were statistically significant differences (p<0.001) in the average assessment scores for students using pre-class assignments in Achieve compared to those who did not.
-
Students who completed pre-class activities had final exam scores that were 6.8 percentage points higher than students who didn’t. These values were also statistically significant (p<0.001)
-
The more pre-class activities that a student engaged in, the higher their assessment scores and final exam scores were. These values were also statistically significant (correlation = 0.71, p<0.001)
The study found that instructors also had strong perceptions of how pre-class activities in Achieve supported their students. They reported that pre-class activities helped their students to stay on track with assigned reading, understand the concepts that would be covered in class, and gain a basic understanding of concepts. Students reported high perceptions of pre-class activities in Achieve as well. Like their instructors, students agreed that the pre-class work helped them keep up with their reading, come to class prepared to participate, and support their comprehension.
“We’re particularly excited to provide instructors with deeper insights into how particular student subgroups benefit from using it – for example, those less motivated or academically prepared to succeed. These kinds of insights are only possible because of our partnerships with instructors and students and the rigorous research practices we employ, which include two levels of Institutional Review Board approval for every study of Achieve,” said Kara McWilliams PhD., Macmillan Learning’s VP of Impact Research and the study’s Primary Investigator.
About The Research
Because the study was conducted with students, Macmillan Learning's processes, data handling, and researcher credentials were reviewed and approved by the Institutional Review Boards (IRB) at each institution and a third party. IRB review ensures that the study is rigorous and ethical. Forty instructors elected to participate in the study and their students were offered the opportunity to opt-in, and 74% of students chose to participate.
Groups were compared on three dependent variables: scores on assessments in Achieve, scores on final exams given in the course where Achieve was used, and student’s likelihood to recommend a course using Achieve to a friend. The effect of using pre-class activities in Achieve on the dependent variables was also evaluated. Analyses indicated that students’ high school grade point average and that their baseline levels of motivation were related to the dependent variables, so they were controlled for as covariates. And, students were grouped within instructors so hierarchical linear regression was used to investigate the within and between group differences, where required.
This research has been reviewed by Macmillan Learning’s Impact Research Advisory Council (IRAC), which is comprised of experts in educational technology, methodology, and psychometrics.
“Having had the privilege of participating in Macmillan’s peer review process for The Flipped Effect, I have had a chance to see first-hand how seriously they are taking this critical part of their process. This company is collaborating with serious academics to make sure that the claims they make about their product are true. Their commitment is a model for the sector," said Michael Feldstein, IRAC member and Executive Director, Empirical Educator Project.
The research presented today is part of a larger body of research conducted by Macmillan Learning’s Learning Science team that began in 2017 and helped guide development of the Achieve platform. The team has been studying its efficacy overall, and among subgroups of students to evaluate whether the tool supports all students - like those more and less academically prepared to succeed, more and less motivated to succeed, first generation students, and those with competing demands like jobs and families, among other important cohorts.
Macmillan Learning also presented during the conference “The formative evaluation of Achieve: a novel approach to rapid-cycle evaluation of digital learning tools early in development.” They shared an entirely new research process to test and improve learning products outside of live courses during early development. These methods can be used by other organizations to improve products, to begin to understand chosen implementation patterns, and to glean exploratory evidence of learner effectiveness.
About Achieve
Achieve is an evolutionary digital learning platform that includes learning materials with a comprehensive set of interconnected teaching and assessment tools. It offers the best features of each of Macmillan Learning’s digital solutions in one platform that is intuitive to use, accessible for students of all abilities, and is flexible for students and instructors. Achieve was developed in partnership with students and instructors with the goal of supporting students of all levels motivation and preparedness and helping to engage students in and out of class so that they have better outcomes. To that end, instructors facilitate can learn in the way that best suits their class, whether it’s traditional, online, blended, or fully “flipped” classrooms.
About Macmillan Learning
Macmillan Learning is a privately-held, family owned company that improves lives through learning. By linking research to learning practice, we develop pioneering products and learning materials for students that are highly effective and drive improved outcomes. Our engaging content is developed in partnership with the world's best researchers, educators, administrators, and developers. To learn more, please visit macmillanlearning.com or see us on Facebook, Twitter, LinkedIn or join our Macmillan Community.
