One strategy that has gained popularity for effectively implementing an active learning course design is a “flipped classroom’’ where students engage with the course material before coming to class. The literature on the effectiveness of using pre-class activities to improve learner outcomes is mixed. Consequently, we evaluated the use of pre-class activities in a new digital learning tool, Achieve, to investigate whether engaging in pre-class activities influenced assessment scores. We partnered with 40 instructors on the evaluation of pre-class activities in Achieve. Instructors chose whether or not to implement pre-class activities in their course, naturally categorizing students into “pre-class users” or “non pre-class users”. In total, 2,251 students consented to participate in the study (74% of the population), 1,372 engaged in pre-class activities and 879 did not. Groups were compared on three dependent variables: student likelihood to recommend a course using Achieve to a friend, summative assessment scores in Achieve, and in-course final exam scores. We also evaluated the effect of utilization of pre-class activities in Achieve on the dependent variables. Engagement in pre-class activities in Achieve had a significant effect on each of the dependent variables, even when prior academic performance, baseline level of motivation, and the instructor were controlled. We also found a small (based on Cohen’s classification), but significant relationship between the extent to which a student engaged in pre-class activities in Achieve and assessment scores. Finally, results uncovered evidence that instructors and students perceive pre-class activities in Achieve to contribute to positive student behaviors. And, there is evidence that use of pre-class activities significantly contributes to a student’s likelihood to recommend a course where Achieve will be used to a friend, but that it accounts for a small proportion of the variability in the outcome.