The Reproductibility Project and Textbook Reporting of Psychological Science

0 0 738

Originally posted on October 20, 2015.

In response to the big “Reproducibility Project” news that only 36 percent of a sample of 100 psychological science studies were successfully replicated, psychologists have reassured themselves that other fields, including medicine, also have issues with reproducibility. Moreover, differing results sometimes illuminate differing circumstances that produce an effect.

Others have agreed on a lesson for textbook authors. “A finding is not worth touting or inserting in the textbooks until a well-powered, pre-registered, direct replication is published,” argues Brent Roberts. “The conclusions of textbooks should be based not on single studies but on multiple replications and large-scale meta-analyses,” advise Wolfgang Stroebe and Miles Hewstone.

Those are high standards that would preclude textbook authors reporting on first-time discoveries, some of which are based on big data. Ironically, it would even preclude reporting on the one-time Reproducibility Project finding (can it be replicated?). Even so, my introductory psychology co-author, Nathan DeWall, and I are cautious about reporting single-shot findings. Some intriguing new studies end up not in our texts but in our next-edition resource files, marked “needs a replication.” And we love meta-analyses, which give us the bigger picture, digested from multiple studies.


So, I wondered: How did we do? How many of the nonreproducible studies ended up in Psychology, 11th Edition? Checking the list, my projects manager, Kathryn Brownson, found three of the 100 studies in our bibliography—one of which successfully replicated, one of which produced insufficient data for a replication, and one of which failed to replicate.

Thus, from page 504:

In several studies, giving sugar (in a naturally rather than an artificially sweetened lemonade) had a sweet effect: It strengthened people’s effortful thinking and reduced their financial impulsiveness (Masicampo & Baumeister, 2008; Wang & Dvorak, 2010).

will likely become:

In one study, giving sugar (in a naturally rather than an artificially sweetened lemonade) had a sweet effect: It reduced people’s financial impulsiveness (Wang & Dvorak, 2010).

Ergo, out of 5174 bibliographic citations, one citation—and its five associated text words—will end up on the cutting room floor.

Tags (2)
About the Author
David Myers has spent his entire teaching career at Hope College, Michigan, where he has been voted “outstanding professor” and has been selected by students to deliver the commencement address. His award-winning research and writings have appeared in over three dozen scientific periodicals and numerous publications for the general public. He also has authored five general audience books, including The Pursuit of Happiness and Intuition: Its Powers and Perils. David Myers has chaired his city's Human Relations Commission, helped found a thriving assistance center for families in poverty, and spoken to hundreds of college and community groups. Drawing on his experience, he also has written articles and a book (A Quiet World) about hearing loss, and he is advocating a transformation in American assistive listening technology (see