The Reproductibility Project and Textbook Reporting of Psychological Science

[david_myers]

Originally posted on October 20, 2015.

In response to the big “Reproducibility Project” news that only 36 percent of a sample of 100 psychological science studies were successfully replicated, psychologists have reassured themselves that other fields, including medicine, also have issues with reproducibility. Moreover, differing results sometimes illuminate differing circumstances that produce an effect.

Others have agreed on a lesson for textbook authors. “A finding is not worth touting or inserting in the textbooks until a well-powered, pre-registered, direct replication is published,” argues Brent Roberts. “The conclusions of textbooks should be based not on single studies but on multiple replications and large-scale meta-analyses,” advise Wolfgang Stroebe and Miles Hewstone.

Those are high standards that would preclude textbook authors reporting on first-time discoveries, some of which are based on big data. Ironically, it would even preclude reporting on the one-time Reproducibility Project finding (can it be replicated?). Even so, my introductory psychology co-author, Nathan DeWall, and I are cautious about reporting single-shot findings. Some intriguing new studies end up not in our texts but in our next-edition resource files, marked “needs a replication.” And we love meta-analyses, which give us the bigger picture, digested from multiple studies.

So, I wondered: How did we do? How many of the nonreproducible studies ended up in Psychology, 11th Edition? Checking the list, my projects manager, Kathryn Brownson, found three of the 100 studies in our bibliography—one of which successfully replicated, one of which produced insufficient data for a replication, and one of which failed to replicate.

Thus, from page 504:

In several studies, giving sugar (in a naturally rather than an artificially sweetened lemonade) had a sweet effect: It strengthened people’s effortful thinking and reduced their financial impulsiveness (Masicampo & Baumeister, 2008; Wang & Dvorak, 2010).

will likely become:

In one study, giving sugar (in a naturally rather than an artificially sweetened lemonade) had a sweet effect: It reduced people’s financial impulsiveness (Wang & Dvorak, 2010).

Ergo, out of 5174 bibliographic citations, one citation—and its five associated text words—will end up on the cutting room floor.