Do Replication Failures Discredit Psychological Science?

[david_myers]

Psychological science has taken some body blows of late, with famous findings challenged by seeming failures to replicate.

The problem isn’t just that prolific researchers Brian Wansink and Derek Stapel faked data, or that David Rosenhan (of “On Being Sane in Insane Places” fame) and personality researcher Hans Eysenck have been accused of doing likewise. Every discipline has a few self-promoting deceivers, and more who bend the truth to their side.

And it’s not just critics arguing (here and here) that a few celebrated findings, such as the tribalism of the Stanford Prison and Robbers Cave experiments, were one-off, stage-managed happenings. Or that some findings of enormous popular interest—brain training for older folks, implicit bias training programs, or teaching to learning styles—all produce little enduring benefit.

The problem is that other findings have also not been consistently reproducible.  The effects of teachers’ expectations, power posing, willpower depletion, facial feedback, and wintertime depression (seasonal affective disorder) have often failed to replicate or now seem more modest than widely claimed.

Moreover, the magnitude and reliability of stereotype threat, growth mindset benefits, and the marshmallow test (showing the life success of 4-year-olds who can delay gratification) are, say skeptics, more mixed and variable than often presumed.

Hoo boy. What’s left? Does psychology’s knowledge storehouse have empty shelves? Are students and the public justifiably dismayed? As one former psychology student tweeted: “I took a [high school] psychology class whose entire content was all of these famous experiments that have turned out to be total horse**bleep**. I studied this! They made me take an exam! For what?” To which others responded:

• “I'm putting all my chips on neuroscience, I refuse to listen to psychologists ever again, they had their chance.”
• “Imagine if you'd spent 10 years getting a PhD in this stuff, going into $200k in debt.”
• “You can learn more from life never mind a psychology lesson just take a look around fella.”
• “I have a whole damn degree full of this @#$%.”

But consider:

• How science works. Yes, some widely publicized studies haven’t replicated well. In response to this, we textbook authors adjust our reporting. In contrast to simple common sense and to conspiracy theories, science is a self-checking, self-correcting process that gradually weeds out oversimplifications and falsehoods. As with mountain climbing, the upward march of science comes with occasional down slopes.
• Some phenomena are genuine, but situation specific. Some of the disputed phenomena actually have been replicated, under known conditions. One of my contested favorite experiments—the happy pen-in-the-teeth vs. pouting pen-in-the-lips facial feedback effect—turns out to replicate best when people are not distracted by being videotaped (as happened in the failure-to-replicate experiments). And stepping back to look at the bigger picture, the Center for Open Science reports that its forthcoming analysis of 307 psychological science replications found that 64 percent obtained statistically significant results in the same direction as original studies, with effect sizes averaging 68 percent as large. The bottom line: Many phenomena do replicate.
• What endures and is left to teach is . . . everything else. Memories really are malleable. Expectations really do influence our perceptions. Information really does occur on two tracks—explicit and implicit (and implicit bias is real). Partial reinforcement really does increase resistance to extinction. Human traits really are influenced by many genes having small effects. Group polarization really does amplify our group differences. Ingroup bias really is powerful and perilous. An ability to delay gratification really does increase future life success. We really do often fear the wrong things. Sexual orientation really is a natural disposition that’s neither willfully chosen nor willfully changed. Split-brain experiments really have revealed complementary functions of our two brain hemispheres. Electroconvulsive therapy really is a shockingly effective treatment for intractable depression. Sleep experiments really have taught us much about our sleeping and dreaming. Blindsight really does indicate our capacity for visual processing without awareness. Frequent quizzing and self-testing really does boost students’ retention. But enough. The list of repeatedly confirmed, humanly significant phenomena could go on for pages.

So, yes: Let’s teach the importance of replication for winnowing truth. Let’s separate the wheat from the chaff. Let’s encourage critical thinking that’s seasoned with healthy skepticism but not science-scorning cynicism. And let us also be reassured that our evidence-derived principles of human behavior are overwhelmingly worth teaching as we help our students appreciate their wonder-full world.

(For David Myers’ other essays on psychological science and everyday life, visit TalkPsych.com; follow him on Twitter: @DavidGMyers.)

[ScottL]

The discipline of scientific psychology is largely based upon large samples and p values. There are inherently arbitrary assumptions. To put it much more simply than it really is, the "facts" of psychology rest on phenomena that due to chance.

There are single subject designs that do not rest on this foundation. I haven't made a study of this but I believe that those results are much more replicable.

Something to think about...