Crisis or Common Sense? Two Ways to Approach Scientific Replication

[nathan_dewall]

Originally posted on September 24, 2015.

This past weekend, I gave myself an odd birthday present. I entered an ultramarathon. If you’ve read my posts, you know I like to run. For my birthday, I wanted to run 100 miles as fast as I could. Luckily, I had a perfect opportunity. There was a 24 hour running race within driving distance of my house.

There was a bigger purpose in my run. I could determine whether a recent test of my speed and endurance would replicate. Two weeks ago, I ran 100 miles in 22 hours and 10 minutes.

Replication is important. It tells whether repeating the essence of an experiment will produce the same result. The more the same sequence of events produces a similar outcome, the more we can depend on it. 

Psychology is embroiled in a current debate about replicability. All psychologists agree that replication is important. That is a requirement before you get your card when you join the psychologist club. The debate centers on the meaning of non-replication. A recent report found that 64 percent of the tested psychological effects did not replicate. Some have declared a war on current scientific practices, hoping to inch the non-replication rate down to a less newsworthy percentage. Others, such as Lisa Feldman Barrett, argue that non-replication is a part of science. It tells us just as much about why things do happen as to why they don’t.

My birthday run had everything I needed to make a replication attempt. Nearly everything was identical to the last time I ran 100 miles. The course consisted of a flat, concrete loop that was nearly one mile long. I ate the same foods, drank the same amount of water, and got the same amount of sleep the night before. All signs pointed to an exact replication.

Then the race started. The first 50 miles breezed by. I was over an hour faster than my previous run, but I felt pretty good. By mile 65, I was mentally fatigued. By mile 70, my body was exhausted. By the time I hit mile 75, I was done. Less than 16 hours had passed, but I was mentally and physically checked out. No replication.

There are at least two ways I can deal with this non-replication. The first is to panic. Either the people who counted my laps at the previous race did something wrong, I reported something wrong, or something else is wrong. It is as if it never happened. The next time someone asks me my personal record, I can tell them. But I must tell them that I don’t trust it. “Probably just a one-off,” I might say. “Tried to replicate it two weeks later and came up short.”

A second approach is to try to understand what contributed to the non-replication. Most things were the same. But some things were different, among them the wear and tear that long running has on the body and mind. Maybe I wasn’t fully recovered from the previous race. Maybe I ran too fast too soon. Or maybe I’m just not that fast.

Either way, it tells us a different story about replication. Replication science is possible, but we will always have non-replications. And those non-replications aren’t badges of shame. They tell us as much about the complexity of human psychology as the truth about how certain situations make us think, feel, and act.

It would be great if psychology’s non-replication rate dwindled to less than 5 percent. I doubt that will ever happen. Humans are squirrely animals. No matter how much we want to do the same thing twice, sometimes it doesn’t happen.