What happens when peer reviews are not blind: Experimental design practice

[sue_frantz]

Here is another opportunity to give students practice in experimental design. While this discussion (synchronous or asynchronous) or assignment would work well after covering research methods, using it in the Intro Psych social psych chapter as a research methods refresher would work, too.

By way of introduction, explain to your students how double-blind peer review works and why that’s our preferred approach in psychology.

Next, ask students to read the freely-available, less-than-one-page article in the September 16, 2022 issue of Science titled “Reviewers Award Higher Marks When a Paper’s Author Is Famous.”

While I have a lot of love for research designs that involves an entire research team brainstorming for months, I have a special place in my heart for research designs that must have occurred to someone in the middle of the night. This study has to be the latter. If you know it is not, please do not burst my bubble.

A Nobel Prize winner (Vernon Smith) and his much lesser-known former student (Sabiou Inoua) wrote a paper and submitted it for publication in a finance journal. The journal editor and colleagues thought, “Hey, you know what would be interesting? Let’s find out if this paper would fly if the Nobel Prize winner’s name wasn’t on it. Doesn’t that sound like fun?”

The study is comprised of two experiments (available in pre-print). In the first experiment, the experimenters contacted 3,300 researchers asking if they would be willing to review an economics paper based on a short description of the paper. Those contacted were randomly assigned to one of three conditions. Of the one-third who were told that the author was Smith, the Nobel laureate, 38.5% agreed to review the paper. Of the one-third who were told that the author was Inoua, the former student, 28.5% agreed to review. Lastly, of the one-third who were given no author names, 30.7% agreed to review. Even though there was no statistical difference between these latter two conditions as reported in the pre-print, the emotional difference must be there. If I’m Inoua, I can accept that many more people are interested in reviewing a Nobel laureate’s paper than are interested in reviewing mine. What’s harder to accept is that my paper appears even less interesting when my name is on it than when my name is not on it. I know. I know. Statistically, there is no difference. But, dang, if I were in Inoua’s shoes, it would be hard to get my heart to accept that.

Questions for students:

1. In the first experiment, what was the independent variable? Identify the independent variable’s experimental conditions and control condition.
2. What was the dependent variable?

Now let’s take a look at the second experiment. For their participants, the researchers limited themselves to those who had volunteered to review the paper when they had not been given the names of either of the authors. They randomly divided this group of reviewers into the same conditions as in the first experiment: author identified as Vernon Smith, author identified as Sabiou Inoua, and no author name given. How many reviewers recommended that the editor reject the paper? With Smith’s name on the paper, 22.6% said reject it. With Inoua’s name on the paper, 65.4% said reject. With no name on the paper, 48.2% said reject. All differences are statistically significant. Now standing in Inoua’s shoes, the statistical difference matches my emotional reaction. Thin comfort.

Questions for students:

1. In the second experiment, what was the independent variable? Identify the independent variable’s experimental conditions and control condition.
2. What was the dependent variable?

The researchers argue that their data reveal a status bias. If you put a high status name on a paper, more colleagues will be interested in reviewing it, and of those who do review it, more will advocate for its publication. Double-blind reviews really are fairer, although the researchers note that in some cases—especially with pre-prints or conference presentations—reviewers may know who the paper belongs to even if the editor conceals that information. With this pair of experiments, the paper sent out for review was not available as a pre-print nor had it been presented at a conference.

The stickier question is how to interpret the high reject recommendations for Inoua as compared to when no author was given. While the experimenters intended to contrast Smith’s status with Inoua’s, there is a confounding variable. Reviewers who are not familiar with Sabiou Inoua will not know Inoua’s race or ethnicity for certain, but they might guess that Inouan is a person of color. A quick Google search finds Inoua’s photo and CV on Chapman University’s Economic Science Institute faculty and staff page. He is indeed a person of color from Niger.

Is the higher rejection rate for when Inoua’s name was on the paper due to his lower status? Or due to his race or ethnicity? Or was it due to an interaction between the two?

Questions for students:

Design an experiment that would test whether Inoua’s higher rejection rate was more likely due to status or more likely due to race/ethnicity.

1. Identify your independent variable(s), including the experimental and control conditions.
2. Identify your dependent variable.

If time allows, give students an opportunity to share other contexts where double-blind reviews are or would be better.