Menu choices: A lesson in external validity

[sue_frantz]

When I discuss experimental design in Intro Psych, I usually focus on the independent variable and dependent variable and how they are operationally defined as well as design considerations such as the experimenters and participants being blind to conditions. (I’ve read some articles where the authors say that the experimenters and participants were blinded. It brings me up short every time.) It wouldn’t hurt me to spend a little time talking about external validity, especially how we may sacrifice external validity in a lab study as a sort of proof of concept, and then follow up with a study that has more external validity. A recent JAMA article provides a nice illustration of how this can work—and gives students some experimental design practice.

After covering experimental design, describe this freely available study on fast food menu choices (Wolfson et al., 2022). The researchers hypothesized that the study “participants would be more likely to select sustainable options when viewing menus with positive or negative framing compared with control labels.” In an online questionnaire, 5,049 “[p]articipants were shown a fast food menu and prompted to select 1 item they would like to order for dinner.” The independent variable was menu labeling. “Participants were randomized to view menus with 1 of 3 label conditions: a quick response code label on all items (control group); green low–climate impact label on chicken, fish, or vegetarian items (positive framing); or red high–climate impact label on red meat items (negative framing).” The primary dependent variable was the number of participants who selected a menu item that was not red meat.

In the control condition, 49.5% of participants selected something other than red meat. In the positive framing condition (green labels on non-red meat items), 54.4% selected something other than red meat. In the negative framing condition (red labels on red meat items), 61.1% selected something other than red meat. All differences were statistically significant.

In the limitations section of the article, the researchers acknowledge that this study assessed hypothetical food purchases rather than actual food purchases. As such, the study lacks external validity. They also acknowledge that social desirability may have also influenced the results, but they think that the anonymity of the online study may have mitigated the effects. I’m less convinced. Participants may have been more likely to select non-red meat options partly to look like better people to themselves and partly because they guessed the hypothesis and wanted to help out the researchers.

In any case, this study found positive results that may be worth investigating further. The challenge for your students is to design a study that has greater external validity. How could the same research hypothesis be tested in real world conditions? Give students a couple minutes to think about this on their own and then ask students to discuss in small groups. What problems can students envision in conducting such a study? For example, would a local fast food restaurant be okay with putting red or green labels on their menu boards?

One last comment about social desirability. In a real fast food restaurant, if someone chose to order a green-labeled item, for the purpose of the hypothesis, does it matter if they ordered it because they wanted to have a positive impact (or less negative impact) on the planet, because they wanted to think of themselves as a good person, or because they wanted to look good to others?

References

Wolfson, J. A., Musicus, A. A., Leung, C. W., Gearhardt, A. N., & Falbe, J. (2022). Effect of climate change impact menu labels on fast food ordering choices among US adults: A randomized clinical trial. JAMA Network Open, 5(12), e2248320. https://doi.org/10.1001/jamanetworkopen.2022.48320