Life After P-hacking
Joe Simmons is an Associate Professor at the Wharton School of the University of Pennsylvania, where he teaches a course on Managerial Decision Making. He has two primary areas of research. The first explores the psychology of judgment and decision-making, with an emphasis on understanding and fixing the errors and biases that plague people’s judgments, predictions, and choices. The second area focuses on identifying and promoting easy-to-adopt research practices that improve the integrity of published findings. Joe is also an author of Data Colada, an online resource that attempts to improve our understanding of scientific methods, evidence, and human behavior, and a co-founder of AsPredicted.org, a website that makes it easy for researchers to properly pre-register their studies.
P-hacking is the practice of conducting many analyses on the same dataset until one achieves a reportable, statistically significant result (p < .05). P-hacking can lead researchers to believe in, and publish, findings that are false (i.e., not replicable), and it is likely the major reason why, in many fields (e.g., psychology, medicine), too many published findings do not replicate. Not surprisingly, the recognition that too many published findings do not replicate has led many different researchers to advocate for making many different changes to the way we report and conduct our research. In this talk, I discuss which of these proposed changes will effectively improve the way we do science and which will be ineffective or harmful. For example, I will conclude that pre-registration will do a lot of good, and that meta-analytic thinking will do a lot of harm.
Digital Reader: Resources Recommended by the Speaker
Nelson LD, Simmons J, Simonsohn U. 2018. Psychology’s Renaissance. Annual Review of Psychology, 69:511-534.