Today, 270 researchers investigating the reproducibility of psychological science published their findings in Science Magazine. Launched nearly four years ago and coordinated by the Center for Open Science, the Reproducibility Project: Psychology has produced the most comprehensive investigation ever done about the rate and predictors of reproducibility in a field of science. The project conducted replications of 100 published findings of three prominent psychology journals. They found that regardless of the analytic method or criteria used, fewer than half of their replications produced the same findings as the original study. Mallory Kidwell, one of the project coordinators from the Center for Open Science, concluded, “The results provide suggestive evidence toward the challenges of reproducing research findings, including identifying predictors of reproducibility and practices to improve it.”
Science is unique from other ways of gaining knowledge by relying on reproducibility to gain confidence in ideas and evidence. Reproducibility means that the results recur when the same data are analyzed again, or when new data are collected using the same methods. As noted by Angela Attwood, team member from University of Bristol, “Scientific evidence does not rely on trusting the authority of the person that made the discovery. Rather, credibility accumulates through independent replication and elaboration of the ideas and evidence.”
The team emphasized that a failure to reproduce does not necessarily mean the original report was incorrect. Elizabeth Gilbert, team member from the University of Virginia, noted, “A replication team must have a complete understanding of the methodology used for the original research, and shifts in the context or conditions of the research could be unrecognized but important for observing the result.” Belen Fernandez-Castilla, team member from Universidad Complutense de Madrid, added, “Scientists investigate things that are not yet understood, and initial observations may not be robust.”
Yet a problem for psychology and other fields is that incentives for scientists are not consistently aligned with reproducibility. “What is good for science and what is good for scientists are not always the same thing. In the present culture, scientists’ key incentive is earning publications of their research, particularly in prestigious outlets,” said Ljiljana Lazarević, team member from the University of Belgrade. Research with new, surprising findings is more likely to be published than research examining when, why, or how existing findings can be reproduced. As a consequence, it is in many scientists’ career interests to pursue innovative research, even at the cost of reproducibility of the findings.
Fewer than half of the original findings were successfully replicated. This held true across multiple different criteria of success. The team noted three basic reasons this might occur: 1) Even though most replication teams worked with the original authors to use the same materials and methods, small differences in when, where, or how the replication was carried out might have influenced the results. 2) The replication might have failed to detect the original result by chance. 3) The original result might have been a false positive. Johanna Cohoon, another Project Coordinator from the Center for Open Science, concluded that, “The findings demonstrate that reproducing original results may be more difficult than is presently assumed, and interventions may be needed to improve reproducibility.” In keeping with the goals of openness and reproducibility, every replication project posted its methods on a public website, and later added their raw data and computer code for reproducing their analyses.
Many organizations, funders, journals, and publishers are already working on improving reproducibility. For example, in 2014, the journal Psychological Science - one of the journals included in this study - implemented practices such as badges to acknowledge open sharing of materials and data to improve reproducibility. “Efforts include increasing transparency of original research materials, code, and data so that other teams can more accurately assess, replicate, and extend the original research, and pre-registration of research designs to increase the robustness of the inferences drawn from the statistical analyses applied to research results,” said Denny Borsboom, a team member from the University of Amsterdam who was also involved in creation of the Transparency and Openness Promotion (TOP) Guidelines published in Science in June.
Since the Reproducibility Project began in 2011, similar projects have emerged in other fields such as the Reproducibility Project: Cancer Biology. And, a discipline of metascience is emerging - scientific research about scientific research. These and the widespread efforts to improve research transparency and reproducibility are indications that, as suggested by team member Susann Fiedler from the Max Planck Institute for Research on Collective Goods, “Science is actively self-examining and self-correcting to maximize the quality and efficiency of the research process in the service of building knowledge for the public good.”
The Reproducibility Project: Psychology is funded by the Laura and John Arnold Foundation.
The Center for Open Science (COS) is a non-profit technology startup founded in 2013 with a mission to increase openness, integrity, and reproducibility of scientific research. COS pursues this mission by building communities around open science practices, supporting metascience research, and developing and maintaining free, open source software tools. The Open Science Framework (OSF), COS’s flagship product, is a web application that connects and supports the research workflow, enabling scientists to increase the efficiency and effectiveness of their research. Researchers can use the OSF to collaborate, document, archive, share, and register research projects, materials, and data. Learn more at cos.ioand osf.io, or follow us on Twitter @OSFramework.
The Reproducibility Project: Psychology Represents the collective efforts of 270 researchers. The following is a selection of contacts for the project to facilitate connection with local or regional researchers that can speak about the project, findings, and implications.
Patrick Goodbourn, firstname.lastname@example.org
Martin Voracek, email@example.com
Wolf Vanpaemel, firstname.lastname@example.org
Štěpán Bahník, email@example.com
Susann Fiedler, firstname.lastname@example.org
Cathy O.-Y. Hung, email@example.com
Samuel Shaki, Samuel_shaki@hotmail.com
Michelangelo Vianello, firstname.lastname@example.org
Alissa Melinger, email@example.com
Ljiljana Lazarević, firstname.lastname@example.org
Gustav Nilsonne, email@example.com
Alejandro Vasquez Echeverria, firstname.lastname@example.org