Center for Open Science receives a grant from the James S. McDonnell Foundation to study the impact of Registered Reports

The Center for Open Science (COS) is pleased to announce that it has received a $165,591 grant from the James S. McDonnell Foundation to undertake two studies evaluating the impact of Registered Reports (RRs) on research quality and outcomes. RRs were introduced in 2013 as an innovative method for improving reproducibility. RRs clarify the distinction between confirmatory and exploratory research, and eliminate a variety of questionable research practices, including low statistical power, selective reporting of results, and publication bias. Simultaneously, RRs preserve complete flexibility to report serendipitous findings as exploratory findings.

In a RR, authors submit their research plans for journal peer review prior to knowing the study outcomes and receive an in-principle commitment from the journal to publish the results as long as the work is performed as proposed with evidence of competent execution and interpretation. While it seems straightforward to make a theoretical case for how the use of RRs improve the robustness and reliability of research findings there is no meaningful evidence about whether RRs improve and accelerate knowledge accumulation compared to the typical models for peer review and publication.

In the first study, COS’s Metascience team will conduct a comparison of citation impact between published RRs and comparison articles from the same journals using a variety of metrics. The second study will conduct a blind evaluation of those same articles for research quality and outcomes. Evaluators will assess the research question, methods, and results of the blinded papers on a variety of criteria including creativity and importance of the research question, the quality of the research methodology, the nature and importance of the results, and appropriateness of the interpretation/conclusions. The results will provide an initial, naturalistic assessment of the quality and outcomes of RRs compared to other published articles.

“Moving the initial peer review and publishing commitment earlier changes many incentives,” said Tim Errington, who leads the Metascience program for COS. “While the logical case for RRs is very strong, especially in the context of empirical evidence of the shortcomings of the present publication model, such as p-hacking and publication bias, empirical evidence is essential for maximizing the quality of RR implementation.”

“JSMF is pleased to be contributing to the important work of the Center for Open Science,” added Susan Fitzpatrick, JSMF President. “When changes are made to the existing norms and practices of academic science, the interventions should be studied scientifically. Documenting the effects of registered reports on academic publishing will yield important insights for the future of scholarly communication.”

These two studies began on September 1, 2017, will occur concurrently and will be completed over 16 months. They will be published as Registered Reports.

###

About the James S. McDonnell Foundation

Founded in 1950 by aerospace pioneer James S. McDonnell, the Foundation was established to "improve the quality of life," and does so by contributing to a generation of new knowledge through its support of research and scholarship. Established as the McDonnell Foundation and renamed the James S. McDonnell Foundation in 1984 in honor of its Founder, the Foundation has pursued its goals by supporting scientific, educational and charitable causes locally, nationally, and internationally.

Contacts for the James S. McDonnell Foundation

Media: Susan M. Fitzpatrick, Ph.D., President: info@jsmf.org | 314-721-1532 

Web: https://www.jsmf.org

About Center for Open Science

The Center for Open Science (COS) is a non-profit technology organization with a mission to increase openness, integrity, and reproducibility of scientific research. COS pursues this mission by building communities around open science practices, supporting metascience research, and developing and maintaining free, open source software tools. The Open Science Framework (OSF), COS’s flagship product, is a web application that connects and supports the research workflow, enabling scientists to increase the efficiency and effectiveness of their research. Researchers use the OSF to collaborate, document, archive, share, and register research projects, materials, and data. Learn more at cos.io and osf.io.

Contacts for the Center for Open Science

Media: Rusty Speidel: rusty@cos.io | 434-284-3403

Registered Reports: David Mellor: david@cos.io