Just over three years ago, we launched the Prereg Challenge with support from the Laura and John Arnold Foundation as an education campaign designed to initiate preregistration as a regular habit prior to conducting a study. The impact of this campaign has been substantial. It has helped mainstream the practice of preregistration within the psychological science research community, and has introduced preregistration to dozens more disciplines. The tangible impacts of this campaign fall into three areas:
The primary goal of the Prereg Challenge was to initiate and support adoption of a new behavior within the research community: to preregister the study and data analysis plans before conducting a study. With the exception of pre-clinical research, few scientists preregistered their work prior to this project. For more than 70% of Challenge participants, this was the first time they preregistered their research. To support successful completion of these initial preregistrations we dedicated ourselves to help researchers navigate this new behavior. Most applicants integrated feedback and were approved after one resubmission with guidance from COS staff. Entrants could also seek guidance from our free statistics and methods consulting service, funded by LJAF. Preregistration was also included in OSF and reproducible practices webinars and workshops offered by our training team. In addition, the Prereg Challenge likely encouraged preregistration more broadly, as the number of all preregistrations placed on OSF increased substantially, doubling each year from the start of the Prereg Challenge.
An enduring impact of the Prereg Challenge will be the knowledge base and infrastructure built to sustain it. The expertise that we gained through the process of reviewing over 3,000 preregistrations allowed us to compile a holistic understanding of how the practice can be best implemented. COS applied those lessons to many different research practices (see the Preregistration Revolution for examples). The resulting educational resources includes peer reviewed commentaries, blogs, FAQs, practical tips for preregistering different study designs, webinars, lessons plans, and presentations. These comprehensive resources are freely available for use by the scientific community. This page includes information on addressing reviewer comments that undermine the benefits of preregistration, the importance of documenting deviations from the preregistered plan in a transparent way, and how to plan for complex study designs or longitudinal work.
Adoption of preregistration required supporting infrastructure and a means to publicly share registrations. Building the infrastructure on OSF required nearly a year of research and development across the organization to design and build this dedicated workflow, perform quality testing, and create training content to support users as soon as it was released. The resulting workflow is available on the OSF and enables thorough preregistrations that can be saved, edited before submission, reviewed by experts, revised for completeness, and persistently stored in the registry. This major feature of OSF was substantially improved over the course of the campaign and was even adapted for two related projects, one in political science (see the Election Research Preacceptance Competition) and the other for Registered Report submissions. The Registered Report workflow was created in response to external evaluation (OA here) that noted opaque practices in some early registered reports.
The body of research produced by those who have published the results of their preregistered work is impressive (see recent prize winners and this Zotero collection). The research spans several disciplines, including: social psychology, cognitive psychology, neuroscience, developmental psychology, political science, ecological and evolutionary biology, meta-science, and a few examples from other social science disciplines and the life sciences. This research came from scholars at every career stage.
Is the resulting research “better” than it would have been without preregistration? We have reason to believe it is because:
The most substantial challenge with this campaign was the lag time between conducting research and having it published. Even with a relatively high rate of final publication, studies often spend years in review through multiple journals. We were optimistic in our ability to quickly attract thousands of researchers within the first year in order to have sufficient time to have the final work published. As a confirmation of an intuitive speculation, some of the fastest times from preregistration to publication were with articles that used Registered Reports, as the substantial peer review occurred early on and there was no need to “shop” a paper around to multiple journals.
A second challenge was that reviewing submitted preregistrations for completeness and adherence to the campaign requirements was a larger time commitment than expected because of two concerns that arose through legal advice that we received. Those concerns were 1) that OSF users had a reasonable expectation of privacy of their OSF projects that our administrative review process could only assure through limiting reviews to COS employees, and 2) that preregistration reviewers could not also be eligible for a prize. The original plan to conduct reviews with an external pool of reviewers was thus infeasible.
Another challenge was how best to introduce and educate the community and then convince them to implement a new habit. There was a substantial lag time between initial contact with researchers and submission of a first preregistration. Though this campaign reached many individuals, it took time for them to consult with their colleagues and required an opportunity to begin a confirmatory project that was an appropriate fit for preregistration. As is often the case with culture change efforts, simple inertia and familiarity with the status-quo slowed progress toward initiating a preregistration. We addressed this challenge using the partnerships to disseminate our outreach materials and encourage adoption. Additionally, we used grassroots efforts like blog posts and emails to educate and recruit potential participants.
The final challenge was the persistent misconceptions about preregistration. These include fears about research ideas being “scooped” by competitors and the worry that preregistration would not permit authors to present the results of exploratory, unplanned research. We addressed those concerns through case studies presented in blog posts and more prominent mentioning of features such as embargoes. The huge growth in registration shows that many misconceptions about preregistration have been addressed within pockets of the research community.
There has been incredibly rapid adoption of preregistration in some communities. In areas of social and cognitive psychology, unregistered research is more likely to be questioned. Many of the strongest proponents of open science practices have led their colleagues by example and demonstrated what an ideal project looks like. In these communities, we no longer need to convince researchers to preregister. Instead, the challenges consist of improvement of early practices and adherence to preregistered plans (along with proper disclosure of unregistered changes).
We expect that the transition to increased expectations for preregistered reporting is already happening. In part because of the Prereg Challenge, many researchers have now preregistered at least once. One’s first preregistration is typically fairly basic. But with practice, subsequent plans are improved. Our main avenue for enabling the self-motivated research is through education materials and example preregistrations. We believe that existing momentum and expectations are sufficient to support those disciplines where preregistration is becoming the norm. Adhering to a preregistered plan is important in order to preserve the distinction between confirmatory and exploratory analyses. At a minimum, we expect researchers to transparently disclose deviations. These expectations are not always met, sometimes because of poor habits, and sometimes because of reviewer or editorial feedback. In some cases, getting an ideal preregistration to be reported properly not only requires convincing the lab members of the merits and proper practices of preregistration, but also holding reviewers and editors to the same expectations. This is a daunting task, given the amount of work it has taken just to convince researchers to preregister. We believe one method to encourage proper reporting is through scaffolding the report and are investigating various OSF solutions to help an author do so.
However, outside of the communities where preregistration is gaining significant traction, there remains substantial hurdles that have not changed over the course of this project period. Reaching those communities will require additional focus. More funding agencies are mentioning preregistration in their calls, which will increase the support needed for researchers who are going to be faced with making their first preregistration. We are actively pursuing multiple strategies to do break into these new areas. These strategies include 1) advocacy and education in disciplines that are new to us such as pre-clinical biomedical research and education sciences; 2) consultation with research funders and publishers that span across multiple disciplines where preregistration can be more widely implemented; and 3) customized OSF Registries that will give organizations the ability to promote preregistration more directly. The resources and expertise we have gained through the Prereg Challenge are essential to these efforts.
Preregistering adds transparency to research but this practice also poses a potential risk for the researcher, who is opening up their work to more scrutiny. Members of the research community are increasingly checking for these poor research practices. However, we cannot afford to shame imperfect early adopters, who might retreat from future open work. Our strategy to date has been to share the best examples we see and provide guidance for best practices, but the community is increasingly expecting higher standards and is calling out improper work when they see it. We will continue to encourage and model constructive feedback.
Another risk will be a failure to change norms and policies in societies or journals that are very resistant to change. Progress could stall if researchers become more entrenched between those that advocate for improved practices and those that do not. Our holistic approach to addressing these problems remains the same: 1) study the extent of the problem through research into the barriers of replication, 2) advocate for better policies and practices, and 3) build tools to enable ideal practices such as preregistration.
At COS, our business development process and infrastructure development are both progressing toward branded, customized OSF Registries. Currently, OSF Registries is a discovery portal registrations on the OSF and for two other connected registries. In the near future, customized registries will give institutions, funding agencies, and academic societies the opportunity to develop standards appropriate for the preregistering the work that they support. The existing language created for the Preregistration Challenge (now referred to as OSF Prereg) is designed to be generalizable and discipline-agnostic. However, this misses opportunities to use more discipline-specific and customized work. For example, the simple term “replication” means something very specific and different in many biomedical settings than it does in the social sciences, despite the fact that the process is identical. By allowing individual organizations to define the standards that make the most sense for their world, we will make preregistration more accessible to that community and transfer some community-building and advocacy opportunities to that organization. This mirrors the strategies we have employed for OSF for Institutions and OSF Preprints.
COS is continuing to implement effective strategies to realize open science practices in publishing and funding. This involves implementing the TOP Guidelines with the major publishers, standardizing disclosure practices (including whether or not the work was preregistered) with TOP Statements that can be stored in article meta-data, and increasing the usability of Open Science Badges (also via article meta-data). We also support grassroots efforts to implement related initiatives, notably “Registered Reports Now!” that gives researchers language to advocate for the format. Most of our policy efforts culminate with advocating for preregistration in one form or another. There are also ongoing developments at funders such as the NIH and several smaller organizations to encourage or even require more preregistration, though there is still confusion and misconception. Some common feedback we receive includes uncertainty about what should be preregistered, fear of requiring something that they do not yet fully understand, or simple institutional inertia to policy change. We are working with organizations to overcome these barriers through education and by implementing incentives to preregister even in the absence of of a policy requirement (e.g. an output of this workshop will be reviewer criteria to evaluate grants based on open science principles).
Prior to the Prereg Challenge, preregistration was a rarely-discussed practice outside of the clinical disciplines. Even in clinical science, registration was more for discoverability about ongoing drug trials and less about making clear distinctions between confirmatory and exploratory work. Those discussions existed at the time of the Prereg Challenge, but over the course of this campaign, there has been a surge of discussion devoted to this topic, helping to shape the practice of preregistration into a better tool for the research community. The Prereg Challenge has brought preregistration to the attention of of the scientific community. Our mission going forward is to continue to capitalize on that attention and support better science.
The “Replication Crisis” has evolved over the course of this campaign as it has begun to shift toward a revolution in conducting more credible science. Headlines such as “More social science studies just failed to replicate. Here’s why this is good,” “Psychology’s Replication Crisis Has Made The Field Better” and “Want to fix science’s Replication Crisis? Then Replicate” are focusing on the positive improvements that are taking place. This is replacing earlier narratives that focus on a replication crisis or even fraudulent activity.
The shift toward improving research and using preregistration by the psychology and social science research community, benefited from the tools and resources COS built as a result of this and other LJAF funding. We will continue to support those communities using the expertise we have gained, the infrastructure we built, and the community partnerships we formed. Future work will focus on raising expectations of and providing support for researchers who practice preregistration and on bringing preregistration into research areas where it is not yet common, such as education sciences and pre-clinical biomedical research.
210 Ridge McIntire Road
Suite 500
Charlottesville, VA 22903-5083
Email: contact@cos.io
Unless otherwise noted, this site is licensed under a Creative Commons Attribution 4.0 International (CC BY 4.0) License.
Responsible stewards of your support
COS has earned top recognition from Charity Navigator and Candid (formerly GuideStar) for our financial transparency and accountability to our mission. COS and the OSF were also awarded SOC2 accreditation in 2023 after an independent assessment of our security and procedures by the American Institute of CPAs (AICPA).
We invite all of our sponsors, partners, and members of the community to learn more about how our organization operates, our impact, our financial performance, and our nonprofit status.
