Take the $1,000,000 Preregistration Challenge. It's good for you and your research.


Preregistration is important because it increases the credibility of hypothesis testing by confirming in advance what will be analyzed and reported. We will give 1,000 awards of $1,000 each to promote education and experience with preregistration. Award winners will have completed a preregistration using the Open Science Framework, and published the results in an eligible journal.


Scientific Values vs. Scientific Practices

What is good for science and what is good for scientists are not always aligned. However, scientists must publish. Publishing success is more likely with innovative, positive, and tidy results – even at the expense of transparent, reproducible research. For example, a researcher may recall the original study purpose and analysis plan differently after observing the results and unintentionally bias conclusions toward results that look good for publication and ignore results that do not.

Confirmatory analyses are determined in advance; exploratory analyses are responsive to what is observed in the data. Confirmatory analyses maximize diagnostic value of statistical inferences; exploratory analyses provide opportunity for discovery of unexpected outcomes.

Preregistration distinguishes confirmatory (hypothesis testing) and exploratory (hypothesis generating) analyses. Confirmatory analyses are determined in advance; Exploratory analyses are responsive to what is observed in the data. Confirmatory analyses maximize diagnostic value of statistical inferences; Exploratory analyses provide opportunity for discovery of unexpected outcomes. Both are important for scientific discovery. The same data cannot be used to both generate and test a hypothesis. With preregistration, confirmatory analyses are planned in advance in order to retain the validity of their statistical inferences, and exploratory analyses are reported as post hoc investigations that might inspire confirmatory tests in future studies.

The Preregistration Challenge

Preregistration is an unfamiliar practice in many areas of science. We will give 1,000 awards of $1,000 each to promote education and experience with preregistration. Award winners will have completed a preregistration using the Open Science Framework, a free scholarly commons, and published the results in an eligible journal. All it takes is a single experiment and its analysis to be eligible. 

Share this handout for a brief overview and links to more information, and begin your preregistration today!


How to Earn the Prize

  1. Plan your study and analyses.

    Design your study and how you will analyze the data. The Preregistration workflow on the Open Science Framework (OSF) will guide you through study design and analysis planning. You can save your progress and return later to continue.

    Alternatively, if you prefer to design off-line or in a collaborative editor, you can use this Google doc template or this Word template and move the content to the OSF Preregistration workflow later.

  2. Submit your plan on the Open Science Framework.

    Once you and your collaborators complete the design and analysis plan, submit it for review to assess eligibility for the prize.

  3. Research plan review.

    We will review your plan to check for completeness and adherence to the eligibility criteria (see the tab "Eligibility Criteria" on the left). This is not peer review of your research. Once accepted, your research plan will be registered and eligible for the prize. Preregistrations must be pre-approved to be eligible.

  4. Do your science.

    This part is all up to you.

  5. Analyze according to your preregistered plan.

    For the confirmatory phase, analyze your data according to the preregistered plan, and write up the results. Additional exploratory analysis is encouraged, but must be clearly differentiated from the confirmatory analyses in your report.

  6. Write up your results.

    Report all of the preregistered analyses in your manuscript. Clearly distinguish preregistered analyses from additional exploratory analyses. Include a URL link to the study preregistration. Make sure that the article includes everything on this checklist when it is published. It is your responsibility to ensure that all checklist items are in your article, the publishing journal is not involved in the administration of the Preregistration prize.

  7. Submit your manuscript for publication.

    Submit your manuscript to an eligible journal (view the "Eligible Journals" tab on the left) for peer review. 

  8. Submit your published article to confirm completion of the Preregistration Challenge.

    As soon as your article is published online, submit notice of publication for review to confirm that the final article meets all Preregistration Challenge eligibility requirements. If it does, the article is eligible for the next prize award date.

  9. Receive your $1000 prize!

    The 1,000 prizes will be awarded across 4 award dates. If more eligible articles are submitted than available awards during that award period, then the eligible articles will be ranked by Preregistration date with earlier registrations being awarded first. Non-awarded entries will remain in the eligible pool for the next award date:

    • July 1, 2017: 100 Prizes
    • January 1, 2018: 100 Prizes
    • July 1, 2018: 250 Prizes
    • December 31, 2018: 500 Prizes

Eligibility Criteria

There is a detailed description of the terms and conditions for the Preregistration Challenge to meet legal requirements and maximize the fair administration of the process. Here are highlights of key items:

  1. Articles must be published in an eligible journal.
  2. Before becoming a preregistration, submissions to the Challenge must be reviewed for adherence to the eligibility criteria. Once accepted, submissions become a static, time stamped preregistration. Please do not begin your study until it is registered. Submissions are generally reviewed within the same business day.
  3. The published article must also be reviewed and approved for meeting eligibility criteria.
  4. U.S. law restricts eligibility of contests like this one from being administered within the U.S. State Department's list of embargoed countries. Therefore, residents of those countries may not participate in the Preregistration Challenge.
  5. The use of pre-existing data may make your registration ineligible for the Preregistration Challenge. See the FAQ tab for more info.
  6. When awarded, one prize is given per published article to the article author that submitted the preregistration that was approved by the Challenge administrators.
  7. Awardees can earn only one prize each, but can enter as many times as they wish and can be co-authors on other awarded entries.
  8. Prizes will be awarded at pre-determined dates to eligible entrants. If more eligible entrants exist than available prizes, entrants will be ranked based on the date of registration.
  9. Entering the Preregistration Challenge requires that you agree to all of its terms. If you do not agree to the terms you may still continue, use the form, and register your research study without entering the Challenge.

As experience is gained with preregistration and the Challenge process, we expect that we will identify ways to improve the process and requirements. We will make updates as is needed and document those changes. Prior entrants will not lose eligibility based on future changes. The eligibility criteria and preregistration process is currently version 1.0.0. Check the change log here.

We welcome questions and feedback (prereg@cos.io).

FAQ

Things that always come up or get asked frequently can probably be found here.

No. Confirmatory analyses are planned in advance, but they can be conditional. A pre-analysis plan might specify preconditions for certain analysis strategies and what alternative analysis will be performed if those conditions are not met. For example, if an analysis strategy requires data for a variable to be normally distributed, the analysis plan can specify evaluating normality and an alternate non-parametric test to be conducted if the normality assumption is violated.

For conditional analyses, we suggest that you define a 'decision-tree' containing logical IF-THEN rules that specify the analyses that will be used in specific situations. Here are some example decision trees. In the event that you need to conduct an unplanned analysis, preregistration does not prevent you from doing so. Preregistration simply makes clear which analyses were planned and which were not.

Yes. The central aims of preregistration are to distinguish confirmatory and exploratory analyses in order to retain the validity of their statistical inferences. Selective reporting of planned analyses is problematic for the latter.

Yes. Selective interpretation of pre-planned analyses can disrupt the diagnosticity of statistical inferences. For example, imagine that you planned 100 tests in your preregistration, and then reported all 100, 5 of which achieved p < .05. It is possible (even likely) that those five significant results are false positives. If the paper then discussed just those five and ignored the others, the interpretation could be highly misleading. Planning in advance is necessary but not sufficient for preserving diagnosticity.

To reduce interpretation biases, confirmatory research designs often have a small number of tests focused on the key questions in the research design, or adjustments for multiple-tests are included in the analysis plan. It may be that some preregistered analyses are dismissed as inappropriate or ill-conceived in retrospect, but doing that explicitly and transparently assists the reader in evaluating the rest of the confirmatory results.

No. Preregistration distinguishes confirmatory and exploratory analyses (Chambers et. al, 2014). Exploratory analysis is very important for discovery and hypothesis generation. Simultaneously, results from exploratory analyses are more tentative, p-values are less diagnostic, and additional data is required to subject an exploratory result to a confirmatory test. Making the distinction between exploratory and confirmatory analysis more transparent increases credibility of reports and helps the reader to fairly evaluate the evidence presented (Wagenmakers et al., 2012).

Exploratory and confirmatory research are both crucial to the process of science. In exploratory work, the researcher is looking for potential relationships within a dataset, effects of a candidate drug, or differences between two groups. The researcher wants to minimize the chance of making a Type II error, or a false negative, because finding something new and unexpected could be an important new discovery.

In confirmatory work, the researcher is rigorously testing a predicted effect. The specific hypothesis is very clear, and she has specified one way to test that hypothesis. The goal of confirmatory research is to minimize the Type I error rate, or false positives.

The purpose of preregistration is to make sure the distinction between these two processes are very clear. Once a researcher begins to slightly change the way to test the hypothesis, the work should be considered exploratory.

At least one confirmatory test must be specified in each preregistration.

Perhaps. A goal of pre-analysis plans is to avoid analysis decisions that are contingent on observed results (except when those contingencies are specified in advance, see above). This is more challenging for existing data, particularly when outcomes of the data have been observed or reported. Standards for effective preregistration using existing data do not yet exist. We are using the Preregistration Challenge to help develop such standards. As such, we have defined initial eligibility standards with pre-existing data and expect these to be refined over time.

When you submit your research plan, you will identify whether existing data is included in your planned analysis. For some circumstances, you will describe the steps that will ensure that the data or reported outcomes do not influence the analytical decisions. Below are the categories for which preregistration may still be eligible for the Preregistration Challenge:

  1. Registration prior to collection of data: As of the date of submission of Research Plan for Preregistration, the data have not yet been collected, created, or realized. In this scenario, the Entrant must certify that the data do not exist to retain eligibility.
  2. Registration prior to any human observation of the data: As of the date of submission, the data exist but have not yet been quantified, constructed, observed, or reported by anyone - including individuals that are not associated with the proposed Study and Research Plan. Examples include museum specimens that have not been measured, or data that have been collected by non-human collectors and are inaccessible. In this scenario, the Entrant must certify that the data have not been observed by anyone and how this is the case to retain eligibility.
  3. Registration prior to Entrant access to the data: As of the date of submission, the data exist, and have not been accessed by the Entrant, or the Entrant’s Study collaborators. Commonly, this includes data that has been collected by another researcher or institution. In this scenario, the Entrant must certify that they have not accessed the data, explain who has accessed the data, and justify how any observation, analysis, and reporting of that data avoids compromising the confirmatory nature of the Research Plan. The justification will be reviewed to determine eligibility.
  4. Registration prior to Entrant analysis of the data: As of the date of submission, the data exist and have been accessed by the researcher, though no analysis has been conducted related to the Research Plan. Common situations for this are the existence of a large dataset that is the subject of many studies over time, or a split sample in which a portion is not analyzed to be subjected to confirmatory testing after exploratory analysis of the other data. In this scenario, the Entrant must certify that they have not analyzed the data related to the Research Plan (including calculation of summary statistics), explain what other analysis or reporting of the data has been done by the Entrant or others, and justify how any prior observation, analysis, and reporting of that data avoid compromising the confirmatory nature of the Research Plan.

There are several research circumstances that present challenges to conducting preregistered research.

  • Studies in which you are not conducting statistical inference testing. Most existing preregistration models are designed to reduce bias when the researcher intends to apply statistical inference techniques to collected data. There are many publishable, peer-reviewed endeavors for which this is not the case such as qualitative research and some kinds of observational studies.
  • Hypothesis testing using pre-existing data. Using previously-collected data places additional burden on the researcher to avoid analysis decisions that are contingent on the data and research outcomes. For example, seeing a simple summary of descriptive statistics prior to inferential testing can influence the choice of test and comparison of conditions or variables.
  • Field studies. Field science can be particularly challenging to preregister. Sample size, measured variables, and even design may have to respond to unpredictable events. Pilot trials, feedback from peers, and additional time or imagination in the planning phase can help make registered plans more accurate, including identification of data collection contingencies in advance.

If the present preregistration process does not fit your research approach effectively, and you believe that there are ways to conduct preregistered research in your field, we encourage you to contact us to help develop and specify a preregistration process for your work (prereg@cos.io). The Preregistration Challenge is both an educational effort to encourage preregistration, and a research effort to develop effective preregistration processes that cover the wide variety of research approaches in science.

Registered Reports are a particular publication format in which the preregistered plan undergoes peer review in advance of observing the research outcomes. However, in the case of Registered Reports, that review is about the substance of the research and is overseen by journal editors. Research designs that pass peer review are offered ‘in principle acceptance’ (IPA) ensuring that the results are guaranteed to be published regardless of findings, as long as the methodology is carried out as described. Registered Reports are currently offered at 32 journals.

Participants in the Prereg Challenge are welcome and encouraged to submit their preregistered designs to the Registered Reports mechanism at their preferred journal. We recommend undergoing peer review at the journal first and preregistering for the Prereg Challenge after obtaining in-principle acceptance. That way, the registered Prereg Challenge will not need to be revised following substantive peer review at the journal.

Preregistration is relatively new to many people, so you may get questions from reviewers or editors during the review process. Below are some possible issues you may encounter and suggested strategies.


Possible editorial feedback: Reviewers or editors may request that you remove an experiment, study, analysis, variable, or design feature because the results are null results or marginal.

Prereg requirement: All preregistered analysis plans must be reported. Selective reporting undermines the diagnosticity of reported statistical inferences.

Possible response to the editor: The results of these tests are included because they stem from prespecified analyses in order to conduct a confirmatory test. Removing these results because of their non-significance would perpetuate publication bias already present in the literature (Chambers et al., 2014Simmons et al., 2011Wagenmakers et al., 2012).

Notes: If the reviewer/editor proposes a reason why they believe the null result could be explained by a design flaw, it can often be helpful/appropriate to leave the test in, but discuss the reviewers concerns about the validity of that particular test/design feature in a discussion section.


Possible editorial feedback: Why are you referring to a preregistered plan and reporting them separately from other analyses?

Prereg requirement: The published article must make clear which analyses were part of the confirmatory design (usually distinguished in the results section with confirmatory and exploratory results sections), and there must be a URL to the preregistration on the OSF.

Possible response to the editor: The registration was certified prior to the start of data analysis. This defines analyses that were prespecified and confirmatory versus those which were not prespecified and therefore exploratory. Clarifying this allows readers to see that the hypotheses, analyses, and design that were prespecified have been accurately and fully reported (Jaeger & Halliday, 1998Kerr, 1998Thomas & Peterson, 2012).


Possible editorial feedback: Editor requests that you perform additional tests.

Prereg requirement: Additional tests are fine, they just need to be distinguished clearly from the confirmatory tests.

Possible response to the editor: Yes, these additional analyses are informative. We made sure to distinguish them from our preregistered analysis plan, which is the most resistant to the problem of alpha inflation. These analyses provide additional information for learning from our data.

Preparing these plans does require time and effort. However, every decision included in the preregistration process is one that a researcher will have to make at some point anyway, and making these decisions up front, before data collection begins, can improve your workflow and reduce subtle biases. Our goal is to create a system that is transparent and easy to use, and ultimately increases the efficiency and effectiveness of research design. We welcome feedback (prereg@cos.io) about how to make it even better.

Scientists at almost every career level are under exceptional pressure to publish. Also, evidence suggests that publishable results are often not easily reproduced (Begley & Ellis, 2012Open Science Collaboration, 2015Prinz et al., 2011). Therefore, we have created this incentive for researchers to try preregistration as a formalization of the idealized model of confirmatory hypothesis testing. An indicator of success will be measured by the number of participants who register analysis plans after participating in the Prereg Challenge because they have found it to improve their workflow and their confidence in their findings.

Preregistration has existed in a more limited form for clinical trials, but it is relatively new for basic and preclinical sciences. There are both strong theoretical reasons to preregister (Chambers et. al, 2014Nosek & Lakens, 2014Simmons, Nelson, & Simonsohn, 2011Thomas & Peterson, 2012), and some empirical evidence that suggests that it does impact research outcomes (Kaplan and Irvin, 2015). We are conducting the Preregistration Challenge to increase experience and evidence about preregistration in the basic sciences. We will conduct and support research efforts to evaluate preregistration through the Prereg Challenge and registration on the OSF more generally.

You may embargo your preregistration plan for up to 4 years following review to keep the details from public view. All registrations eventually become public because that is part of the purpose of a registry - to reduce the file-drawer effect (sometimes called the grey literature). It is possible to withdraw your preregistration, but a notification of the withdrawal will be public.

Yes! After you complete the research plan, you will have the option of registering it without submitting for review for the Prereg Challenge. Review is required to be eligible for a $1,000 prize, but the Preregistration workflow is available for general use.


Still have questions? We would be happy to help and appreciate your feedback. Contact us at prereg@cos.io

Eligible Journals

Entrants must publish their eligible preregistered study in a journal listed here to be eligible for a Preregistration Challenge award. Note that the Preregistration Challenge is administered independently from any editorial, peer review, or publication process in these journals.

Is Your Favorite Journal Not Yet Eligible?

If a journal relevant to your research area does not appear on this list, you can assist in helping it become an eligible journal by encouraging the publisher or editor to take concrete steps to improve transparency and reproducibility such as signing the TOP Guidelines, making Registered Reports a submission option, and adopting badges to acknowledge open practices.

Loading

Journal Title Publisher Society Affiliation Subject Area

Research Plan Review

    Each submission must pass a review process in which the statistical methods of the proposed study and its analyses are checked for completeness and adherence to Preregistration Challenge eligibility requirements. This review does not assess the substance of the research, or the validity of the research design or statistical methodology and has no impact on the independent editorial decisions of any journal.

    Prereg Challenge administrators and reviewers will review the submitted study design and analysis descriptions, and determine whether all question fields were answered with enough detail to fully pre-specify the design and analysis plan, and follow the eligibility requirements. 

    Here are the guidelines that reviewers will use when evaluating your submitted plans.

    Only studies that pass this review process are eligible for the Preregistration Challenge. Studies that do not pass this review may still be registered on the Open Science Framework independent of the Prereg Challenge.

Article Review

    Following independent peer review and acceptance for publication at a Prereg Challenge eligible journal, you will submit a link to the "in press" or published article online, and a copy of the journal-formatted published article for final review for eligibility for a Prereg Challenge award.

    Prereg Challenge administrators and reviewers then review the article to ensure that it meets all of the eligibility criteria including: following the preregistered design, providing a link to the preregistration, making clear distinction between preregistered confirmatory analyses and other analyses, and following the preregistered analysis plan.

    It is possible that deviations from the sampling, design, or analysis plans occurred in the course of conducting the research. Sometimes these deviations will render the preregistration ineligible for the Prereg Challenge award. In any case, deviations from preregistered plans must be documented explicitly in the article.

    Here are some example scenarios about possible changes to the research plan and how to respond to them.

Questions? Please contact us as prereg@cos.io or take advantage of our free statistical consulting services.