Improving the quality and specificity of preregistration

Dr. Marjan Bakker, Tilburg University —

Preregistration, which is specifying a research plan in advance of the study, is seen as one of the most important ways to improve the credibility of research findings. With preregistration, a clear distinction between planned and unplanned analyses can be made, thereby eliminating the possibility of making data-contingent decisions (Nosek, Ebersole, DeHaven, & Mellor, 2018). Over the last years, preregistration is gaining more and more popularity. For example, the number of preregistrations at OSF has approximately doubled yearly from 38 in 2012 to 36,675 by the end of 2019 (http://osf.io/registries).

However, to eliminate all possibilities to make data-contingent decisions, all these decisions, also called researcher degrees of freedom (Wicherts et al., 2016), should be specified beforehand in the preregistration in a specific, precise, and exhaustive way. Specific means that all phases of the research and all choices are covered. Precise means that each aspect of the preregistration is open to only one interpretation. Exhaustive means that it explicitly excludes the possibility of deviations from the preregistered plan. For example, a description like ‘we will remove participants that do not follow the instructions seriously’ leaves ample room for a decision to remove different sets of participants (e.g., those who answered all questions with the same answer, those who were extremely fast or extremely slow). A specific, precise, and exhaustive preregistration would include the protocol to decide on which exact grounds participants are excluded and explicitly state that no other participants will be excluded. Based on our own experience with writing preregistrations and with reading the preregistrations of others, we realized that it is quite difficult to write such a preregistration.

We, therefore, wondered how writing specific, precise, and exhaustive preregistrations can be facilitated. One way to do so, is with formats that contain instructions about what to include in the preregistration. Currently, multiple preregistration formats are available. These formats range from ones that offer hardly any instructions to others with instructions to provide a high level of detail about many aspects of the study. We wanted to know whether these preregistrations formats can help researchers write better preregistrations. Therefore, we evaluated the extent to which different preregistration formats restricted the opportunistic use of 29 researcher degrees of freedom (Wicherts et al., 2016). We compared the two preregistration formats most commonly used at the start of our study: Standard Pre-Data Collection Registration, an unstructured format with minimal direct guidance to give researchers flexibility for what to pre-specify, and Prereg Challenge registration (now called “OSF Preregistration”), a structured format with detailed instructions.

In this preregistered study, that is currently published in PLOS Biology, we indeed found that registrations from Structured formats (Mdn = 0.81) received higher median Transparency Scores than those from Unstructured formats (Mdn = 0.57), U = 2053, p < .001, Cliff’s Delta = 0.49. This result indicates that following structured preregistration formats helps write more specific, precise, and exhaustive preregistrations. However, the Transparency Score scale ranges from 0 (worst) to 3 (best), and the highest Transparency Score was 1.47 for a preregistration that followed the Structured format. Thus, also the preregistrations that followed the Structured format didn’t perform that impressive and had still ample room for improvement.

These relatively low Transparency Scores can be partly explained by us being very strict. For example, to get a Transparency Score of 3, authors needed to explicitly exclude the possibility of deviating from the preregistration (e.g., state that they would not remove any outliers and would not deviate from this decision). You can argue that this is too strict and that preregistration is explicitly restricting deviations by definition. However, we do not know this for sure. For example, if a researcher does not state anything about the handling of outliers, it could mean they will not remove any, or it could mean that they will make decisions about outliers after the fact.    

When scoring all these preregistrations according to our protocol, we also found that some preregistrations were so ambiguous that it was hard to make sense of the planned research. Hypotheses, which can be considered the heart of the preregistration, were often not clearly stated, making it already hard for our coders to count the number of hypothese. For example, it was sometimes unclear whether a preregistration contains multiple hypotheses for multiple outcome variables or just one hypothesis. Furthermore, we found some instances in which both a positive and a negative relationship between a set of two variables was expected.

Therefore, based on our study, we suggest improvements to the preregistration process to restrict researcher degrees of freedom better. First, preregistration formats should help authors to write clear, precise, and exhaustive preregistrations. This can be done by having extensive formats that cover all possible researcher degrees of freedom. These formats can be adapted for specific fields with their specific researcher degrees of freedom (e.g., a preregistration format for fMRI studies that contain questions concerning the exact processing of the data; https://osf.io/6juft/). Furthermore, these formats could contain checkboxes to ensure that statements about specific researcher degrees of freedom are exhaustive (e.g., “no other outlier removal procedures will be used than the one specified above”). Second, preregistrations could prompt researchers to articulate each hypothesis and how it will be tested explicitly. This can be done by numbering the specific hypotheses and the accompanying predictions and relate the specific measures, analyses, and inference criteria to these hypotheses. Third, preregistration may benefit from peer-review before the collection of the data. This peer review could be formal, such as in a Registered Report, but also informal by a colleague who could use our scoring protocol for this (https://osf.io/v8yt4/). With these improvements, we can increase the quality and specificity of preregistrations and improve the credibility of research findings.

Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 115(11), 2600-2606. doi:10.1073/pnas.1708274114

Wicherts, J. M., Veldkamp, C. L. S., Augusteijn, H. E. M., Bakker, M., van Aert, R. C. M., & Van Assen, M. A. L. M. (2016). Degrees of freedom in planning, running, analyzing, and reporting psychological studies. A checklist to avoid p-hacking. Frontiers in psychology, 7, 1832. doi:10.3389/fpsyg.2016.01832