Open Science aims to make scientific processes more transparent and the results more accessible to the research community. Preregistration is one of the available open science methods, which enables researchers to report a study design or analysis plan before the study’s results are known. The goal is to increase transparency and reduce unethical research behaviors by eliminating the opportunity to engage in p-hacking, data dredging and the like.
Updating a preregistration is a way to transparently report these changes.
COS’s Mark Call hosted a March 29, 2022 webinar with members of the open science community to discuss the purpose of updating preregistrations. With researchers from around the world on the webinar, the roundtable speakers also discussed what funders and readers are looking for when reviewing registration updates, and how the opportunity to update is impacting their research and funding process.
Speakers included:
- Kristin Eldon Whylly, Program Officer & Senior Project Manager, Templeton World Charity Foundation
- Crystal Young, Senior Research Analyst, The Dartmouth Institute
- Pamela Davis-Kean, Professor of Psychology and Associate Director, Michigan Institute for Data Science.
- Katie Corker, Associate Professor, Grand Valley State University
Young kicked things off with a colorful anecdote about the value of preregistration, aside from scientific rigor. “I also think about it from a really practical standpoint. When I taught, I would always tell my students that the best person to answer questions about how you were thinking at the beginning of your study is You Six Months Ago,” Young said. “But, unfortunately, You Six Months Ago is not available for emails, not available to talk.”
Young said she values the preregistration document as a record of the study’s plan and purpose. It has ultimately helped her get to publication faster because it’s helped align the research team’s goals and methods from the outset.
Davis-Kean said she thinks of the preregistration as a “lab notebook” allowing you to “map out everything we’ve done so somebody can replicate it in the future.”
Davis-Kean noted writing the preregistration document even makes the science more fun for her, because it allows the team to engage in a more robust discussion ahead of time. “It makes for a really nice research package that we have to give to the community,” she said.
Whylly brought the perspective of the funder eyeing preregistered studies. “If we’re going to fund projects that are really making an impact and are contrarian discoveries, we really want to make sure it can earn the trust of the broader scientific community,” Whylly said.
Whylly manages an initiative that seeks to better understand the science behind human consciousness. Drawing on best practices in open science and adversarial collaboration, the “Accelerating Research on Consciousness” initiative uses preregistration to promote open and rigorous engagement among leaders of opposing theories.
“The idea is to kill off incompatible theories,” Whylly said. “For us, preregistration is crucial because we really need each of the theory leaders to sign off on what the different results of the research are going to say. And preregistration has really given us that tool to get them [...] to commit to the results.”
But, because we can’t time-travel into the future, either, we need to be able to update our preregistrations as the data and plans change.
As Davis-Kean noted, these studies often generate very large datasets that are sometimes not what we expected, or the data show that the planned analysis is not as relevant as another type of analysis might be.
“I think this experience of ‘plans changing’ is really common,” said Corker. “This is one of the most persistent myths of preregistration, that you’re locked into the plan that you originally preregister, and are not able to make changes.”
Instead, you just disclose the changes you made in a registration update and/or your final paper. Or, if a peer reviewer suggests an alternative analysis, you provide the results both ways so the reader can decide for themselves, Corker said.
From a funder’s perspective, Whylly said, the updates and disclosures “all sound very much in line with what we would like to see.”
“We just want to be able to track what has been changed and why it’s been changed,” Whylly said. “As a funder, we work very hard to find the best experts in the areas we want to fund, and we don’t want to necessarily impose on these experts as they’re completing their research. We don't want to stifle innovation, and sometimes updating the preregistration is necessary to meet that innovation.”
You can watch the full webinar. Or, if you’re already convinced of the need to update your registration, you can skip ahead to the 36:00 minute mark to see how to do it.
