Why pre-registration might be better for your career and well-being

October 9th, 2018, Gró Einarsdóttir


Registered Reports is a publishing format in which peer review occurs before results are known. This changes the equation of scientific publishing. Reviewers are focused on answering the questions: Is this work worth doing? If so, what is the best way to answer those questions? Instead of trying to "fix" work that has already been completed. Authors are rewarded for the parts of the scientific process that that they control, the importance and rigor of their work, and not on the parts that they do not have control over, the results.

Below, Gró Einarsdóttir details her experience with Registered Reports as a graduate student at the University of Gothenburg.

I recently published my first preregistered study. A lot has been written about the scientific merits of preregistration as this practice enables a clear distinction between testing a hypothesis and generating one (see tips for further reading at the end of this post).  However, scientists are not only motivated by the desire to improve knowledge. Like everyone else, scientists also want to advance their careers and perhaps even feel good while doing so. In this blogpost I want to share my personal experience of how it felt to conduct a Registered Report and compare that to doing research the “traditional” way.  

I began working on a project in early 2013 that in many ways represents the “traditional” way of doing psychological research. As a master student I started to collect data that would later be used as the building blocks of my doctoral thesis. The data collection was expensive as it was based on a representative sample of the population. My supervisors and I constructed the questionnaire ourselves using a large number of questions. We had some ideas about the expected relationships, but our hypotheses were vague and none of them were written down before the data collection began. Five years later we published the results in the journal Nordic Psychology.

So, what happened during those 5 years?

During these five years I analyzed the data in every way imaginable. I conducted descriptive analysis, regression analysis, hierarchical regressions, ANOVAs, MANOVAs, EFAs, CFAs, Multi-trait Multi-method analysis, did multiple imputations to handle missing data, excluded covariates, included covariates, and tried seemingly endless variations of indexing my items. I sent the manuscript to multiple journals and repeatedly got rejected. After each rejection I re-wrote most of the manuscript. 

I would like to think that the choice of analysis in the published version of this research is based on integrity. I didn’t willingly try to hide the flaws in the data or exclude observations or variables just to get a significant p-value. And in some sense the publishing of the article took so long because I was still learning how to analyze and write up research results. However, I have often felt that this process was needlessly long and the bitter taste of sunk costs linger on my pallet.

On a more personal note, working on the same scientific article for five years is extremely frustrating. This may sound a bit dramatic, but it did have a serious impact on my sanity. Finding the motivation to do one more analysis, or make one more revision, has taken all of my strength. Since the goals of the research were so vague from the beginning, and since the progress has been so intangible, I have been very unsure whether I am moving ahead or in circles. I have wanted to quit more times than I care to remember. They say that success comes from the ability to go from failure to failure with undiminished enthusiasm. My enthusiasm was certainly diminished. 

The turning point

Based on this experience I knew that something needed to change in order for me to be able to carry on. Luckily, I was introduced to the open science movement by Rebecce Willén, then a PhD student and my department, but now the founder of the Institute for Globally Distributed Open Research and Education (IGDORE). This opened up my eyes to a world of possibilities. I was fortunate enough to get the support and freedom of my supervisors to carry out a Registered Report. To my relief the process of doing a Registered Report was not only more rigorous but was much more rewarding both pragmatically and personally speaking. Below I list some of the reasons why. 

Getting peer-review comments that you can actually address

Once I had written up my proposal for the Registered Report I sent it in for a journal review. There I got a lot of additional feedback from experts in the field. The reviewers pointed out that I needed to clarify my theoretical rationale upon which I had built my hypotheses. With the help of those comments I was able to go back to the literature and clarify my thinking and make my hypotheses more specific. To a young researcher, getting feedback from experts is really valuable. I truly needed this help for the research to get better. In contrast, getting comments on how to improve your theories, design, and hypotheses when your research is already conducted is often demoralizing. If you are operating on tight schedule with a limited budget you may simply not have the time or the money to redo the study. In that situation your only option is to discuss the limitations and hopefully apply what you have learnt later on in your academic career. 

The peace of mind that comes from getting accepted for publication in advance

After the reviews came back I did some major revisions to the preregistration and submitted it again. The journal editor was happy with the changes and gave me an in-principle acceptance. This meant that if I followed my research plan the journal guaranteed that they would publish my findings regardless of the results. Not only was I certain that I had tried my utmost to conduct research to the highest standards, I also knew that this meant that I would be able to finish my PhD on time since the requirements at my university was to have at least one published article when you finish. What a relief!

Diagnostic feedback

Since I had specified my theoretical rationale, my hypotheses, my sampling plan, my analyses, and my criteria for evaluating supporting evidence in advance it was much clearer to me what I was going to do and what I expected to find than in earlier projects. In contrast, during the "traditional" project I was constantly worried that my revisions were effected by hindsight bias. Following a revision my supervisor would often rejoice when he felt that the changes reflected our thinking at the onset of the project. Although it is possible that each revision clarified our original thinking, I feared that it was equally likely that the initial hypothesis was so vague that there were multiple ways to analyze and write up the results. I didn't want to get stuck in "I-knew-it-all-along" thinking, where I used the same data to generate both predictions and explanations of the findings. However, since we hadn't written down our hypothesis beforehand I felt that I had little to protect myself from falling into that trap. Having my predictions time stamped in black and white, for the registered report, really calmed my nerves and made me much more confident in interpreting the results. Not only did it make the feedback from the results more diagnostic since I had a clearer distinction between expected findings and unexpected findings, it also felt like a protective shield against my crippling worries.

It's Quicker

Many researchers are afraid to conduct preregistered research because they are pressed for time and worry that the additional preparation needed for a registered report will slow them down. Although I cannot speak for everyone, in my experience, publishing preregistered research is from start to finish a much shorter process. Going through the process helped me plan my study better and systematized my thinking before data collection began, which ultimately saved me a lot of time and grief since I made fewer irreversible mistakes. My registered report took less than a year to conduct, while the “traditional” research took five years. Although the two studies differ in more ways than one and are therefore not necessarily comparable, the time difference is still too big to be ignored! 

My advice to young researchers

I am now at the end of my graduate training and I just received my PhD. If I could share any advice with those who are just about to start their PhD education it would be this: Start with doing a preregistered study! Not only will you learn good scientific craftsmanship right from the start, but I suspect that your experience will be more personally and academically rewarding if you follow this path towards progress. 

Gró Einarsdóttir

Ph.D in Social Psychology

Related reading: Interview: Randy McCarthy discusses his experiences with publishing his first Registered Report

For more selfish reasons for preregistering your study:

Wagenmakers, E-J. & Dutilh, G. (2016). Seven selfish reasons for preregistration.

Further readings on the scientific merits of pre-registrations:

Nosek, B. A., & Bar-Anan, Y. (2012). Scientific utopia: I. Opening scientific communication. Psychological Inquiry, 23(3), 217-243.

Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 201708274.

Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7(6), 615-631.

John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524-532.

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359-1366.

Recent Blogs

The Content of Open Science

What Second Graders Can Teach Us About Open Science

What's Going on With Reproducibility?

Open Science and the Marketplace of Ideas

3 Things Societies Can Do to Promote Research Integrity

How to Manage and Share Your Open Data

Interview with Prereg Challenge Award Winner Dr. Allison Skinner

Next Steps for Promoting Transparency in Science

Public Goods Infrastructure for Preprints and Innovation in Scholarly Communication

A How-To Guide to Improving the Clarity and Continuity of Your Preregistration

Building a Central Service for Preprints

Three More Reasons to Take the Preregistration Challenge

The Center for Open Science is a Culture Change Technology Company

Preregistration: A Plan, Not a Prison

How can we improve diversity and inclusion in the open science movement?

OSF Fedora Integration, Aussie style!

Replicating a challenging study: it's all about sharing the details.

Some Examples of Publishing the Research That Actually Happened

How Preregistration Helped Improve Our Research: An Interview with Preregistration Challenge Awardees

Are reproducibility and open science starting to matter in tenure and promotion review?

The IRIS Replication Award and Collaboration in the Second Language Research Community

We Should Redefine Statistical Significance

Some Cool New OSF Features

How Open Source Research Tools Can Help Institutions Keep it Simple

OSF Add-ons Help You Maximize Research Data Storage and Accessibility

10 Tips for Making a Great Preregistration

Community-Driven Science: An Interview With EarthArXiv Founders Chris Jackson, Tom Narock and Bruce Caron

A Preregistration Coaching Network

Why are we working so hard to open up science? A personal story.

One Preregistration to Rule Them All?

Using the wiki just got better.

Transparent Definitions and Community Signals: Growth in the Open Science Community

We're Committed to GDPR. Here's How.

Preprints: The What, The Why, The How.

The Prereg Challenge Is Ending. What's Next?

We are Now Registering Preprint DOIs with Crossref

Using OSF in the Lab

Psychology's New Normal

How Open Commenting on Preprints Can Increase Scientific Transparency: An Interview With the Directors of PsyArxiv, SocArxiv, and Marxiv

The Landscape of Open Data Policies

Open Science is a Behavior.

Why pre-registration might be better for your career and well-being

Interview: Randy McCarthy discusses his experiences with publishing his first Registered Report

Towards minimal reporting standards for life scientists

Looking Back on the Prereg Challenge and Forward To More Credible Research

OSF: Origin, growth, and what’s next

A Critique of the Many Labs Projects

The Rise of Open Science in Psychology, A Preliminary Report

Strategy for Culture Change

New OSF Registries Enhancements Improve Efficiency and Quality of Registrations

Registered Reports and PhD’s – What? Why? How? An Interview with Chris Chambers

How to Collaborate with Industry Using Open Science

How to Build an Open Science Network in Your Community

Seven Reasons to Work Reproducibly

COS Collaborates with Case Medical Research to Support Free Video Publishing for Clinicians and Researchers

Advocating for Policy Improvements at Your Institution

Announcing a joint effort to improve research transparency: FAIRSharing and TOP Guidelines

OSF as a tool for managing course-embedded research projects

Journals test the Materials Design Analysis Reporting (MDAR) checklist

Now you can endorse papers on OSF Preprints with Plaudit

Many Labs 4: Failure to Replicate Mortality Salience Effect With and Without Original Author Involvement

Approach and vision for the OSF Preprint infrastructure

Conflict between Open Access and Open Science: APCs are a key part of the problem, preprints are a key part of the solution

Re-engineering Ethics Training: An Interview with Dena Plemmons and Erica Baranski

Answering Your Preregistration Questions

UBC leads the way as first Canadian institutional OSF member

2019 Recap: OSF Growth and Open Knowledge Exchange

Getting Started with OSF

This website relies on cookies to help provide a better user experience. By clicking Accept or continuing to use the site, you agree. For more information, see our Privacy Policy and information on cookie use.