Why pre-registration might be better for your career and well-being

Registered Reports is a publishing format in which peer review occurs before results are known. This changes the equation of scientific publishing. Reviewers are focused on answering the questions: Is this work worth doing? If so, what is the best way to answer those questions? Instead of trying to "fix" work that has already been completed. Authors are rewarded for the parts of the scientific process that that they control, the importance and rigor of their work, and not on the parts that they do not have control over, the results.

Below, Gró Einarsdóttir details her experience with Registered Reports as a graduate student at the University of Gothenburg.

I recently published my first preregistered study. A lot has been written about the scientific merits of preregistration as this practice enables a clear distinction between testing a hypothesis and generating one (see tips for further reading at the end of this post).  However, scientists are not only motivated by the desire to improve knowledge. Like everyone else, scientists also want to advance their careers and perhaps even feel good while doing so. In this blogpost I want to share my personal experience of how it felt to conduct a Registered Report and compare that to doing research the “traditional” way.  

I began working on a project in early 2013 that in many ways represents the “traditional” way of doing psychological research. As a master student I started to collect data that would later be used as the building blocks of my doctoral thesis. The data collection was expensive as it was based on a representative sample of the population. My supervisors and I constructed the questionnaire ourselves using a large number of questions. We had some ideas about the expected relationships, but our hypotheses were vague and none of them were written down before the data collection began. Five years later we published the results in the journal Nordic Psychology.

So, what happened during those 5 years?

During these five years I analyzed the data in every way imaginable. I conducted descriptive analysis, regression analysis, hierarchical regressions, ANOVAs, MANOVAs, EFAs, CFAs, Multi-trait Multi-method analysis, did multiple imputations to handle missing data, excluded covariates, included covariates, and tried seemingly endless variations of indexing my items. I sent the manuscript to multiple journals and repeatedly got rejected. After each rejection I re-wrote most of the manuscript. 

I would like to think that the choice of analysis in the published version of this research is based on integrity. I didn’t willingly try to hide the flaws in the data or exclude observations or variables just to get a significant p-value. And in some sense the publishing of the article took so long because I was still learning how to analyze and write up research results. However, I have often felt that this process was needlessly long and the bitter taste of sunk costs linger on my pallet.

On a more personal note, working on the same scientific article for five years is extremely frustrating. This may sound a bit dramatic, but it did have a serious impact on my sanity. Finding the motivation to do one more analysis, or make one more revision, has taken all of my strength. Since the goals of the research were so vague from the beginning, and since the progress has been so intangible, I have been very unsure whether I am moving ahead or in circles. I have wanted to quit more times than I care to remember. They say that success comes from the ability to go from failure to failure with undiminished enthusiasm. My enthusiasm was certainly diminished. 

The turning point

Based on this experience I knew that something needed to change in order for me to be able to carry on. Luckily, I was introduced to the open science movement by Rebecce Willén, then a PhD student and my department, but now the founder of the Institute for Globally Distributed Open Research and Education (IGDORE). This opened up my eyes to a world of possibilities. I was fortunate enough to get the support and freedom of my supervisors to carry out a Registered Report. To my relief the process of doing a Registered Report was not only more rigorous but was much more rewarding both pragmatically and personally speaking. Below I list some of the reasons why. 

Getting peer-review comments that you can actually address

Once I had written up my proposal for the Registered Report I sent it in for a journal review. There I got a lot of additional feedback from experts in the field. The reviewers pointed out that I needed to clarify my theoretical rationale upon which I had built my hypotheses. With the help of those comments I was able to go back to the literature and clarify my thinking and make my hypotheses more specific. To a young researcher, getting feedback from experts is really valuable. I truly needed this help for the research to get better. In contrast, getting comments on how to improve your theories, design, and hypotheses when your research is already conducted is often demoralizing. If you are operating on tight schedule with a limited budget you may simply not have the time or the money to redo the study. In that situation your only option is to discuss the limitations and hopefully apply what you have learnt later on in your academic career. 

The peace of mind that comes from getting accepted for publication in advance

After the reviews came back I did some major revisions to the preregistration and submitted it again. The journal editor was happy with the changes and gave me an in-principle acceptance. This meant that if I followed my research plan the journal guaranteed that they would publish my findings regardless of the results. Not only was I certain that I had tried my utmost to conduct research to the highest standards, I also knew that this meant that I would be able to finish my PhD on time since the requirements at my university was to have at least one published article when you finish. What a relief!

Diagnostic feedback

Since I had specified my theoretical rationale, my hypotheses, my sampling plan, my analyses, and my criteria for evaluating supporting evidence in advance it was much clearer to me what I was going to do and what I expected to find than in earlier projects. In contrast, during the "traditional" project I was constantly worried that my revisions were effected by hindsight bias. Following a revision my supervisor would often rejoice when he felt that the changes reflected our thinking at the onset of the project. Although it is possible that each revision clarified our original thinking, I feared that it was equally likely that the initial hypothesis was so vague that there were multiple ways to analyze and write up the results. I didn't want to get stuck in "I-knew-it-all-along" thinking, where I used the same data to generate both predictions and explanations of the findings. However, since we hadn't written down our hypothesis beforehand I felt that I had little to protect myself from falling into that trap. Having my predictions time stamped in black and white, for the registered report, really calmed my nerves and made me much more confident in interpreting the results. Not only did it make the feedback from the results more diagnostic since I had a clearer distinction between expected findings and unexpected findings, it also felt like a protective shield against my crippling worries.

It's Quicker

Many researchers are afraid to conduct preregistered research because they are pressed for time and worry that the additional preparation needed for a registered report will slow them down. Although I cannot speak for everyone, in my experience, publishing preregistered research is from start to finish a much shorter process. Going through the process helped me plan my study better and systematized my thinking before data collection began, which ultimately saved me a lot of time and grief since I made fewer irreversible mistakes. My registered report took less than a year to conduct, while the “traditional” research took five years. Although the two studies differ in more ways than one and are therefore not necessarily comparable, the time difference is still too big to be ignored! 

My advice to young researchers

I am now at the end of my graduate training and I just received my PhD. If I could share any advice with those who are just about to start their PhD education it would be this: Start with doing a preregistered study! Not only will you learn good scientific craftsmanship right from the start, but I suspect that your experience will be more personally and academically rewarding if you follow this path towards progress. 

Gró Einarsdóttir

Ph.D in Social Psychology

Related reading: Interview: Randy McCarthy discusses his experiences with publishing his first Registered Report

For more selfish reasons for preregistering your study:

Wagenmakers, E-J. & Dutilh, G. (2016). Seven selfish reasons for preregistration.

Further readings on the scientific merits of pre-registrations:

Nosek, B. A., & Bar-Anan, Y. (2012). Scientific utopia: I. Opening scientific communication. Psychological Inquiry, 23(3), 217-243.

Nosek, B. A., Ebersole, C. R., DeHaven, A. C., & Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences, 201708274.

Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7(6), 615-631.

John, L. K., Loewenstein, G., & Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524-532.

Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359-1366.