Registered Reports and PhD’s – What? Why? How? An Interview with Chris Chambers

This interview was originally published in December 2018 of PsyPag Quarterly, Issue 109, and was reposted with permission from PsyPAG, which is supported by the British Psychological Society (BPS).

In the Registered Reports publishing model, peer review and in-principle acceptance (provisional acceptance for publication provided the authors follow their registered protocol) are based on the research question and methodological rigour, and happen before the study is run and results are obtained. In addition to the benefits to science as a whole, Registered Reports provide many positives for the researcher. I interviewed Chris Chambers, professor of cognitive neuroscience at Cardiff University, Chair of the Registered Reports Committee supported by the Center for Open Science, and one of the founders of Registered Reports. I spoke to Chris about why we need Registered Reports and specifically how they can work for PhD students. 

Emma: Could you briefly describe what Registered Reports are, and why they are important for psychology?

Chris: The standard model of publishing is that you design your research, conduct it, analyse your results and interpret your results. Then you write a paper, send it to a journal as a package and then it is reviewed as a package. Editors and reviewers look at the importance of the question, the rigour of the method, but they also look at the results. They judge the significance of the research based upon the outcomes, and they make a decision as to whether you should get published or not. But this model comes with a price, which is that results end up influencing “publishability”. Whether you get positive results often ends up determining whether or not you get published, and when you put researchers under pressure to get positive results, they will give you positive results. They will do it by changing their predictions, by p-hacking, by engaging in questionable practices and so on.

So Registered Reports try to break this outcome-based system by making results a dead currency. To do that you have to split the review process into two phases: One in which you only review rationale, question, theory, method, before the results are known, before you do the research and then that goes out to review. And one in which after the results are in you check whether the authors did what they said they would, whether their conclusions are based on the evidence that they have obtained. That is the Registered Reports model. So, the Registered Reports model involves the stage 1 review process where the protocol is assessed before results are known, and based on peer review of that protocol the journal guarantees to publish the final article regardless of outcome.

Emma: That sounds amazing for psychology and for science generally. For PhD students are there any personal benefits in writing a Registered Report?

Chris: One of the key benefits for students is that they gain an understanding of what’s required to do minimally-biased deductive research, by which I mean research in which the reliability of the outcome is as protected as possible from our own fallibility as human researchers. We see what we want to see in data, and I think when you go through this Registered Reports process as a researcher, whether you’re a PhD student or a professor, you learn an awful lot about the danger of bias. So there’s a pedagogical element to it, you really learn how to do science properly if you’re doing hypothesis testing. I think from the very basic point of view for a PhD student you will learn the right things at the right time. Practically, one of the advances is that if you get started with this early on in your PhD you will end up producing outcomes sooner. The reason for that is when you break the review process up into two stages you are much less likely to get rejected at the first stage because the reviews that come in at that point can help you fix the research before it’s too late. Then when you come back at the second stage a large quantity of that review has already been undertaken, so you’re not re-litigating those aspects that have already been assessed. By the time you finish your data acquisition and analysis you’re much closer to the end destination of publication than you would otherwise be if you went down the traditional route where now you have to package it, sell it to a journal, get rejected, and so on. I think there is a fundamental scientific benefit, and there is also a career benefit. That career benefit isn’t just about doing things in a shorter amount of time, but also about having more to show on your CV. If you get to the end of your PhD and you’ve primarily done Registered Reports, then you’ll have more outputs because they will be undertaken within the timeframe of the PhD. Even those Registered Reports that are not completed at stage 2 by the time you’ve finished your PhD will be provisionally accepted, so on your CV there will be something that has already undergone peer review and is almost complete. That sounds very careerist, actually the main benefit is the scientific benefit; the benefit of understanding what is required to design an experiment properly, to know proper statistical preparation. Statistics isn’t just about analysing data so that you get significant results, it’s about knowing how to plan a study properly so that you have adequate power, that you have a proper analysis plan that can falsify a prediction based on theory, and using a well operationalised variable that’s relevant for that theory. These are all issues that are best addressed when you don’t know results, because results blind us to other things.

Emma: So if I want to write a Registered Report where should I start?

Chris: The place to begin is: What’s your question? Every PhD student really should begin by trying to define their question. What question, or small set of questions are you really trying to address? Break it down into something that is definable. If you’re doing hypothesis-driven research then think about your theory, and then from there generate predictions based on theory. The best Registered Reports are the ones that say there is a particular theory or big question in this area, there is a controversy or an uncertainty in which this, this, or this could be true, and regardless of the outcome it’s important to know the answer. The way to begin is really to think about the question and of course the method. And not to rush it too much, take your time to design that first study carefully.

Emma: Speaking of time, do you have any specific advice for PhD students on how to fit Registered Reports into the short timescales of a PhD?

Chris: Get started early, like you did. The first Registered Report could be a replication because you learn so much from a replication. You learn a lot of the basic statistical methods that you need to know. You learn to ask why it’s worth answering a particular question.

Emma: And when you say early?

Chris: Within the first two years. If you think about a PhD taking three years I would say that doing a Registered Report in the third year can be challenging, just because of the amount of planning that’s required. I have students doing Registered Reports in the first two years no problem. Ideally in the first year because you can generate a protocol and submit it to a journal. There will be a downtime of maybe eight to twelve weeks while it’s being reviewed but you can do other things in the meantime – you can work on another one, you can work on an exploratory study which is not registered, you can start thinking more about theory. There are lots of things you can be doing; multitasking, essentially.

Emma: Do you have any advice on persuading a supervisor to try Registered Reports if they’re not keen?

Chris: You’ve got to put the question to them in the first place. You will learn a lot about what kind of scientist they are from the answer they give you. I think that’s instructive for you to know as a student. If they respond very negatively to the idea it might be because they have a lab which has a particular brand; they have a certain theory or a result which they always have to produce. A lot of ECR’s are stuck in this brand-generating machine of a lab where they have a huge file-drawer of results which are inconsistent with the lab’s brand, and the lab only publishes results that are consistent with their brand. These little empires exist all over science. If you have huge file-drawers as a PI, then the Registered Reports model is a threat because it prevents you from selecting what to publish based on results, so it takes away the file-drawer effect. They may well react positively, and a lot of people do. But some will react negatively and it’s important to know what do in that situation. It may be the case that you can convince them by talking about the benefits for your career from doing this: That a Registered Report is good training for you, it’s a publication regardless of outcome in a good journal. It’s a way of generating and signalling to the community that you care about transparency and reproducibility, and ultimately they’re well cited. It’s a way of producing a unit of knowledge that is probably more reliable than your average paper. Which is frankly what we’re here for, right! Maybe that won’t be enough, and you can talk to them about ways that it’s changing the incentive structures in science. If you’re in a lab that has a very strong brand and runs up against opponents who try to kill that brand, a Registered Report can disarm those opponents by preventing rivals from blocking publications from that lab based upon the results. It disarms everyone; results are a currency and Registered Reports neutralises them.

Emma: Would you recommend having that conversation before you start your PhD?

Chris: Yes, it certainly couldn’t hurt, particularly if you have multiple options in front of you. In those situations you’re interviewing them as much as they’re interviewing you. One thing that a Registered Report does, if you’re able to pursue it as a student or indeed at any level, it de-stresses your life because you know that you are no longer a slave to your results. You no longer have to pray for the significant effect, you no longer have to sit on the edge of your seat, you no longer have to be tempted to p-hack, or change your hypothesis to fit your results, because now the results are simply the results and you can do good science.

Emma: You mentioned that Registered Reports are well cited, how do they compare to a standard paper?

Chris: On average, across all citation indexes, they are cited at or above the average impact factors of the journals in which they’re published. Somewhere between basically equivalent to the impact factor to 200% of the impact factor. Registered Reports are more likely to produce negative results, because when you eliminate bias from a system that’s full of bias towards positive results, you will inevitably generate more negative results. Even though they will generate more negative results, but more believable results, they are cited just as much. And that’s encouraging because it means that despite all the corrupt incentives in science, we are still scientists, and we still care about results that we can believe more than results that we question.

Emma: What is the future of Registered Reports?

Chris: I would like them offered at every empirical journal that publishes hypothesis-driven research. I don’t think it should be mandatory, I don’t think it should be the only way to publish research because there are legitimate reasons for doing exploratory science, mining datasets for patterns and signals without any particular hypothesis and question. There are many good reasons for poking around with experiments and just trying things out and piloting things. That’s all great, we shouldn’t be turning that off. But what we should be doing is creating an alternative choice that people can follow if they want to. On top of it being an option at all journals, I would also like to see it being offered by funding agencies as a track that authors and grantees can follow. You apply for Registered Report grants, which are accepted by journals and funders at the same time. So on the same day you get your funding awarded, you get your paper accepted.

Emma: Where should I go to if I want to do a Registered Report?

Chris: If you go to the website cos.io/rr that’s the central knowledge hub for Registered Reports. It contains a full description of what they are and why they’re important. It includes a complete, curated list of all journals that currently offer them, which as of today is 97 journals* [101 when I transcribed this a few days later! - EH]. It includes links to the database for published Registered Reports, materials for authors, reviewers, and editors, and lots of FAQ’s. You can also ask me questions by contacting me on email or Twitter: @chrisdc77

Correspondence:

Emma Henderson

PhD Student, Department of Management, Kingston University London

Emma.Henderson@kingston.ac.uk

* This number has grown to 195 since this publication. For an up-to-date list, please see cos.io/rr