Improving the Credibility of Scientific Research: Social Psychology releases special issue of 15 Registered Reports attempting to replicate important results in social psychology

May 19, 2014, Charlottesville, Virginia, USA. Journal articles are the currency of science, but incentives favoring novelty over accuracy potentially undermine the credibility of the scientific literature. A special issue of the journal Social Psychology, published today, adopts a new Registered Reports format as a way to shift the incentives and increase the credibility of published results.

Registered Reports emphasize the importance of the research question and the quality of methodology by conducting the peer review prior to data collection and analysis. Accepted papers then are virtually guaranteed publication in the journal if the authors follow through with the registered methodology. “Registered reports eliminates the bias against negative results in publishing because the results are not known at the time of review” said Daniel Simons, Professor at University of Illinois, Urbana-Champaign and co-Editor of Registered Replication Reports at Perspectives on Psychological Science. Chris Chambers, Senior Research Fellow at Cardiff University, section editor at Cortex and AIMS Neuroscience, and chair of the Registered Reports Committee supported by the Center for Open Science (COS) adds, “Because the study is accepted in advance, the incentives for authors change from producing the most beautiful story to producing the most accurate one.”

The special issue of Social Psychology contains 15 papers, all of which are Registered Reports (RR) and all of which were designed to replicate important classic or contemporary results from social psychology. According to Daniel Lakens, Assistant Professor at Eindhoven University of Technology and co-Editor of the special issue, “This special issues demonstrates the viability of the Registered Reports model and also provides substantial evidence about a variety of important areas of research in social psychology.”

The 27 original findings replicated in these papers cover a wide range of topics such as the primacy of warmth in forming impressions of people’s personality (Original: Asch 1946; Replication: Nauts, Langner, Huijsmans, Vonk, & Wigboldus, 2014), the “Romeo and Juliet” effect in romantic relationships (Original: Driscoll, Davis, & Lipetz, 1972; Replication: Sinclair, Hood, & Wright, 2014), effects of stereotypes on math performance (Original: Shih, Pittinsky, & Ambady, 1999; Replications: Gibson, Losee, & Vitiello, 2014; Moon & Roeder, 2014), the effect of experiencing physical warmth on judgments of interpersonal warmth (Original: Williams & Bargh, 2008; Replication: Lynott, Corker, Wortman, Connell, Donnellan, Lucas, & O’Brien, 2014), and the impact of recalling moral behavior on the perception of brightness (Original: Banerjee, Chatterjee, & Sinha, 2012; Replication: Brandt, IJzerman, & Blanken, 2014).

Christian Unkelbach, Professor at the University of Cologne and Editor-in-Chief of Social Psychology, concluded “We are very pleased with the special issue in demonstrating the potential value of RR’s in advancing scientific research and communication. We are adopting the model as a regular submission option for the journal, and we encourage other journals to do so.”

Selected Results

  • A replication of a classic study by Asch (1946) that had been done without formal inferential statistical analysis allowed Nauts, Langner, Huijsmans, Vonk, and Wigboldus (2014) to re-examine whether warmth is a central trait when forming impressions. They found no evidence that warmth is more important than other traits, such as intelligence, and they remind readers of the fact that Asch never interpreted his own work in the way it is now cited in Introduction to Psychology textbooks.
  • The Many Labs replication project (Klein et al., 2014) was a large collaboration in which 36 research teams from around the world collected data from 6,344 participants to assess variation in replicability of 13 classic and contemporary effects across samples and settings. This large project revealed relatively little variation in effect sizes across labs, and demonstrated that crowdsourcing offers a feasible way to collect large sample sizes and gain substantial knowledge about replicability.
  • Three research teams (Lynott et al., 2014) collaboratively replicated a study on the relationship between physical warmth and judgments of personal warmth (Williams & Bargh, 2008) which has been widely cited in the media and the scientific literature. The researchers collected data from 861 participants, but did not observe an effect of physical warmth on prosociality of reward choice.
  • Zezelj & Jokic (2014) attempted to resolve the discrepancies between the results of previous findings on the effect that psychological distance from an event influences has on how people think about that event. They highlight subtle differences in the experimental paradigms that could explain the discrepancy. This article is accompanied by a discussion among the original authors and the replication team in which new data are brought to bear.

Quoted Contacts

Other Contacts

  • Brian Nosek: co-Editor of the special issue; Associate Professor, University of Virginia; Director, Center for Open Science;
  • George Banks: member Registered Reports Committee; Assistant Professor Longwood University;
  • Alex Holcombe: co-Editor Registered Replication Reports, Perspectives on Psychological Science; member Registered Reports Committee;
  • Edward (Ted) Miguel: Director of Berkeley Initiative for Transparency in the Social Sciences (BITSS); member Registered Reports Committee; Professor, University of California, Berkeley;

Frequently Asked Questions

If a replication fails to obtain the original result, does that mean that the original is false?

No. It could also occur if there were an error in the replication, or if the replication were underpowered, or if there were changes in the sample, setting, materials, or procedure that were critical for obtaining the original result. The replications in the special issue were highly powered and designed to eliminate any differences from the original that were known to be critical in obtaining the result. With any replication, though, there could be differences in methodology that had not yet been recognized as important in obtaining the effect. Direct replications like these can improve our understanding of the conditions necessary to obtain an effect.

Does the proportion of studies successfully replicated in the special issue say something about the replicability of social psychology more generally?
No. Authors self-selected the studies they wanted to replicate. The peer review process evaluated each submission independently (only 15 of the original 40 submissions were included in the special issue). Consequently, the collection of studies is highly selective, and the proportion of successful replications applies only to the particular set of studies chosen for this issue.

Does a failure to replicate an original result suggest that the original was obtained fraudulently?
No. Failure to replicate a result is an ordinary event. Fraud is an extraordinary event.

Does a failure to replicate mean that the replication attempt was flawed?
No. It could indicate a problem with the replication. But, it could also indicate that there was an error in the original result, that the original result was a false positive, or that the methodological differences between the replication and the original study are critical for obtaining the effect.

That’s a lot of no’s. Then what does a failure to replicate mean?
If the results of the original and replication differ, they are two data points for understanding the investigated effect. Only rarely is a single study definitive. Evidence for a phenomenon is a slow accumulation of results across a variety of approaches. Differences among those results is fodder for researchers to better understand the effect in terms of when and how it occurs. The replications in the special issue follow a variety of good practices to ensure that they are highly informative for understanding the reproducibility of the original results including: (a) high-powered tests (i.e., large samples) to minimize likelihood of false negatives, (b) pre-registration of design and analysis plans to maximize the quality and interpretability of the confirmatory research designs, (c) peer review prior to data collection to maximize the quality of the methodology, and (d) explicit discussion of differences between original design and replication to identify potential reasons for different results, or to illustrate the generalizability of the original effect for successful replications.