Many Labs 2 Project Overview

A team of 186 researchers conducted replications of 28 classic and contemporary findings in psychology.  The study examined the extent to which variability in replication success can be attributed to the study sample.  

Overall, 14 of the 28 findings failed to replicate despite the massive sample size, with more than 60 laboratories contributing samples from all over the world to test each finding.  If a finding replicated, it replicated in most samples with occasional variation in the magnitude of the findings.  If a finding was not replicated, it failed to replicate with little variation across samples and contexts. This evidence is inconsistent with a popular explanation that failures to replicate in psychology are likely due to changes in the sample between the original and replication study. 

This paper is the latest of six major replication projects in the social and behavioral sciences published since 2014.  These projects are a response to collective concern that the reproducibility of published findings may not be as robust as is assumed, particularly because of publication pressures that may lead to publication bias in which studies and findings with negative results are ignored or unpublished.

The results of the study reinforced changes in the field in recent years, including: 

  1. The importance of open sharing of all materials of published studies
  2. The value of preregistration in reducing bias and increase rigor
  3. The benefits of using Registered Reports to emphasize the importance of the research question and the quality of methodology.

Many Labs 2 Press Release

Many Labs 2 Project on OSF

Commentary from Original Authors

Many Labs 2 Final Paper 

Predicting replication outcomes in the Many Labs 2 study

Replication Research Team

    Corresponding Team

    Richard A. Klein
    Université Grenoble Alpes

    Michelangelo Vianello
    University of Padua

    Fred Hasselman
    Radboud University Nijmegen

    Brian Nosek
    University of Virginia, 
    Center for Open Science

    Additional Authors

    Elizabeth L. Haines, Ph.D.
    William Paterson University

    Carmel Levitan 
    Occidental College

    Ronaldo Pilati
    University of Brasilia

    Original Authors

    Author Email
    Huang, Y., Tse, C. S., & Cho, K. W. (2014). Living in the north is not necessarily favorable: Different metaphoric associations between cardinal direction and valence in Hong Kong and in the United States. European Journal of Social Psychology Chi-Shing Tse -, Yanli Huang -
    Kay, A. C., Laurin, K., Fitzsimons, G. M., & Landau, M. J. (2013). A functional basis for structure-seeking: Exposure to structure promotes willingness to engage in motivated action. Journal of Experimental Psychology: General. Advance online publication. (Study 2) A. Kay -
    Alter, A. L., Oppenheimer, D. M., Epley, N., & Eyre, R. N. (2007). Overcoming intuition: metacognitive difficulty activates analytic reasoning. Journal of Experimental Psychology: General, 136, 569-576. Adam Alter -
    Tversky, A., & Gati, I. (1978). Studies of similarity. Cognition and categorization, 1(1978), 79-98. Itamar Gati -
    Graham, J., Haidt, J., & Nosek, B. A. (2009). Liberals and conservatives rely on different sets of moral foundations. Journal of Personality and Social Psychology, 96(5), 1029–1046. Jesse Graham -
    Rottenstreich, Y., & Hsee, C. K. (2001). Money, kisses, and electric shocks: On the affective psychology of risk. Psychological Science, 12(3), 185-190. (study 1) Yuval Rottenstreich -
    Bauer, M. A., Wilkie, J. E., Kim, J. K., & Bodenhausen, G. V. (2012). Cuing Consumerism Situational Materialism Undermines Personal and Social Well-Being. Psychological Science, 23(5), 517-523. Galen Bodenhausen -
    Miyamoto, Y, & Kitayama, S. (2002). Cultural variation in correspondence bias: The critical role of attitude diagnosticity of socially constrained behavior. Journal of Personality and Social Psychology, 83,1239-1248. Yuri Miyamoto -
    Inbar, Pizarro, Knobe, & Bloom (2009). Disgust sensitivity predicts intuitive disapproval of gays. Emotion, 9, 435-439. Yoel Inbar -, David Pizarro -
    Critcher, C. R., & Gilovich, T. (2008). Incidental environmental anchors. Journal of Behavioral Decision Making, 21, 241-251. Clayton Critcher -
    Van Lange, P. A. M., Otten, W., De Bruin, E. M. N., & Joireman, J. A. (1997). Development of prosocial, individualistic, and competitive orientations: Theory and preliminary evidence. Journal of Personality and Social Psychology, 4, 733 - 746.
    Hauser, M. et al. (2007). A Dissociation Between Moral Judgments and Justifications. Mind & Language, 22, 1-21. Fiery Cushman -, Liane Young -
    Anderson, C., Kraus, M. W., Galinsky, A. D., & Keltner, D. (2012). The local-ladder effect social status and subjective well-being. Psychological science,23(7), 764-771. Cameron Anderson -
    Ross, L., Greene, D., & House, P. (1977). The “false consensus effect”: An egocentric bias in social perception and attribution processes. Journal of Experimental Social Psychology, 13(3), 279-301. (Study 1, supermarket) L. Ross -
    Ross, L., Greene, D., & House, P. (1977). The “false consensus effect”: An egocentric bias in social perception and attribution processes. Journal of Experimental Social Psychology, 13(3), 279-301. (study 1, traffic ticket) L. Ross -
    Giessner, S. R., & Schubert, T. W. (2007). High in the hierarchy: How vertical location and judgments of leaders’ power are interrelated. Organizational Behavior and Human Decision Processes, 104(1), 30-44. (Study 1a?) Steffen Giessner -
    Tversky, A., Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 211, 453-458. Daniel Kahneman -
    Hauser, M. et al. (2007). A Dissociation Between Moral Judgments and Justifications. Mind & Language, 22, 1-21. Fiery Cushman -, Liane Young -
    Risen, J. L., & Gilovich, T. (2008). Why people are reluctant to tempt fate. Journal of personality and social psychology, 95(2), 293-307. Jane Risen -
    Savani, K., Markus, H. R., Naidu, N. V. R., Kumar, S., & Berlia, N. (2010). What counts as a choice? US Americans are more likely than Indians to construe actions as choices. Psychological Science, 21(3), 391-398. K. Savani -
    Norenzayan, A. et al. (2002). Cultural preferences for formal versus intuitive reasoning. Cogn. Sci. 26, 653–684 A Norenzayan -
    Hsee, C. K. (1998). Less is better: When low-value options are valued more highly than high-value options. Journal of Behavioral Decision Making, 11(2), 107-121. Christopher Hsee -
    Gray, K., & Wegner, D. M. (2009). Moral typecasting: divergent perceptions of moral agents and moral patients. Journal of personality and social psychology,96(3), 505. (study 6) Kurt Gray -
    Zhong, C. B., & Liljenquist, K. (2006). Washing away your sins: Threatened morality and physical cleansing. Science, 313, 1451–1452 Chenbo Zhong - Chenbo.Zhong@Rotman.Utoronto.Ca>, Katie Liljenquist -
    Schwarz, N., Strack, F., & Mai, H. P. (1991). Assimilation and contrast effects in part-whole question sequences: A conversational logic analysis. Public Opinion Quarterly, 55, 3-23. Norbert Schwarz -
    Shafir, E. (1993). Choosing versus rejecting: Why some options are both better and worse than others. Memory & Cognition, 21(4), 546-556. Eldar Shafir -
    Zaval, L., Keenan, E. A., Johnson, E. J., & Weber, E. U. (2014). How warm days increase belief in global warming. Nature Climate Change. Lisa Zaval -
    Knobe, J. (2003). Intentional Action and Side Effects in Ordinary Language. Analysis, 63, 190-194. Joshua Knobe -

    Independent Commentary

    The following researchers have made public comments that are skeptical of replication efforts in general, or question the credibility of the findings from prior replication efforts.

    Daniel Gilbert
    Harvard University

    Timothy Wilson
    University of Virginia

    Susan Fiske
    Princeton University

    Lisa Feldman Barrett
    Northeastern University

    Fritz Strack
    University of Wuerzburg

    Richard Nisbett
    University of Michigan

    Wolfgang Stroebe
    Utrecht University/Groningen University

    These researchers have made public comments that are generally supportive of replication efforts and/or have expressed concerns that reproducibility needs to be improved.

    Richard Lucas
    Michigan State

    Simine Vazire
    UC Davis

    Sanjay Srivastava
    University of Oregon

    Rolf Zwaan
    Erasmus University

    Betsy Levy Paluck
    Princeton University

    Neil Lewis
    Cornell University

    Hal Pashler
    UC San Diego

    Morton Gernsbacher
    University of Wisconsin-Madison

    John Ioannidis
    Stanford University

    This website relies on cookies to help provide a better user experience. By clicking Accept or continuing to use the site, you agree. For more information, see our Privacy Policy and information on cookie use.