Many Labs 2 Project Overview


A team of 186 researchers conducted replications of 28 classic and contemporary findings in psychology.  The study examined the extent to which variability in replication success can be attributed to the study sample.  

Overall, 14 of the 28 findings failed to replicate despite the massive sample size, with more than 60 laboratories contributing samples from all over the world to test each finding.  If a finding replicated, it replicated in most samples with occasional variation in the magnitude of the findings.  If a finding was not replicated, it failed to replicate with little variation across samples and contexts. This evidence is inconsistent with a popular explanation that failures to replicate in psychology are likely due to changes in the sample between the original and replication study. 

This paper is the latest of six major replication projects in the social and behavioral sciences published since 2014.  These projects are a response to collective concern that the reproducibility of published findings may not be as robust as is assumed, particularly because of publication pressures that may lead to publication bias in which studies and findings with negative results are ignored or unpublished.

The results of the study reinforced changes in the field in recent years, including: 

  1. The importance of open sharing of all materials of published studies
  2. The value of preregistration in reducing bias and increase rigor
  3. The benefits of using Registered Reports to emphasize the importance of the research question and the quality of methodology.


Many Labs 2 Press Release


Many Labs 2 Project on OSF



Commentary from Original Authors

Many Labs 2 Final Paper 



Predicting replication outcomes in the Many Labs 2 study


Replication Research Team


    Corresponding Team

    Richard A. Klein
    Université Grenoble Alpes
    raklein22@gmail.com

    Michelangelo Vianello
    University of Padua
    michelangelo.vianello@gmail.com

    Fred Hasselman
    Radboud University Nijmegen
    f.hasselman@pwo.ru.nl

    Brian Nosek
    University of Virginia, 
    Center for Open Science
    nosek@cos.io


    Additional Authors

    Elizabeth L. Haines, Ph.D.
    William Paterson University
    HainesE@wpunj.edu

    Carmel Levitan 
    Occidental College
    levitan@oxy.edu

    Ronaldo Pilati
    University of Brasilia
    rpilati@gmail.com


    Original Authors

    Author Email
    Huang, Y., Tse, C. S., & Cho, K. W. (2014). Living in the north is not necessarily favorable: Different metaphoric associations between cardinal direction and valence in Hong Kong and in the United States. European Journal of Social Psychology Chi-Shing Tse - cstse@cuhk.edu.hk, Yanli Huang - yanlihuang1010@gmail.com
    Kay, A. C., Laurin, K., Fitzsimons, G. M., & Landau, M. J. (2013). A functional basis for structure-seeking: Exposure to structure promotes willingness to engage in motivated action. Journal of Experimental Psychology: General. Advance online publication. (Study 2) A. Kay - aaron.kay@duke.edu
    Alter, A. L., Oppenheimer, D. M., Epley, N., & Eyre, R. N. (2007). Overcoming intuition: metacognitive difficulty activates analytic reasoning. Journal of Experimental Psychology: General, 136, 569-576. Adam Alter - aalter@stern.nyu.edu
    Tversky, A., & Gati, I. (1978). Studies of similarity. Cognition and categorization, 1(1978), 79-98. Itamar Gati - itamar.gati@mail.huji.ac.il
    Graham, J., Haidt, J., & Nosek, B. A. (2009). Liberals and conservatives rely on different sets of moral foundations. Journal of Personality and Social Psychology, 96(5), 1029–1046. Jesse Graham - jesse.graham@usc.edu
    Rottenstreich, Y., & Hsee, C. K. (2001). Money, kisses, and electric shocks: On the affective psychology of risk. Psychological Science, 12(3), 185-190. (study 1) Yuval Rottenstreich - yuval@rady.ucsd.edu
    Bauer, M. A., Wilkie, J. E., Kim, J. K., & Bodenhausen, G. V. (2012). Cuing Consumerism Situational Materialism Undermines Personal and Social Well-Being. Psychological Science, 23(5), 517-523. Galen Bodenhausen - galen@northwestern.edu
    Miyamoto, Y, & Kitayama, S. (2002). Cultural variation in correspondence bias: The critical role of attitude diagnosticity of socially constrained behavior. Journal of Personality and Social Psychology, 83,1239-1248. Yuri Miyamoto - ymiyamoto@wisconsin.edu
    Inbar, Pizarro, Knobe, & Bloom (2009). Disgust sensitivity predicts intuitive disapproval of gays. Emotion, 9, 435-439. Yoel Inbar - yoelinbar@gmail.com, David Pizarro - pizzaro@gmail.com
    Critcher, C. R., & Gilovich, T. (2008). Incidental environmental anchors. Journal of Behavioral Decision Making, 21, 241-251. Clayton Critcher - claytoncritcher@berkeley.edu
    Van Lange, P. A. M., Otten, W., De Bruin, E. M. N., & Joireman, J. A. (1997). Development of prosocial, individualistic, and competitive orientations: Theory and preliminary evidence. Journal of Personality and Social Psychology, 4, 733 - 746.
    Hauser, M. et al. (2007). A Dissociation Between Moral Judgments and Justifications. Mind & Language, 22, 1-21. Fiery Cushman - Fiery_Cushman@brown.edu, Liane Young - liane.young@bc.edu
    Anderson, C., Kraus, M. W., Galinsky, A. D., & Keltner, D. (2012). The local-ladder effect social status and subjective well-being. Psychological science,23(7), 764-771. Cameron Anderson - anderson@haas.berkeley.edu
    Ross, L., Greene, D., & House, P. (1977). The “false consensus effect”: An egocentric bias in social perception and attribution processes. Journal of Experimental Social Psychology, 13(3), 279-301. (Study 1, supermarket) L. Ross - lross@stanford.edu
    Ross, L., Greene, D., & House, P. (1977). The “false consensus effect”: An egocentric bias in social perception and attribution processes. Journal of Experimental Social Psychology, 13(3), 279-301. (study 1, traffic ticket) L. Ross - lross@stanford.edu
    Giessner, S. R., & Schubert, T. W. (2007). High in the hierarchy: How vertical location and judgments of leaders’ power are interrelated. Organizational Behavior and Human Decision Processes, 104(1), 30-44. (Study 1a?) Steffen Giessner - sgiessner@rsm.nl
    Tversky, A., Kahneman, D. (1981). The framing of decisions and the psychology of choice. Science, 211, 453-458. Daniel Kahneman - kahneman@princeton.edu
    Hauser, M. et al. (2007). A Dissociation Between Moral Judgments and Justifications. Mind & Language, 22, 1-21. Fiery Cushman - Fiery_Cushman@brown.edu, Liane Young - liane.young@bc.edu
    Risen, J. L., & Gilovich, T. (2008). Why people are reluctant to tempt fate. Journal of personality and social psychology, 95(2), 293-307. Jane Risen - jane.risen@chicagobooth.edu
    Savani, K., Markus, H. R., Naidu, N. V. R., Kumar, S., & Berlia, N. (2010). What counts as a choice? US Americans are more likely than Indians to construe actions as choices. Psychological Science, 21(3), 391-398. K. Savani - ksavani@nus.edu.sg
    Norenzayan, A. et al. (2002). Cultural preferences for formal versus intuitive reasoning. Cogn. Sci. 26, 653–684 A Norenzayan - ara@psych.ubc.ca
    Hsee, C. K. (1998). Less is better: When low-value options are valued more highly than high-value options. Journal of Behavioral Decision Making, 11(2), 107-121. Christopher Hsee - Christopher.Hsee@chicagobooth.edu
    Gray, K., & Wegner, D. M. (2009). Moral typecasting: divergent perceptions of moral agents and moral patients. Journal of personality and social psychology,96(3), 505. (study 6) Kurt Gray - kurtgray@unc.edu
    Zhong, C. B., & Liljenquist, K. (2006). Washing away your sins: Threatened morality and physical cleansing. Science, 313, 1451–1452 Chenbo Zhong - Chenbo.Zhong@Rotman.Utoronto.Ca>, Katie Liljenquist - k.liljenquist@byu.edu
    Schwarz, N., Strack, F., & Mai, H. P. (1991). Assimilation and contrast effects in part-whole question sequences: A conversational logic analysis. Public Opinion Quarterly, 55, 3-23. Norbert Schwarz - norberts@usc.edu
    Shafir, E. (1993). Choosing versus rejecting: Why some options are both better and worse than others. Memory & Cognition, 21(4), 546-556. Eldar Shafir - shafir@princeton.edu
    Zaval, L., Keenan, E. A., Johnson, E. J., & Weber, E. U. (2014). How warm days increase belief in global warming. Nature Climate Change. Lisa Zaval - lisazaval@gmail.com
    Knobe, J. (2003). Intentional Action and Side Effects in Ordinary Language. Analysis, 63, 190-194. Joshua Knobe - joshua.knobe@yale.edu


    Independent Commentary


    The following researchers have made public comments that are skeptical of replication efforts in general, or question the credibility of the findings from prior replication efforts.


    Daniel Gilbert
    Harvard University
    gilbert@wjh.harvard.edu
    https://psychology.fas.harvard.edu/people/daniel-gilbert

    Timothy Wilson
    University of Virginia
    tdw@virginia.edu
    http://people.virginia.edu/~tdw/

    Susan Fiske
    Princeton University
    sfiske@princeton.edu
    https://psych.princeton.edu/person/susan-fiske

    Lisa Feldman Barrett
    Northeastern University
    l.barrett@neu.edu
    https://lisafeldmanbarrett.com/

    Fritz Strack
    University of Wuerzburg
    strack@psychologie.uni-wuerzburg.de
    http://strack.socialpsychology.org/

    Richard Nisbett
    University of Michigan
    nisbett@umich.edu
    http://www-personal.umich.edu/~nisbett/

    Wolfgang Stroebe
    Utrecht University/Groningen University
    W.Stroebe@uu.nl 
    https://stroebe.socialpsychology.org/

    These researchers have made public comments that are generally supportive of replication efforts and/or have expressed concerns that reproducibility needs to be improved.


    Richard Lucas
    Michigan State 
    lucasri@msu.edu
    https://msu.edu/~lucasri/lucas.html

    Simine Vazire
    UC Davis 
    simine@gmail.com
    https://www.simine.com/

    Sanjay Srivastava
    University of Oregon
    sanjay3k@gmail.com
    https://pages.uoregon.edu/sanjay/

    Rolf Zwaan
    Erasmus University
    zwaan@fsw.eur.nl
    https://www.egsh.eur.nl/people/rolf-zwaan/

    Betsy Levy Paluck
    Princeton University
    epaluck@princeton.edu
    http://www.betsylevypaluck.com/

    Neil Lewis
    Cornell University
    nlewisjr@cornell.edu
    https://neillewisjr.com/

    Hal Pashler
    UC San Diego
    hpashler@gmail.com
    http://www.psychology.ucsd.edu/people/profiles/hpashler.html

    Morton Gernsbacher
    University of Wisconsin-Madison
    magernsb@wisc.edu
    http://www.gernsbacherlab.org/

    John Ioannidis
    Stanford University
    jioannid@stanford.edu
    https://profiles.stanford.edu/john-ioannidis


    This website relies on cookies to help provide a better user experience. By clicking Accept or continuing to use the site, you agree. For more information, see our Privacy Policy and information on cookie use.