The Rise of Open Science in Psychology, A Preliminary Report

Open science is on the rise. Across disciplines, there are increasing rates of sharing data, making available underlying materials and protocols, and preregistering studies and analysis plans.  Hundreds of services have emerged to support open science behaviors at every stage of the research lifecycle.  But, what proportion of the research community is practicing open science? Where is penetration of these behaviors strongest and weakest?  Answers to these questions are important for evaluating progress in culture reform and for strategic planning of where to invest resources next.  

The hardest part of getting meaningful answers to these questions is quantifying the population that is NOT doing the behaviors.  For example, in a recent post, Nici Pfeiffer summarized the accelerating growth of OSF users on the occasion of hitting 150,000 registered users.  That number and non-linear growth suggests cultural movement associated with this one service, but how much movement?

To get better insight, we conducted a crowdsourcing project to assess market penetration of OSF among active, tenure-track faculty of psychology departments.  We started here because psychology is where OSF started. In a recent small study, we observed that in early 2013, 90% of new OSF users were from social-behavioral sciences; in early 2016, 57% of new users were from the social-behavioral sciences; and in early 2018, it was 44% of new users.  So, while the OSF user community is diversifying across research disciplines, psychology offers a good opportunity to assess its market penetration from its origins.

Obviously, OSF adoption by faculty is not a direct or complete indicator of open science behavior.  Researchers use a variety of tools to share their work and preregister studies. Also, the act of registering an account is just an entry point to the behaviors; it is not indicative of the frequency or comprehensiveness of open science actions.  Finally, in many research collaborations, faculty are probably less likely to be doing the open science actions than the student and post-doctoral collaborators (Ex: I have never done the final action of registering a study on OSF myself, despite being attached to many dozens of them).  These caveats are important for recognizing the meaning of the adoption rates. Even so, it is plausible that OSF adoption is positively correlated with other open science behaviors, and—because it is a popular service—the aggregate summary will provide complementary insight to other data collections such as an excellent survey by Betsy Levy Paluck, Ted Miguel, and their colleagues retrospectively documenting when social-behavioral researchers first did an open science action.

For our project, coders identified all active, tenure-track research faculty in a psychology department and then searched whether they had OSF accounts.  Because department websites reliably list all faculty, this provides an effective way to define a population and identify non-users. So far, we have coded 69 psychology departments and a total of 1,987 faculty.  Of them, 35% have OSF accounts. Based on Rogers’ (1962) classic diffusion of innovation model, this suggests that OSF adoption is well into the early majority.

There is also notable variability in adoption across nations, academic ranks, and subdisciplines.  For example, here is a table of adoption rates by country. Of those with reasonably large sample sizes, the UK, Netherlands, and Australia stand-out with stronger adoption; the US, Canada, and Germany have comparatively weaker adoption.

Similar magnitude of variation is observed across academic rank.  Assistant Professors and their equivalent depending on the national context have the highest adoption rates (44.6%), followed by Associate Professors (39.1%), and then Full Professors (31.3%).  This variation is even more interesting when the sample is separated by subdiscipline in the Table below.

Social Psychology stands out on adoption rates, already well into the late majority; and, near complete adoption among Assistant Professors in Social Psychology.  On the other end, the more applied areas of clinical, education, and health psychology are just barely past early adopter phase.

 Finally, it is interesting to observe adoption over time.  The Figure below charts the percentage of faculty adopting OSF by six-month intervals.  Overall, more Professors are adopters than Assistant Professors or Associate Professors, but that masks a base-rate difference—59% of the sample is comprised of Professors, and 21% each for Assistant Professors and Associate Professors.  Compared to base rates, Assistant (25%) and Associate (22%) Professors are somewhat over-represented among adopters as of 2019 indicating that the earlier career faculty are more likely to be adopters. However, the over-representation of Assistant Professors peaked in the first half of 2015 (30% of the total), and more senior faculty are now closing the gap.  Also, the rate of growth peaked in the 1st half of 2017 with 4.4% of all faculty joining during that six month span. The first half of 2019 is not over yet, but it may be similar to the second half of 2018’s 2.5% growth rate. If growth remains between 2.5% and 4% of all faculty every six months, then we can expect 50% overall to be adopters by between early 2021 and late 2022.

Conclusion

Using OSF adoption as a proxy for engagement with open science behavior, this analysis shows rapid growth during the last six years.  Coupled with survey findings directly measuring open science behaviors, the cumulative evidence suggests a rapid shifting of norms in psychology toward open science.  

The present crowdsourcing project is not complete.  If you would like to add your department, or another, or want to help us start to code other disciplines, take a look at this link for instructions about how to help.