The Rise of Open Science in Psychology, A Preliminary Report

June 3rd, 2019, Brian Nosek


diffusion of innovation model


Open science is on the rise. Across disciplines, there are increasing rates of sharing data, making available underlying materials and protocols, and preregistering studies and analysis plans.  Hundreds of services have emerged to support open science behaviors at every stage of the research lifecycle.  But, what proportion of the research community is practicing open science? Where is penetration of these behaviors strongest and weakest?  Answers to these questions are important for evaluating progress in culture reform and for strategic planning of where to invest resources next.  


The hardest part of getting meaningful answers to these questions is quantifying the population that is NOT doing the behaviors.  For example, in a recent post, Nici Pfeiffer summarized the accelerating growth of OSF users on the occasion of hitting 150,000 registered users.  That number and non-linear growth suggests cultural movement associated with this one service, but how much movement?


To get better insight, we conducted a crowdsourcing project to assess market penetration of OSF among active, tenure-track faculty of psychology departments.  We started here because psychology is where OSF started. In a recent small study, we observed that in early 2013, 90% of new OSF users were from social-behavioral sciences; in early 2016, 57% of new users were from the social-behavioral sciences; and in early 2018, it was 44% of new users.  So, while the OSF user community is diversifying across research disciplines, psychology offers a good opportunity to assess its market penetration from its origins.


Obviously, OSF adoption by faculty is not a direct or complete indicator of open science behavior.  Researchers use a variety of tools to share their work and preregister studies. Also, the act of registering an account is just an entry point to the behaviors; it is not indicative of the frequency or comprehensiveness of open science actions.  Finally, in many research collaborations, faculty are probably less likely to be doing the open science actions than the student and post-doctoral collaborators (Ex: I have never done the final action of registering a study on OSF myself, despite being attached to many dozens of them).  These caveats are important for recognizing the meaning of the adoption rates. Even so, it is plausible that OSF adoption is positively correlated with other open science behaviors, and—because it is a popular service—the aggregate summary will provide complementary insight to other data collections such as an excellent survey by Betsy Levy Paluck, Ted Miguel, and their colleagues retrospectively documenting when social-behavioral researchers first did an open science action.


For our project, coders identified all active, tenure-track research faculty in a psychology department and then searched whether they had OSF accounts.  Because department websites reliably list all faculty, this provides an effective way to define a population and identify non-users. So far, we have coded 69 psychology departments and a total of 1,987 faculty.  Of them, 35% have OSF accounts. Based on Rogers’ (1962) classic diffusion of innovation model, this suggests that OSF adoption is well into the early majority.

diffusion of innovation model

There is also notable variability in adoption across nations, academic ranks, and subdisciplines.  For example, here is a table of adoption rates by country. Of those with reasonably large sample sizes, the UK, Netherlands, and Australia stand-out with stronger adoption; the US, Canada, and Germany have comparatively weaker adoption.


OSF adoption by nation
Similar magnitude of variation is observed across academic rank.  Assistant Professors and their equivalent depending on the national context have the highest adoption rates (44.6%), followed by Associate Professors (39.1%), and then Full Professors (31.3%).  This variation is even more interesting when the sample is separated by subdiscipline in the Table below.

OSF adoption by academic rank

Social Psychology stands out on adoption rates, already well into the late majority; and, near complete adoption among Assistant Professors in Social Psychology.  On the other end, the more applied areas of clinical, education, and health psychology are just barely past early adopter phase.


Finally, it is interesting to observe adoption over time.  The Figure below charts the percentage of faculty adopting OSF by six-month intervals.  Overall, more Professors are adopters than Assistant Professors or Associate Professors, but that masks a base-rate difference—59% of the sample is comprised of Professors, and 21% each for Assistant Professors and Associate Professors.  Compared to base rates, Assistant (25%) and Associate (22%) Professors are somewhat over-represented among adopters as of 2019 indicating that the earlier career faculty are more likely to be adopters. However, the over-representation of Assistant Professors peaked in the first half of 2015 (30% of the total), and more senior faculty are now closing the gap.  Also, the rate of growth peaked in the 1st half of 2017 with 4.4% of all faculty joining during that six month span. The first half of 2019 is not over yet, but it may be similar to the second half of 2018’s 2.5% growth rate. If growth remains between 2.5% and 4% of all faculty every six months, then we can expect 50% overall to be adopters by between early 2021 and late 2022.


OSF Adoption by Psychology Faculty over time


Conclusion


Using OSF adoption as a proxy for engagement with open science behavior, this analysis shows rapid growth during the last six years.  Coupled with survey findings directly measuring open science behaviors, the cumulative evidence suggests a rapid shifting of norms in psychology toward open science.  


The present crowdsourcing project is not complete.  If you would like to add your department, or another, or want to help us start to code other disciplines, take a look at this link for instructions about how to help.



Recent Blogs

The Content of Open Science

What Second Graders Can Teach Us About Open Science

What's Going on With Reproducibility?

Open Science and the Marketplace of Ideas

3 Things Societies Can Do to Promote Research Integrity

How to Manage and Share Your Open Data

Interview with Prereg Challenge Award Winner Dr. Allison Skinner

Next Steps for Promoting Transparency in Science

Public Goods Infrastructure for Preprints and Innovation in Scholarly Communication

A How-To Guide to Improving the Clarity and Continuity of Your Preregistration

Building a Central Service for Preprints

Three More Reasons to Take the Preregistration Challenge

The Center for Open Science is a Culture Change Technology Company

Preregistration: A Plan, Not a Prison

How can we improve diversity and inclusion in the open science movement?

OSF Fedora Integration, Aussie style!

Replicating a challenging study: it's all about sharing the details.

Some Examples of Publishing the Research That Actually Happened

How Preregistration Helped Improve Our Research: An Interview with Preregistration Challenge Awardees

Are reproducibility and open science starting to matter in tenure and promotion review?

The IRIS Replication Award and Collaboration in the Second Language Research Community

We Should Redefine Statistical Significance

Some Cool New OSF Features

How Open Source Research Tools Can Help Institutions Keep it Simple

OSF Add-ons Help You Maximize Research Data Storage and Accessibility

10 Tips for Making a Great Preregistration

Community-Driven Science: An Interview With EarthArXiv Founders Chris Jackson, Tom Narock and Bruce Caron

A Preregistration Coaching Network

Why are we working so hard to open up science? A personal story.

One Preregistration to Rule Them All?

Using the wiki just got better.

Transparent Definitions and Community Signals: Growth in the Open Science Community

We're Committed to GDPR. Here's How.

Preprints: The What, The Why, The How.

The Prereg Challenge Is Ending. What's Next?

We are Now Registering Preprint DOIs with Crossref

Using OSF in the Lab

Psychology's New Normal

How Open Commenting on Preprints Can Increase Scientific Transparency: An Interview With the Directors of PsyArxiv, SocArxiv, and Marxiv

The Landscape of Open Data Policies

Open Science is a Behavior.

Why pre-registration might be better for your career and well-being

Interview: Randy McCarthy discusses his experiences with publishing his first Registered Report

Towards minimal reporting standards for life scientists

Looking Back on the Prereg Challenge and Forward To More Credible Research

OSF: Origin, growth, and what’s next

A Critique of the Many Labs Projects

The Rise of Open Science in Psychology, A Preliminary Report

Strategy for Culture Change

New OSF Registries Enhancements Improve Efficiency and Quality of Registrations

Registered Reports and PhD’s – What? Why? How? An Interview with Chris Chambers

How to Collaborate with Industry Using Open Science

How to Build an Open Science Network in Your Community

Seven Reasons to Work Reproducibly

COS Collaborates with Case Medical Research to Support Free Video Publishing for Clinicians and Researchers

Advocating for Policy Improvements at Your Institution

Announcing a joint effort to improve research transparency: FAIRSharing and TOP Guidelines

OSF as a tool for managing course-embedded research projects

Journals test the Materials Design Analysis Reporting (MDAR) checklist

Now you can endorse papers on OSF Preprints with Plaudit

Many Labs 4: Failure to Replicate Mortality Salience Effect With and Without Original Author Involvement

This website relies on cookies to help provide a better user experience. By clicking Accept or continuing to use the site, you agree. For more information, see our Privacy Policy and information on cookie use.