Why are we working so hard to open up science? A personal story.

December 8th, 2017, Brian Nosek


inner demons

In the Fall of 2011, Sarah Mackenzie, the maid of honor at my wedding, was diagnosed with a rare form of ovarian cancer. Sarah and her family were motivated to learn as much as they could about the disease to advocate for her care. They weren’t scientists, but they started searching the literature for relevant articles. One evening, Sarah called us, angry. Every time she found an article that might be relevant to understanding her disease, she ran into a paywall requiring $15-$40 to access it. Public money had paid for the research, yet she was barred from making any use of it. Luckily, she had us. Most people in Sarah’s position don’t have the luxury of friends at wealthy academic institutions with subscriptions to the literature.  

During this time, I was pursuing an interest in the business models of scholarly communication. I wanted to understand the ways in which these models interfered with the dissemination of knowledge that could improve quality of life. Sarah’s experience illustrated one key part of the problem--the outcomes of research should be public goods, but the business models of publishing make them exclusive goods. Lack of access to published literature limits our ability to apply what we know to improving others’ quality of life. If doctors can’t access the literature, they can’t keep up with the latest innovations for care. If policy makers can’t access the literature, they can’t create evidence based policies. To advance solutions and cures, the outcomes of research must be open.

During the next few years, Sarah went through the standard treatments with mixed success.  She needed new solutions and was fortunate to qualify for a couple of clinical trials testing new therapies. They were difficult. Sometimes the side effects would have her in the hospital for weeks at a time, or at home but unable to interact with her young boys.

Simultaneously, I was learning a lot about publication bias. For example, despite a requirement that results be reported within 24 months of the end of a trial, less than 50% of trials were so reported. Trials were funded. Patients were put through treatments often with terrible side effects. And, by failing to report the study results there is zero return on investment and needless suffering for vulnerable people.

Moreover, evidence shows that there is bias in which results are reported. Positive results are more exciting and therefore more publishable, negative results are boring or disappointing and more likely to be left out. But, negative data is immensely valuable for getting to important solutions faster and for making evidence-based decisions about what not to invest in next.  Failing to share negative data fuels tragic cycles of dead-end research investments. To advance solutions and cures, the data from research must be open.

The pressures for finding positive results produce additional problems for the content and credibility of research that does get reported. Outcome switching is rampant.  In other words, what researchers observe is often different than what they report. Evidence suggests that rampant selection bias, often occurring unintentionally, produces reported findings that are much rosier than what researchers actually observed. For the doctors, policymakers, child care providers, elder care providers, educators, members of national security apparatus and so many others who depend on published research to inform their choices and strategies, selective reporting results in false hope and ineffective action. To advance the human condition, the process of generating, analyzing, and reporting the data must be open.

Sarah, and many many others like her, can’t access the literature that their tax dollars funded, are being used for research that never gets reported, and are being misled by a literature that looks more beautiful than the actual evidence suggests. This is unethical, both in terms of the use of research dollars and the sacrifices experimental patients like Sarah make to test possible treatments.

Sarah died two days before we hosted the Society for the Improvement of Psychological Science (SIPS) meeting at COS this past July. At the meeting, we discussed the negative impacts of inaccessible outcomes, unavailable data, and doctored results in advancing science in general, and I couldn’t help but think of that impact in very concrete, personal terms. As a society, we didn’t do enough to transform science’s creativity, rigor, resources, and talent into the greatest possible benefit for people like Sarah. There are thousands of others like Sarah who are waiting for science to help them. And, there are going to be thousands more next year, and the next, and the next. Closed science is an impediment to progress.  

We can change this. To improve quality of life, advance more effective cures, and more efficient solutions, we must build the means and create the incentives for scientists to appreciate and embrace two “simple” objectives:

Show your work and share your work.

Really and truly, that’s it. If we can open the marketplace of ideas to the actual evidence and how that evidence was acquired, scientists are smart, they will figure out what is reliable and what needs to be studied next. Open science is the key to reducing waste, accelerating meaningful solutions to the biggest problems faced by our communities, states, nations, business and civic institutions, and to save the lives of millions of people around the world including people like my friend Sarah who desperately need science to survive.

We can improve decisions and quality of life by making the content of research and the process of producing that content transparent, accessible, and reusable.   

Researchers produce lots of research. What proportion of it is open?  Let’s compare 1999 Brian with 2017 Brian. In 1999, I might have produced 25 studies. At the time, none of the materials, none of the data, and none of the code was made publicly accessible to others. One reason for this lack of access was that the Internet was not what it is today--technology makes a difference. But technological change is not enough. Achieving our key outcome requires a strategy that integrates new technological opportunities with a deep understanding of the academic cultures and expectations that work against open research.

1999 Brian was a product of that culture. I didn’t write down my predictions in advance. I just hoped for a positive result that I could publish. If I got a negative result, I was motivated by the culture to toss it aside and try again. I had no incentive to record my predictions or save my “failed” trials.  I have a few papers of mine from that time period. I don’t really know how many studies I ran. I don’t know where the data or materials are -- or even if they still exist. I don’t remember why I picked some studies to write up for publication and not others. Did I leave out some inconvenient results? Did I p-hack? I did the work, and I don’t know that. No one knows. And, there is no way to find out. This reduces the credibility of the research that was published and is a waste of all the research that was lost.

In 2017, I will probably produce about the same total of 25 studies (now “I” is generous because others do most of the work). All 25 are on the OSF. My collaborators preregister the design and upload the materials in advance. After data collection, anonymized datasets and analytic code are added.  All of the content and key process elements of my research are preserved and accessible.  If I leave out studies from my reports, that is discoverable. If I p-hack my results, that is discoverable. If someone wants to reproduce my findings, they can. If someone wants to build on my work, they can. If I forget 16 years from now what work I did and how I did it, it doesn’t matter. It is there. Some parts of the workflow, e.g., email correspondence about ideas, is not yet integrated into OSF, but most of the research is preserved and accessible.

The immediate ROI of my 2017 research will exceed the ROI of my 1999 research because it is done more rigorously and more transparently. Moreover, the long-term ROI of my 2017 research will vastly exceed by 1999 research because it can be reused, reanalyzed, and extended by anyone and for purposes beyond what I envisioned. My 1999 research is gone and can provide no additional benefit.

Every part of the COS strategy moves researchers toward 100% open.  That might start as simple as sharing a poster from a conference on OSF, a poster that would otherwise disappear within weeks.  It might be as easy as sharing a paper on a preprint service, accelerating the date that others can become aware of the work by two years or more, and ensuring that it is openly available forever, even if it is never published in a journal.  These little steps are ways to inculcate a change of mindset -- a mindset that making our work available to others is a good thing.  With technology, incentives, and training, those initial steps can accumulate into a new ethic of openness that influences planning, execution, and reporting of all research.


Recent Blogs

The Content of Open Science

What Second Graders Can Teach Us About Open Science

What's Going on With Reproducibility?

Open Science and the Marketplace of Ideas

3 Things Societies Can Do to Promote Research Integrity

How to Manage and Share Your Open Data

Interview with Prereg Challenge Award Winner Dr. Allison Skinner

Next Steps for Promoting Transparency in Science

Public Goods Infrastructure for Preprints and Innovation in Scholarly Communication

A How-To Guide to Improving the Clarity and Continuity of Your Preregistration

Building a Central Service for Preprints

Three More Reasons to Take the Preregistration Challenge

The Center for Open Science is a Culture Change Technology Company

Preregistration: A Plan, Not a Prison

How can we improve diversity and inclusion in the open science movement?

OSF Fedora Integration, Aussie style!

Replicating a challenging study: it's all about sharing the details.

Some Examples of Publishing the Research That Actually Happened

How Preregistration Helped Improve Our Research: An Interview with Preregistration Challenge Awardees

Are reproducibility and open science starting to matter in tenure and promotion review?

The IRIS Replication Award and Collaboration in the Second Language Research Community

We Should Redefine Statistical Significance

Some Cool New OSF Features

How Open Source Research Tools Can Help Institutions Keep it Simple

OSF Add-ons Help You Maximize Research Data Storage and Accessibility

10 Tips for Making a Great Preregistration

Community-Driven Science: An Interview With EarthArXiv Founders Chris Jackson, Tom Narock and Bruce Caron

A Preregistration Coaching Network

Why are we working so hard to open up science? A personal story.

One Preregistration to Rule Them All?

Using the wiki just got better.

Transparent Definitions and Community Signals: Growth in the Open Science Community

We're Committed to GDPR. Here's How.

Preprints: The What, The Why, The How.

The Prereg Challenge Is Ending. What's Next?

We are Now Registering Preprint DOIs with Crossref

Using OSF in the Lab

Psychology's New Normal

How Open Commenting on Preprints Can Increase Scientific Transparency: An Interview With the Directors of PsyArxiv, SocArxiv, and Marxiv

The Landscape of Open Data Policies

Open Science is a Behavior.

Why pre-registration might be better for your career and well-being

Interview: Randy McCarthy discusses his experiences with publishing his first Registered Report

Towards minimal reporting standards for life scientists

Looking Back on the Prereg Challenge and Forward To More Credible Research

OSF: Origin, growth, and what’s next

A Critique of the Many Labs Projects

The Rise of Open Science in Psychology, A Preliminary Report

Strategy for Culture Change

New OSF Registries Enhancements Improve Efficiency and Quality of Registrations

Registered Reports and PhD’s – What? Why? How? An Interview with Chris Chambers

How to Collaborate with Industry Using Open Science

How to Build an Open Science Network in Your Community

Seven Reasons to Work Reproducibly

COS Collaborates with Case Medical Research to Support Free Video Publishing for Clinicians and Researchers

Advocating for Policy Improvements at Your Institution

Announcing a joint effort to improve research transparency: FAIRSharing and TOP Guidelines

This website relies on cookies to help provide a better user experience. By clicking Accept or continuing to use the site, you agree. For more information, see our Privacy Policy and information on cookie use.