Reproducibility Project: Cancer Biology

Project Overview

The Reproducibility Project: Cancer Biology was an 8-year effort to replicate experiments from high-impact cancer biology papers published between 2010 and 2012. The project was a collaboration between the Center of Open Science and Science Exchange with all papers published as part of this project available in a collection at eLife and all replication data, code, and digital materials for the project available in a collection on OSF.

When preparing replications of 193 experiments from 53 papers there were a number of challenges.

experiments with open data

70%

of experiments required asking for key reagents

69%

of experiments needing a key reagent original authors were willing to share

of protocols completely described

32%

of experiments the original authors were not helpful (or unresponsive)

41%

of experiments the original authors were very helpful

Fully designed protocols were submitted to eLife for peer review before conducting the replications - a publishing format called Registered Reports - to ensure that the proposed experiments were of the appropriate rigor and quality. Accepted replication protocols received a commitment in advance to publish the findings regardless of outcome.

Additional challenges were encountered for the experiments that were conducted.

67%

required modifications to complete

41%

of modifications completely implemented

Ultimately, 50 replication experiments from 23 of the original papers were completed, generating data about the replicability of a total of 158 effects. There are many ways to evaluate and characterize replication outcomes, some simplified summaries of the findings include:

Replication effect sizes were 85% smaller on average than the original findings
46% of effects replicated successfully on more criteria than they failed
Original positive results were half as likely to replicate successfully (40%) than original null results (80%)

Collectively, this evidence suggests opportunities to improve the transparency, sharing, and rigor of preclinical research to advance the pace of discovery.