Assessing the credibility of research claims is a central and continuous part of the scientific process. However, current assessment strategies often require substantial time and effort. In an effort to accelerate research progress, the Center for Open Science (COS) partnered with the Defense Advanced Research Projects Agency's (DARPA) program Systematizing Confidence in Open Research and Evidence (SCORE) in 2019 to work towards developing and deploying automated tools to provide rapid, scalable, and accurate confidence scores for research claims.
Since then, COS has completed extraction of scientific claims from a stratified sample of social-behavioral science papers. In total 7,066 claims were extracted manually, enabling confidence scores to be assigned by human forecasters and algorithms. Concurrently, COS worked with hundreds of researchers to conduct replications and reproductions on a subset of these extracted claims. The team leveraged the Open Science Framework (OSF) for this large-scale collaboration so that materials from the replication and reproduction efforts can be made openly available.
The project, now in its final phase, is beginning the transition from working with the research community at large to conduct reproduction, robustness, and replication studies to preparing the project for public release and organizing the data and materials for the wider research community’s use.
Over the next six months, COS will be working with close collaborators to compile and enhance the project’s data; continue coding core project outcomes; and draft reports on what we learned about replicability, reproducibility, and robustness, as well as the potential for experts, forecasters, and algorithms to contribute scalable tools for research assessment. Some exciting expansions the SCORE project has made over previous large-scale replication projects (e.g., Reproducibility Project: Psychology and Reproducibility Project: Cancer Biology) is to engage a broad range of disciplines across the social-behavioral sciences. The breadth and depth of SCORE data will be a major contribution to the metascientific field. Finally, the outcomes of this project will lay the groundwork for advancements in algorithmic scoring and will spur future research for assessing research credibility.
