Ensuring trust in scientific research is an ongoing endeavor crucial for maintaining public confidence in the scientific process. At the Center for Open Science (COS), we're pleased to introduce our latest initiative: SMART (Scaling Machine Assessments of Research Trustworthiness). Building on the significant progress made during the DARPA-funded SCORE program, SMART aims to advance how we evaluate research claims using novel machine learning algorithm and human deliberation approaches.
Understanding the Need for Automated Research Evaluation
The SCORE program, funded by DARPA from 2019 to 2022, laid the groundwork for scalable algorithms in assessing research credibility. It demonstrated the potential for algorithms to predict the confidence of research claims, paving the way for practical applications. Leveraging this progress, COS, in collaboration with researchers from the University of Melbourne and Pennsylvania State University, has launched the SMART project.
What is SMART?
The SMART project seeks to advance automated confidence evaluation of research claims. Through a grant from the Robert Wood Johnson Foundation (RWJF), we aim to improve the algorithms developed in the SCORE program. Our goal is to streamline the process of assessing research credibility, making it more accessible and scalable.
How Can You Participate?
We invite researchers to submit their papers for evaluation as part of the SMART project. By participating, you'll receive two types of assessments of your research: one produced by algorithms and another by human experts. You will then have the opportunity to provide feedback on each assessment.
Eligibility & Process
We will be accepting papers from now until August 2024.
To participate, your paper must meet the following criteria:
- Report quantitative empirical research based on human, social, or economic data
- Have been recently written, uploaded, submitted, or published
- Include a positive empirical result supporting a focal claim made by the paper
We're particularly interested in preprints from psychology, education, sociology, marketing, and criminology research.We also welcome papers that are currently under review or revision, papers that have been recently accepted for publication, as well as papers that haven’t been submitted to a preprint server or journal.. Unsure if your paper qualifies? Complete the informed consent form, and we'll review it for eligibility.
You must be 18 years or older to participate.
Join Us in Advancing Research Evaluation
By participating in the SMART project, you're contributing to the advancement of research evaluation methods. For more information, check out our webpage. Have additional questions? Reach out to us at smart_prototyping@cos.io.
