Help Contribute to Assessments of Research Credibility

SMART: Scaling Machine Assessments of Research Trustworthiness

The Center for Open Science (COS), along with its collaborators, is building on the work completed during the DARPA-funded SCORE program, which demonstrated the potential of using algorithms to efficiently evaluate research claims at scale. The SCORE program supplements existing research evaluation methods, including human judgment, evidence aggregation, and systematic replication.

Through a grant from the Robert Wood Johnson Foundation (RWJF), COS, in partnership with researchers at the University of Melbourne and Pennsylvania State University, have begun the SMART project, which seeks to advance the development of automated confidence evaluation of research claims (press release here). SMART will extend the research work initiated by the SCORE program through conducting user research and generating additional data to improve the algorithm and human assessment approaches developed during the program.

The first phase of SMART will engage researcher authors to voluntarily submit their papers for evaluation. Papers will undergo two types of assessment by our partners: (1) human assessment and (2) AI assessment. These assessments will be shared with the authors of the papers who will then have the opportunity to provide feedback on the assessments.

We would like to engage the research community as a partner to interrogate the potential for automated research evaluation. Not only will authors who submit a paper receive two separate assessments of the credibility of their research, they will also have the opportunity to provide feedback on the assessment scores, as well as the approach and process. Importantly, authors could provide supporting or contradicting evidence that informs the quality and accuracy of the scores, or they might indicate how their paper was misevaluated or misunderstood. Overall, we hope to gather insight into how researchers perceive automated scoring of research credibility and what might make this most useful to the research community.

If you are interested, consider submitting your paper for inclusion in the study! Please read the information in the ‘Eligibility & Process’ section before enrolling.

How to Enroll

We will be accepting papers from now until August 2024.

If you would like to participate, please fill out the informed consent form. Be sure to attach a recent paper that meets our eligibility criteria before you submit the form.

Should you consent to your paper’s inclusion in this project, we encourage you to notify any co-authors on the paper about this research.

For additional information about this project and what participation may involve, please see our FAQs below. If you have any other questions, please contact us at smart_prototyping@cos.io.

This research is funded by the Robert Wood Johnson Foundation (grant # 79762) and has been approved by the University of Virginia Institutional Review Board for the Social and Behavioral Sciences (protocol # 6226).

Eligibility

To participate in the study you will need to submit an eligible paper for which you are an author. You must be 18 years or older to participate.

The paper should:

• Report quantitative empirical research based on human, social, or economic data
• Have been recently written, uploaded, submitted, or published (ideally, within the last 12 months)
• Include a positive empirical result as support for a focal claim made by the paper (that is, a result that’s supportive of the authors’ claim)

At this time, we are primarily interested in preprints from psychology, education, sociology, marketing, and criminology research. We are also accepting papers that are currently under review or revision, papers that have been recently accepted for publication, as well as papers that haven’t been submitted to a preprint server or journal.

If you are unsure whether your paper would be eligible but would like to participate, please complete the informed consent form and include a link to your paper and we will let you know whether your paper is eligible.

Process

We will be accepting papers from now until August 2024.

Once you provide consent for your paper to be used in this research, we will first confirm that it meets our eligibility criteria. We will then identify a key claim of your paper that will be the focus of the assessment and send it to you for the opportunity to review. We will reach out to you via the email you provide with the selected claim, or to let you know that your paper is not eligible, within one week from when you provide consent. From there, you will have one week to respond with any corrections or suggestions to the selected claim. 

After that time we will incorporate the paper and claim into our dataset to be sent to our partners who will be performing the AI and human assessments. Papers will be sent to our partners in batches (i.e., every couple of months), so your paper will be assessed alongside a number of other papers at a certain period in time, then the batch of assessments will be delivered back to us. 

Once the assessments are complete, we will share your assessment reports with you via email. You can expect to receive your assessment reports within approximately 4-8 weeks of submitting your consent. At that time, we will also ask you to provide feedback on the assessments via a couple of brief surveys (one survey per report). You will have approximately 3 weeks to complete the surveys (and we will send out a few reminders).

A small subset of participants who provide feedback on the assessments will later be invited to participate in user research interviews about their experience with the assessments including potential use cases, interpretation challenges, validity and reliability, and perceived risks. If you are selected to take part in a user research interview, we will reach out to let you know within approximately 2-6 weeks of receiving your survey responses. We will share additional details about the process at that time so you can decide whether you would like to participate.

This research is funded by the Robert Wood Johnson Foundation (grant # 79762) and has been approved by the University of Virginia Institutional Review Board for the Social and Behavioral Sciences (protocol # 6226).

Background

The credibility of research is essential for maintaining public trust in science. Assessing the trustworthiness of research claims is a central, continuous, and laborious part of the scientific process. Assessment strategies range from expert judgment to aggregating existing evidence to systematic replication efforts, requiring substantial time and effort. What if we could create automated methods that achieve similar accuracy in a few seconds? If that were possible, readers, researchers, reviewers, funders, and policymakers could use them to direct attention toward high-confidence claims and improve the allocation of resources to examine claims that are important but uncertain. Automated methods could enable a more accessible and scalable approach to updating confidence in claims as new evidence becomes available. Improving rapid confidence assessment will foster public trust in research and increase access and equity to the assessment of research evidence.

There is substantial evidence that published findings vary in credibility and replicability. The SCORE program, funded by DARPA from 2019 to 2022, built on initial evidence that replicability can be predicted by humans and machines. SCORE extended these efforts with an unprecedented scale and disciplinary scope by including evaluation of claims, a specific assertion reported as a finding in a research paper, published from 2009 to 2018 from 62 journals across the social-behavioral sciences -- such as economics, education, and health research -- with four novel artificial intelligence strategies, and validation evidence from human judgements and empirical replication, robustness, and reproducibility tests contributed by >1,000 researchers. We observed evidence establishing the viability of scalable algorithms for confidence assessment and set the stage to translate these tools to application. For example, in the most recent performance assessment, the algorithms were able to predict the human experts’ evaluation of claims’ confidence scores (i.e., a numerical estimate on a 0-1 interval, indicating the probability of the claim ‘successfully’ replicating) to at least a moderate degree, with each of the three algorithm teams achieving a correlation with human judgments between 0.33-0.40.

We will invite researchers to voluntarily submit their papers for scoring by the algorithms. We will also contract with the RepliCATS team from the University of Melbourne to conduct the structured elicitation process that they used in SCORE to obtain human expert ratings of the same papers. Researchers that submit a paper will receive assessments from both algorithms and humans, and then provide their feedback on the workflow, interface, and assessment scores. Of particular interest will be authors providing supporting or contradicting evidence that informs the quality and accuracy of the scores, and offering them a forum to express how their paper might have been misevaluated or misunderstood. A subset of researchers who submitted their papers, and representing a range of fields and perceptions about the quality and accuracy of the scores, will be invited to participate in user research interviews about their experience with the prototype including potential use cases, interpretation challenges, validity and reliability, and perceived risks. The prototype will be a naturalistic path for generating training and test data complementing the more systematic methods in SCORE.

This research is funded by the Robert Wood Johnson Foundation (grant # 79762) and has been approved by the University of Virginia Institutional Review Board for the Social and Behavioral Sciences (protocol # 6226).