Measuring the transparency and credibility of research is fundamental to our mission. By having measures of transparency and credibility we can learn about the current state of research practice, we can evaluate the impact of our interventions, we can track progress on culture change, and we can investigate whether adopting transparency behaviors is associated with increasing credibility of findings.
Transparency Frameworks
To measure transparency, we have to know what it means to do transparent research. The TOP Guidelines is a modular framework that is applicable across all types of research. With TOP, we can establish a common language such as "Did you share your data?" with recognition that "data" and how they are shared will vary widely across fields and methods. Other reporting standards such as those cataloged at the Equator Network are methodology or field specific definitions that complement TOP by articulating what it means to be transparent for a particular kind of research.
Transparency Surveys
Dozens of surveys have been conducted in different fields assessing researchers’ past, current, and planned transparency behaviors. The Open Scholarship Survey consolidates that work into a single, standardized, modular survey that can be administered to researchers in any field. The advantages of standardization include: more data and investment in validating and improving the specific measures, facilitating comparisons across fields and over time, and lowering the barrier to conducting surveys in new samples with a flexible, off-the-shelf solution.
Automated Extraction of Transparency Behavior from Papers
Many groups have conducted research projects that manually code a sample of papers from a field to assess current practices. These are useful but highly effortful. If machines can be trained to do the work, we will get much more data, more consistently, and much faster. There are at least three groups that have made meaningful progress creating scalable solutions: Ripeta, SciScore, and DataSeer. These groups are trying to make it possible, accurate, and easy to assess many papers for whether the authors shared data, used reporting standards, identified their conflicts of interest, and other transparency relevant actions.
Creating a culture of transparency and openness in research is central to COS’s mission. Effective measurement is essential to advancing that mission. The proliferation of tools and standardization of formats for measuring open practices will inform strategy and impact assessment for all reform-minded organizations and accelerate our collective impact.
