Open Practice Badges enhance openness, which is a core value of scientific practice.

There is no central authority determining the validity of scientific claims. Accumulation of scientific knowledge proceeds via open communication with the community. Sharing evidence for scientific claims facilitates critique, extension, and application. Despite the importance of open communication for scientific progress, present norms do not provide strong incentives for individual researchers to share data, materials, or their research process. Journals can provide such incentives by acknowledging open practices with badges in publications.


There are circumstances, however, in which open practices are not possible or advisable. For example, sharing some human participant data could violate confidentiality. When badge criteria cannot be met, a description in place of the badge can articulate why. Badges do not define good practice; badges certify that a particular practice was followed. Disclosure makes explicit the conditions under which the ethic of openness is superseded by other ethical concerns. Here, we introduce three badges to acknowledge Open Data, Open Materials, and Preregistration.


Badges

Open Data

The Open Data badge is earned for making publicly available the digitally-shareable data necessary to reproduce the reported results. 

Criteria

  1. Digitally-shareable data are publicly available on an open-access repository. The data must have a persistent identifier and be provided in a format that is time-stamped, immutable, and permanent (e.g., university repository, a registration on the Open Science Framework, or an independent repository at www.re3data.org).
  2. A data dictionary (for example, a codebook or metadata describing the data) is included with sufficient description for an independent researcher to reproduce the reported analyses and results. Data from the same project that are not needed to reproduce the reported results can be kept private without losing eligibility for the Open Data Badge.
  3. An open license allowing others to copy, distribute, and make use of the data while allowing the licensor to retain credit and copyright as applicable. Creative Commons has defined several licenses for this purpose, which are described at www.creativecommons.org/licenses. CC0 or CC-BY is strongly recommended.

State of Data Notations
Specification of open data is complicated by the fact that raw, collected data may be processed prior to conducting the reported analyses leading to derived, constructed data based on the raw data. For example, the raw data might be 10 survey responses and the constructed data might be the mean score of those 10 responses. Open data badges assume that at least raw data are available. If only derived, constructed data are available (i.e. the data used to conduct the reported analyses), it is denoted with the badge. Sharing derived datasets must include descriptions of how the data were constructed or, even better, provide the code used to construct the data.

Specification of open data is also complicated by the fact that the data underlying the reported analyses may be a subset of the data collected for the study. Open data badges assume that all collected data are made available. If only the subset of data used to conduct the reported analyses is available, it is denoted with the badge. Sharing reported subsets must include descriptions of how the data were reduced from the complete dataset or, even better, provide the code used to reduce the dataset.

Specification of open data explicitly excludes data that compromises confidentiality or anonymity of human participants. If access to identifying data is necessary to reproduce the reported analyses, then the report is not eligible for an open data badge.


Open Materials

The Open Materials badge is earned by making publicly available the components of the research methodology needed to reproduce the reported procedure and analysis.


Criteria

  1. Digitally-shareable materials are publicly available on an open-access repository. The materials must have a persistent identifier and be provided in a format that is time-stamped, immutable, and permanent (e.g., university repository, a registration on the Open Science Framework, or an independent repository at www.re3data.org).
  2. Infrastructure, equipment, biological materials, or other components that cannot be shared digitally are described in sufficient detail for an independent researcher to understand how to reproduce the procedure.
  3. Sufficient explanation for an independent researcher to understand how the materials relate to the reported methodology.


Preregistration

The Preregistered/Preregistered+Analysis Plan badges are earned for preregistering research.

Preregistered

The Preregistered badge is earned for having a preregistered design. A preregistered design includes: (1) Description of the research design and study materials including planned sample size, (2) Description of motivating research question or hypothesis, (3) Description of the outcome variable(s), and (4) Description of the predictor variables including controls, covariates, independent variables (conditions). When possible, the study materials themselves are included in the preregistration.

Criteria for earning the preregistered badge on a report of research are:

  1. A public date-time stamped registration is in an institutional registration system (e.g., ClinicalTrials.govOpen Science FrameworkAEA RegistryEGAP).
  2. Registration pre-dates the intervention.
  3. Registered design and analysis plan corresponds directly to reported design and analysis.
  4. Full disclosure of results in accordance with registered plan.

Badge eligibility does not restrict authors from reporting results of additional analyses. Results from preregistered analyses must be distinguished explicitly from additional results in the report. Notations may be added to badges. Notations qualify badge meaning: TC, or Transparent Changes, means that the design was altered but the changes and rationale for changes are provided. DE, or Data Exist, means that (2) is replaced with “registration postdates realization of the outcomes, but the authors have yet to inspect or analyze the outcomes.

Preregistered+Analysis Plan

The Preregistered+Analysis Plan badge is earned for having a preregistered research design (described above) and an analysis plan for the research and reporting results according to that plan. An analysis plan includes specification of the variables and the analyses that will be conducted. Guidance on construction of an analysis plan is below.

Criteria for earning the preregistered+analysis plan badge on a report of research are:

  1. A public date-time stamped registration is in an institutional registration system (e.g., ClinicalTrials.gov, Open Science Framework, AEA registry, EGAP).
  2. Registration pre-dates the intervention.
  3. Registered design and analysis plan corresponds directly to reported design and analysis.
  4. Full disclosure of results in accordance with the registered plan.

Notations may be added to badges. Notations qualify badge meaning: TC, or Transparent Changes, means that the design or analysis plan was altered but the changes are described and a rationale for the changes is provided. Where possible, analyses following the original specification should also be provided. DE, or Data Exist, means that (2) is replaced with “registration postdates realization of the outcomes, but the authors have yet to inspect or analyze the outcomes.”

Guidance on Analysis Plans

Procedures

  • What is your planned sample size?
  • If applicable, how many individual units and how many clusters?
  • If you are conducting a randomized control trial or experimental study, how will you randomize?
  • At what level will you randomize (individual or cluster level)?

Exclusions

  • What conditions will lead to data being excluded?

Variable Construction

  • If your predictor variable(s) are not from a single question or measure, how will they be constructed?
  • If your outcome variable(s) are not from a single question or measure, how will they be constructed?

Tests or models

  • What is the quantity you intend to estimate?
  • What is the unit of analysis (if applicable)?
  • What statistical model(s) will you use to test your hypothesis? Please include the type of model (e.g. ANOVA, regression, SEM, etc) as well as the specification of the model (e.g. what variables will be included and how they will be included).
  • If you are comparing multiple conditions or testing multiple outcomes and/or hypotheses, how will you account for this?

In addition, the researcher will be invited to pre-specify procedures that will be used in the event of foreseeable problems (e.g., attrition, noncompliance, failure to enroll the planned number of subjects, etc.) that routinely afflict certain kinds of studies.


Journals or organizations self-select to become badge awarding entities. Badges are awarded to published reports of empirical research and can be part of the peer review process for publication or conducted post-publication. Currently, organizations awarding badges are doing so during the peer review process.

Any organization can issue badges as long as the process and practices are transparent. Reputation of certifying organizations will depend on the quality and reliability of their certification process. Because of this, when badges are mentioned or displayed, the awarding entity must be indicated or obvious.

There are two ways for certifying organizations to award badges for individual studies - disclosure or peer review.

Disclosure

Disclosure requires authors to provide public statements affirming achievement of badge criteria. The certifying organization evaluates the disclosure before issuing the badge, but does not do more than a cursory evaluation of the data, materials, or registration. Such a review might include: confirming that the provided link leads to the data, materials, or registration on a public, open access repository, and that the linked materials are related to the report. Authors follow criteria for each badge and complete disclosure items that will be made publicly available. Authors are accountable to the community for disclosure accuracy. Disclosure items:

Open Data: Authors complete two disclosure items for each Open Data badge application:

  1. Provide the URL, DOI, or other permanent path for accessing the data in a public, open access repository.
  2. Is there sufficient information for an independent researcher to reproduce the reported results? If no, explain.

Open Materials: Authors complete two disclosure items for each Open Materials badge application:

  1. Provide the URL, DOI, or other permanent path for accessing the materials in a public, open access repository.
  2. Is there sufficient information for an independent researcher to reproduce the reported methodology? If no, explain.

Preregistered/Preregistered+Analysis Plan: Authors complete five disclosure items for each Preregistered/Preregistered+Analysis badge application:

  1. Provide the URL, DOI, or other permanent path to the registration (and, if applicable, the analysis plan) in a public, open access repository.
  2. Was the plan registered prior to examination of the data or observing the outcomes? If no, explain.
  3. Were there additional registrations for the study other than the one reported? If yes, provide links and explain.
  4. For Preregistered+Analysis plan badge: were there any changes to the preregistered analysis plan for the primary confirmatory analysis? If yes, explain.
  5. For Preregistered+Analysis plan badge: are all of the analyses described in the registered plan reported in the article? If no, explain.

Preregistration is invalidated if (1) is not provided, or (3) is answered “yes” without strong justification. If (5) is answered “no” without strong justification, manuscript is ineligible for preregistered+analysis plan badge. DE notation is added to badge if (2) is “no.” TC notation is added if (4) is “yes” with strong justification for changes.

Peer Review

Peer review involves independent evaluation of the open data, open materials, or preregistration to verify that badge criteria are met. Authors follow criteria for each badge and complete disclosure about items that will be made publicly available. The certifying organization conducts a formal review of the disclosure and data, materials or registration to verify whether badge criteria are met. Peer review provides independent certification, but is more resource intensive. Badges awarded following peer review receive an additional “PR” notation. In most cases, badge review would occur following acceptance of the report for publication. There are at least four procedures for the review process of badge applications:

  • Reviewers of the report can also review associated data, materials, and preregistration.
  • Additional reviewer(s) can be recruited specifically for badge review.
  • An organization staff member could provide badge review.
  • An independent organization could provide badge review as a service for the certifying organization.

The disclosure and peer review awarding processes described here are not limited to any substantive domain. Additional disclosure items or review practices may be added by the certifying organization based on disciplinary needs. For example, disciplines may have different requirements for evaluating whether the “data necessary to reproduce the reported results” criterion is met for the Open Data badge. Certifying organizations should make publicly available additional criteria and additional specific requirements for meeting all criteria.


There are two common use cases for awarding badges: upon publication and post publication. Journals are likely to employ upon publication review. Emerging publishing platforms and review services might pursue a post publication review process. The following are suggestions for incorporating badge review into the workflow. Certifying organizations may adapt these procedures based on their own needs and idiosyncratic workflows.

Upon publication badge review

Authors may apply for one or more badges at initial submission of a manuscript for review or when submitting the final version following acceptance. Blended models are possible, such as confirming intent to apply for badges during initial submission and then making materials available, providing disclosure, and conducting badge review after acceptance. Illustrative steps:

  1. During manuscript submission, authors review badge criteria to determine whether they wish to apply.
  2. Authors identify whether they will apply for one or more badges for each study appearing in the manuscript.
  3. For each badge selected, authors complete the disclosure items and are informed whether the journal uses a disclosure or peer review for verification. See a disclosure statement template here.
  4. If badge criteria cannot be met, authors have opportunity to provide text to appear in place of the badge, such as “For protection of human participant privacy, the University’s Data Access Committee must review all data access requests. All reasonable data requests from qualified researchers are granted. Contact your university's data access committee to initiate data request process.”
  5. If the article, and badges, are accepted, then the disclosures and badges are printed in the journal article - see Incorporating Badge Visualization into Publications.

Post publication badge review

It is possible to award badges to already published articles. There are several approaches. Feasibility will vary across certifying organizations:

  • Published Addendum: Most journals have procedures for submitting post-publication rectifications or errata. A “Badge Addendum” category could be created. In its simplest version, this addendum would include the disclosure items for each badge. Peer review could be incorporated for verification.
  • Online Addendum: Many infrastructures allow for moderated or reviewed peer commentaries on published articles. The addendum procedure could be implemented as a special commentary or article level metric at the publisher’s website or at an independent outlet.
  • Online Community Review: Authors’ disclosures could be made available to invited or volunteer community members for verification. Badges could be awarded (and revoked) based on community review. Unless the process is actively managed by editors, community reviewers should be identified publicly for accountability. This model fits easily with post-publication review of article content. Reviewers comment on the availability and usability of public materials, data, and registrations.
  • Accreditation Body: An accreditation body (or bodies) could form and provide certification services. Authors would be awarded badges by the accreditation body by requesting to be reviewed. The accreditation body could be endorsed by journals, effectively outsourcing the reviewing responsibility. Or, the accreditation body could administer badges by linking to the citation independently of the journal. Such a body does not yet exist.

Awarded badges become linked to an article. Commonly, the publishing journal awards the badge and then indicates the badges in the article itself. Journals and other awarding entities have idiosyncratic solutions for how best to incorporate the badges into their publishing formats, though they should be machine discoverable and readable. Here, to assist journal editors and publishers, we provide some illustrative examples of how badges could be integrated into articles. In all cases, badge implementation includes confirmation of which badges were earned, and hyperlinks to the open materials, open data, or registration. Click on each image to retrieve a full page PDF of the mockup.

Variation One:

Large, color badges are presented with accompanying text and hyperlinks as a Figure:

badge-variation-1

Variation Two:

Small, gray-scale badges are presented in the article header and accompanying text and hyperlinks appear in a footer at the bottom of the first page:

badge-variation-2


Variation Three:

Small, gray-scale badges are presented in the article header and accompanying text and hyperlinks appear in a “Resources” subsection at the end of the article:

badge-variation-3

Variation Four:

Small, grey badges are presented at the beginning of the study and accompanying text as a subsection prior to text of the experiment:

badge-variation-4

Adoptions

The following journals offer one or more Open Practice Badges to authors:

The following journals will implement badges in the future:

Journals may use the badges without asking for permission or being named as having adopted the badges. To be listed as a journal supporting open practice badges, email badges@cos.io


Endorsements

The following organizations support the use of the Open Practice Badges described in this document. Some may offer services related to the implementation, visualization, or operation of these badges. Organizations may email badges@cos.io to join this list. 

Development of norms or specific expectations about how to fulfill badge criteria

Articulation of criteria for earning badges is necessarily general in this documentation. What qualifies as fulfilling badge criteria may vary across disciplines. Within some disciplines, there exists tacit understanding of what would be sufficient to meet the badge requirements. In many cases, such qualifications may be implicitly known but otherwise unspecified. With use of the badges, instances of adherence to these tacit norms will become explicit and salient. Documentation of those norms will improve the efficiency and consistency in evaluation of articles for badges, and enhance the shared understanding of badge meaning.

Development of technologies that enable open practices can increase expectations for earning badges

As open practices become more common, technologies will emerge to support those practices. These technologies will improve the ease and quality of open practices. For example, development of virtual machines will facilitate re-execution of analysis scripts on original data using the original analytic software. Until such solutions exist, sharing data and analysis scripts will be useful, but time and resource limited because users need to have accessibility to the analysis programs that originally executed the scripts. The practical constraints on open practices reasonably limit the expectations for earning badges to acknowledge open practices. As the practical constraints disappear, expectations for earning badges may increase as a means of promoting the best-available practices.

Baked Badges

At present, badges are issued as flat images in journals. "Baked" Badges are images containing metadata in the header, making them digitally verifiable. COS is developing a badge bakery, hosted through the OSF, to "bake" the metadata about badge issuer, recipient, date, and evidence (URL) into the image.

Reproducibility Badges

A Reproducible badge could be a fourth badge to acknowledge open practices. Elements of the present badges are relevant, but reproducibility is a broader concept. Among the challenges to resolve for a Reproducible badge are: variation in the definition of reproducible across scientific disciplines, defining a general criterion for achieving status as “reproducible”, and managing the evolution of the understanding of reproducible.

Findings and datasets are the products of a context-dependent measurement and analysis. If an original finding is entered into the scientific record, all subsequent realisations will be literal replications of the original, but only if the measurement context (e.g. sampling procedures, analysis, and rules of inference) is realised in exactly the same way. The newly obtained outcome may replicate the original outcome, or it could be a discrepancy. If an existing dataset is used, the realisation typically concerns the original data analysis strategy and rules of inference. In either case, the expected discrepancy can usually be calculated a-priori.

Readers interested in developing the criteria for a reproducibility badge are invited to propose a set of criteria that address the following issues. Proposals should be sent to badges@cos.io and will be evaluated twice yearly by the Open Science Collaboration committee responsible for maintaining badge criteria. When and if there is consensus that the issues listed below have been sufficiently addressed to proceed, the committee will recommend to the Center for Open Science that a Reproducible badge be added to the current set of 3 badges.

Issue #1: Reproduce, Replication, and Replicated

Reproducing the primary findings using original data or replicating a study with a novel data set are distinct processes. Furthermore, there is variation in acceptance of internal replications (e.g. within a single research lab) versus external replications.

See Clemens (2015) for an additional discussion of these proposed definitions across several disciplines.

Various proposals have recommended that badges be awarded to different key elements of a replication project. For example, a “reproduced” badge could be awarded to a study upon publication using original data and analytical methods to reproduce primary study findings to within a sufficiently precise, pre-specified goal. Such a badge could convey three pieces of information: 1) What: analysis / measurement context; 2) Who: independent / internal; and 3) Result: replica / discrepancy.

Furthermore, a “replication” badge could be awarded to the study that attempts to replicate previously-published work.

Finally, a “replicated” badge could be appended to a previously published study whose findings were replicated in a novel study.

Issue #2: Defining “Successful” Reproductions

In all cases, evaluating whether or not the replication was “successful” will require an assessment of the original and replicated findings. The standards for using an original study’s data and analytical code are likely to be different than those that require additional data to be collected. When the Open Science Collaboration evaluated this question, they relied on 5 different measures to assess replicability (OSC 2015, Science). Those measures were significance and p-values, comparison of effect sizes, combining the original and newly-collected data into a meta-analysis, and a purely subjective measure by the replicating researcher.

Furthermore, "success" is indeed a misplaced qualification, because a failure to replicate always implies relative advancement of scientific knowledge whereas replication success "merely" confirms what we already knew before the attempt. So language to describe the replication must convey the nuanced interpretation of the results and reward the advancement of knowledge, which was the result of both the original and replicating researchers.

Issue #3: Effect of “failed” replications, or unreproducible research

If one’s work does not replicate, in accordance with the agreed-upon definition resulting from addressing Issue #2, there are serious possible consequences. Though an idealistic interpretation sees the possibilities for discovering previously unknown moderating variables, many take a failure to replicate as a source of embarrassment or even an indication of fraud. Unlike badges that reward sharing data, sharing materials, or preregistering one’s work, the failure to receive a Replicated/Reproduced/Reproducible badge is more likely to be perceived as a punishment. Finally, “failed” replications would have to be surfaced in a consistent manner, so as to avoid contributing to a new “file drawer” effect. Perhaps this could be done with a “replication attempt exists” badge or with a widely used registry of replication attempts.

Issue #4: Identification of need

The current set of three badges reward practices that aid subsequent researchers in an attempt to replicate findings. Without original data or research materials, replications are very challenging. Preregistration reduces the file drawer of unpublished studies and, when it includes an analysis plan, clarifies the distinction between a-priori hypothesis tests (aka confirmatory tests) and post-hoc exploratory analysis. Before adding a fourth badge and addressing the above issues, it must be clear that a specific need is not currently being met. The Reproducible badge could be a clear indicator that some additional verification took place by a third party or some additional resources or actions were provided by the authors, but clarifying the “value added” by this badge is required.