Supporting Federal Agency Plans as They Move Into a New Era in Open Science

In August 2022, the White House Office of Science and Technology Policy (OSTP) released a memorandum entitled “Ensuring Free, Immediate, and Equitable Access to Federally Funded Research.” Known as the “Nelson memo,” it established new guidance for improving public access to scholarly publications and data resulting from federally supported research. This expanded on prior OSTP guidance by removing or reducing embargoes for access to published articles, by including data and related research outputs as items that should be accessible by the public, and by asserting that all federal agencies that conduct or fund scientific research are to be covered by this new policy.

Importantly, this policy directive included several key details that will improve the transparency, equity, and rigor of scientific outputs. First, it defined “scientific data” to include the “recorded factual material commonly accepted in the scientific community as of sufficient quality to validate and replicate research findings” (emphasis added). While it excludes some items, such as laboratory notebooks and physical materials, this definition does include all digital items needed to replicate research findings. Such materials can consist of analytical code, detailed protocols, and other digital materials, such as survey instruments or content included in the study procedures.

Secondly, it specified that agency plans should cover “scientific data that are not associated with peer-reviewed scholarly publications.” This is game-changing, as findings that remain unpublished represent a trove of information. Publication bias remains a persistent problem that distorts our understanding of how the universe works.¹ Unreported findings represent real, credible knowledge that we need to have available. It is critical to know what has been tested and deemed “unpublishable” because no trends were detected. Such information allows others to try different approaches to known problems. And such information can be weighed against contradictory findings that arise through statistical noise and other false-positive findings in order to give us a more complete picture.

Finally, while the Nelson memo represents a dramatic step forward for open science and trust in the scientific community, it does not go quite far enough in specifying that the full lifecycle of scientific practice should be preserved and shared. The Transparency and Openness Promotion Guidelines (TOP) cover eight practices that should be implemented or preserved in order to increase rigor across the process of science. In addition to data, it articulates practices for sharing analytic code, sharing research materials, increasing clarity of study designs through reporting guidelines, citing data, preregistering studies and their analysis plans to help open the file drawer, and conducting replication studies to increase confidence in published findings.

Federal agencies have until 2025 to implement their updated policies and have been issuing draft policies in recent months. This window represents an important opportunity for the Center for Open Science (COS) to advocate for the best possible policies in order to enable and incentivize open science practices. To date, COS has seen and conducted preliminary evaluations of responses from 9 federal agencies, of at least nineteen federal agencies and 60 divisions or internal offices that are known to conduct or support research.

While the 2022 Nelson memo provides a strong foundation for open science practices, a preliminary review of agency responses reveals considerable variation between agencies on how public access to paper, data, and other research outputs will be handled. In order to achieve the best possible outcome, COS advocates that:  

• Data Management and Sharing Plans (DMSPs) should be publicly available and persistently identified in order to increase discoverability of research outputs.
• DMSPs should be peer-reviewed as part of the same process that other aspects of a proposed research project’s intellectual merit are evaluated, in order to directly incentivize high standards of planned data sharing practices.
• Enforcement of the proposed DMSPs should be consistent across agencies by making future grant support contingent upon prior evidence of data sharing.
• DMSPs should cover all of the materials needed to replicate and validate findings, and these elements should be clearly articulated in guidelines.

As agencies continue to draft their policies, COS will use language and examples from those with stronger policies to encourage adoption of those policies among agencies that have comparatively weaker responses. Seeing example language from other agencies provides an avenue to “raise all boats” if each agency is able to see how others implement strong policies. COS positions that have no adoption among agencies is a potential weakness, as it can reinforce the perception that such a move is too risky to implement.

Next, COS will assess each agency’s existing policy and response to the Nelson memo across the policies and practices covered by the TOP Guidelines. These standards, supported by thousands for journals, publishers, and academic societies, provide a consistent framework for policy implementation. The practices increase rigor and trust in scientific research.

Finally, COS will be assessing agency plans for digital infrastructure to support the expected increase in data and research output sharing. COS believes that open-source digital infrastructure needs to play a central role in enabling these practices and is monitoring agency plans for their data-sharing needs. 

¹ Franco, A., N. Malhotra, and G. Simonovits. 2014. “Publication Bias in the Social Sciences: Unlocking the File Drawer.” Science 345 (6203): 1502–5. https://doi.org/10/f6gnd3.