Inside AI Policy

February 8, 2025

AI Daily News

Johns Hopkins security center proposes controls on data, models posing pandemic risks

By Charlie Mitchell / November 25, 2024

The Johns Hopkins Center for Health Security focuses on governance practices for biological data as well as biosecurity and biosafety issues related to advanced artificial intelligence models, in comments on an initiative for enlisting the Department of Energy’s vast research resources in the development of “frontier AI.”

“CHS’s responses below address data governance practices and risks, balancing national security concerns with the open sourcing of models, and considerations to inform DOE's ongoing AI red-teaming and safety tests for CBRN risks, particularly related to AI models that have biosecurity and biosafety implications,” Johns Hopkins’ CHS commented.

DOE launched the Frontiers in AI for Science, Security and Technology, or FASST, initiative in July to “leverage” the department’s national laboratories “to provide a national AI capability for the public interest.” The Senate Energy and Natural Resources Committee passed bipartisan legislation last week that would enshrine the initiative in law.

DOE on Sept. 12 posted a request for public information on implementing FASST. The comment period closed Nov. 11.

The consumer rights group Public Knowledge in its comments acknowledged there might be a need to limit access to certain AI models, but emphasized that should not translate into blanket restrictions on AI model weights or open-source systems.

The Software Information Industry Association said the DOE research initiative can propel development of AI tools to help meet national energy objectives and generate benefits across government services.

Data governance

In addressing data governance, the Johns Hopkins security center said, “Most biological data should be shared openly to benefit the advancement of biology and life science research broadly, as has been the general practice of this scientific community. We welcome efforts to generate large amounts of high-quality data for training biological AI models (BAIMs) and anticipate that this initiative’s data and training effort will have a wide range of beneficial applications.”

“However,” the center said, “certain subsets of data that we term “highly sensitive biological data” (described below) pose potential risks when used to train AI models. This can include data in the form of natural language or code primarily used for large language models (LLMs) or biological data primarily used to train BAIMs.”

It said, “The datasets we would consider to be most highly sensitive are those that would create pandemic-level risks in new AI models as defined by the ability of such a model to:”

(1) Greatly accelerate or simplify the reintroduction of dangerous extinct viruses or dangerous viruses that only exist now within research labs that could have the capacity to start pandemics, panzootics, or panphytotics; or

(2) Substantially enable, accelerate, or simplify the creation of novel variants of pathogens or entirely novel biological constructs that could start such pandemics

“Determining which outcomes we are trying to prevent (pandemic-level risks) and then working back from that to determine what kinds of capabilities would enable those outcomes, as well as determining what types of data would enable those capabilities to emerge, would help to focus DOE’s resources on the most concerning risks to the public while not impeding the great majority of beneficial research at the intersection of AI and the life sciences,” CHS said.

“DOE should establish data governance practices that prevent the release of highly sensitive biological data from open public use while at the same time allowing researchers with legitimate need to access such data for beneficial purposes to have a path for doing so,” it said.

Models

The Johns Hopkins center provides DOE with three principles “to reduce the potential risks that AI models with these hazardous capabilities are developed and accidentally or deliberately misused in ways that affect national security:”

(1) Prevent DOE’s resources from being used to develop models that are likely to lead to pandemic-level risks. The risks and benefits of allowing such AI models to be developed should be weighed as part of a formal governance process. If such models are to be allowed because benefits are determined to outweigh the extraordinary risks, then the models should not be open sourced or made public.

(2) For any model development process that meets the above criteria and is allowed to proceed, DOE should ensure that such models are subject to adequate cybersecurity standards to avoid risks of illegitimate access or theft or leak of model weights, and it should appropriately address risks from insider threats. While such work should be conducted within secured governmental digital and physical environments (such as testbeds), DOE should ensure similar safety and security standards if the models are to be shared with outside stakeholders (eg, academia or industry partners).

(3) Refrain from publishing model weights, code, or other information enabling fine-tuning or modification that would result in the above-noted pandemic-level outcomes, such as cases in which an AI model could be fine-tuned on highly sensitive biological data or otherwise modified to exhibit hazardous capabilities (eg, via removal of technical safeguards).

And it offers “steps for red-teaming and safety testing” to “reduce pandemic-level risks that could be posed by new AI models:”

  • Step 1. Define hazardous capabilities that could lead to pandemic-level risks: DOE should select these based on their ability to contribute to causing pandemic-level harms and work with policy and scientific experts to horizon scan for emerging capabilities.
  • Step 2. Establish risk thresholds assessed via evaluations: DOE should clearly define risk thresholds (before model evaluation) that are quantifiable via model evaluation for hazardous capabilities and then link these risk thresholds to appropriate mitigation measures that will be implemented if these thresholds are crossed.
  • Step 3. Develop and conduct evaluations for these hazardous capabilities: DOE should standardize evaluations across hazardous capabilities that are both repeatable and quantifiable so that they can be accurately utilized as risk thresholds as discussed in Step 2. Evaluations can take the form of red teaming, automated benchmarking, assessing an AI model’s uplift potential compared to individuals without access to the model via controlled trials, or assessing the extent to which models provide completely novel capabilities (compared to uplift, which makes existing capabilities easier). DOE should conduct these evaluations before model release and deployment so that risk mitigation measures can be implemented before the model release if risks exceed risk thresholds for hazardous capabilities.
  • Step 4. Deploy risk mitigation measures for respective risk thresholds via a tiered system: DOE should plan for risk mitigation measures that correlate with the extent to which a new model exceeds risk thresholds. Such mitigation measures could include a range of actions, such as limiting access to model weights and removing dangerous information from a model after the initial training has been completed, know-your[1]customer screening, restricting access to a model to specific users via application programming interface (API) or other secure means, or pausing/stopping model development altogether.

CHS encourages DOE to “consider the creation of a public-private forum in which representatives of government, academia, industry, and civil society can share information regarding potential risks and mitigation strategies related to AI models that could create new hazardous biological capabilities.”

The center notes, “AI developers and industry are currently best positioned to understand the power, complexities, and technical capabilities of their models, while government and nongovernmental experts on the life sciences, biosafety, and biosecurity are best positioned to understand the nature and likelihood of substantial pandemic threats.”