how an a.i. could churn out millions of bioweapons, and how we'd fight back

Don't worry about the AI, worry about the humans training it, say researchers who found out how easy it is to turn machines evil.

Despite IBM's fiasco with Watson's attempt to help treat cancers, one of the most exciting and promising uses of artificial intelligence is in medicine. When you let machines innovate and help expert clinicians, you get new tools to detect tumors earlier and recipes for new antibiotics when we're in dire need of them. But an international team of scientists who specialize in thinking of nightmare scenarios, then figuring out how likely they are and how we could stop them, asked the terrifying question of whether medical AI could be misused.

Not satisfied with just giving us a horrifying question with which to stay awake at night, they did exactly that and tasked a model designed to come up with new medications with creating brand new bioweapons. It immediately spat out a formula for a more potent VX gas, an already highly effective nerve agent used by North Korean dictator Kim Jong Un to assassinate his brother. Of course, that wasn't the end. In the 40,000 molecules the AI created in just six hours, thousands appeared to be brand new bioweapons deadlier than anything we know of today.

Obviously, the team refused to publish these molecular structures or synthesize them, but they proved a very important point. If you take a system designed to create new chemistry, moving the goal settings from "heal humans" to "harm humans" will turn it into a factory for monstrous poisons, and in the hands of awful people, these AIs could be misused for chemical warfare on unprecedented scales. Computer scientists have been worried about this for a long time, and were disturbed how easy it actually was after carrying out this experiment.

So, time to panic? Not quite, Thankfully, this experiment also provides a solution to the problem it raises. By feeding these new deadly compounds into similar AI models, we can also discover molecular designs for antidotes, and just as easily create cures for deadly new bioweapons. It's also important to keep in mind that many of these agents could be too impractical to synthesize, so even if we knew they would be the deadliest poisons ever unleashed if we created them, the process could be so difficult and unstable, they could never be weaponized.

In the grand scheme of things, yes, we could generate millions of designs for toxins and nerve agents in a matter of weeks. But we could also generate millions of countermeasures while the villains attempt to figure out how to create those millions of molecules in the real world, much less do it at scale and find effective means of distribution. How long will the molecules remain stable? How long can they be stored? Can they be stored in the first place or would they just dissolve their containers? The AIs can't tell us that, only the molecule's specifications.

Now, all that said, it would still be possible for terrorists and warlords to create powerful and effective bioweapons much faster with an AI's help, so this is still a serious threat to consider. But it won't be quite as easy as it seems, and we would be far from powerless to stop them, armed with the same types of models and the infrastructure to mass produce countermeasures to all sorts of possible attacks. Our vigilance is more than warranted, but as long as we can still flip the switch back to "cure," we don't have to fear an AI-accelerated apocalypse.

See: Urbina, F., et al. (2022) Dual use of artificial-intelligence-powered drug discovery. Nat Mach Intell 4, 189–191 DOI: 10.1038/s42256-022-00465-9