When ballistic missiles can see

When I was a kid, I read a lot of sci-fi books. One of the most common themes was “man vs. machine,” which often took the form of robots becoming self-aware and threatening humanity. This theme has also become a staple of Hollywood movies like The Terminator and The Matrix.

Despite the prevalence of this theme, I don’t lose any sleep worrying about this scenario. But I do think we should spend more time thinking about the implications—positive and negative—of recent progress in artificial intelligence, machine learning, and machine vision. For example, militaries have begun to develop drones, ships, subs, tanks, munitions, and robotic troops with increasing levels of intelligence and autonomy.

While this use of A.I. holds great promise for reducing civilian casualties and keeping more troops out of harm’s way, it also presents the possibility of unintended consequences if we’re not careful. Earlier this year, U.N. Secretary General António Guterres called global attention to these threats: “The weaponization of artificial intelligence is a growing concern. The prospect of weapons that can select and attack a target on their own raises multiple alarms…. The prospect of machines with the discretion and power to take human life is morally repugnant.”

Unfortunately, my first attempt to educate myself on autonomous weapons was a bust. I read a book that was dry and felt really outdated. Then a few months ago I picked up Army of None: Autonomous Weapons and the Future of War, by Paul Scharre. It’s the book I had been waiting for. I can’t recommend it highly enough.

Scharre is a great thinker who has both on-the-ground experience and a high-level view. He’s a former Army Ranger who served four tours of combat duty in Iraq and Afghanistan. He then went onto a policy role at the U.S. Department of Defense and led the working group that drafted the government’s policy on autonomous weapons. He’s currently a policy expert at the Center for a New American Security, a center-left think tank in DC.

He is also a good writer. Scharre writes clearly about a huge range of topics: computer science, military strategy, history, philosophy, psychology, and ethics. He gives you the right grounding to start participating in the debate over where our country should draw the line on these powerful technologies.

Scharre makes clear from the beginning that he has no problem with some well-bounded military uses of autonomy. For example, he brings you along for a tour of the U.S. Navy’s Aegis Combat System, an advanced system for tracking and guiding missiles at sea. Aegis has a mode of operation in which human operators delegate all firing decisions to an advanced computer (but can override them if necessary). Why would you want to put a computer in charge? If you’re out at sea and an enemy fires 50 missiles at you all at once, you’d be very happy to have a system that can react much faster than a human could.

Army of None also shows that autonomy has great benefits in environments where humans can’t survive (such as flight situations with high G forces) or in which communications have broken down. It can be enormously helpful to have an unmanned drone, tank, or sub that carries out a clear, limited mission with little communication back and forth with human controllers.

In addition, autonomous weapons could potentially help save civilian lives. Scharre cites robotics experts who argue that “autonomous weapons … could be programmed to never break the laws of war…. They wouldn’t seek revenge. They wouldn’t get angry or scared. They would take emotion out of the equation. They could kill when necessary and then turn killing off in an instant.”

Despite these and other advantages, Scharre does not want the military ever to turn over judgment to computers. To make his case, he offers compelling real-life cases in which human judgment was essential for preventing needless killing, such as his own experiences in Afghanistan. “A young girl of maybe five or six headed out of the village and up our way, two goats in trail. Ostensibly she was just herding goats, but she [was actually] spotting for Taliban fighters.” Scharre’s unit did not shoot. Yes, it would have been legal, but he argues that it would not have been morally right. A robotic sniper following strict algorithms might well have opened fire the second it detected a radio in her hand.

Scharre ends the book by exploring the possibility of an international ban on fully autonomous weapons. He concludes that this kind of absolute ban is not likely to succeed. However, he holds out hope that enlightened self-interest could bring countries together to ban specific uses of autonomous weapons, such as those that target individual people. He also believes it’s feasible to establish non-binding rules of the road that could reduce the potential for autonomous systems to set each other off accidentally. He also believes we could update the international laws of war to embed a common principle for human involvement in lethal force.

There are no easy answers here. But I agree with Scharre that we have to guard against becoming “seduced by the allure of machines—their speed, their seeming perfection, their cold precision.” And we should not leave it up to military planners or the people writing software to determine where to draw the proper lines. We need many experts and citizens across the globe to get involved in this important debate.