by Paul Scharre
341
“the most powerful limitations”: Schelling, Arms and Influence, 164.
341
“ ‘Some gas’ raises complicated questions”: Thomas C. Schelling, The Strategy of Conflict (Cambridge, MA: Harvard University, 1980), 75.
341
Escalation from one step to another: Schelling makes this point in Arms and Influence in a critique of the McNamara “no cities” doctrine. Schelling, Arms and Influence, 165.
342
“If they declare that they will attack”: “Hitlers Bombenterror: ‘Wir Werden Sie Ausradieren,’ ” Spiegel Online, accessed April 1, 2003, http://www.spiegel.de/spiegelspecial/a-290080.html.
342
Complete bans on weapons: This seems to suggest that if lasers were used in future wars for non-blinding purposes and ended up causing incidental blinding, then they would quickly evolve into use for intentional blinding.
342
“antipersonnel land mine”: Human Rights Watch, “Yemen: Houthi Landmines Claim Civilian Victims,” September 8, 2016, https://www.hrw.org/news/2016/09/08/yemen-houthi-landmines-claim-civilian-victims.
343
SMArt 155 artillery shells: “Fitzgibbon Wants to Keep SMArt Cluster Shells,” Text, ABC News, (May 29, 2008), http://www.abc.net.au/news/2008-05-29/fitzgibbon-wants-to-keep-smart-cluster-shells/2452894.
343
poison gas attack at Ypres: Jonathan B. Tucker, War of Nerves: Chemical Warfare from World War I to Al-Qaeda (New York: Pantheon Books, 2006).
343
“or by other new methods”: Declaration (IV,1), to Prohibit, for the Term of Five Years, the Launching of Projectiles and Explosives from Balloons, and Other Methods of Similar Nature. The Hague, July 29, 1899, https://ihl-databases.icrc.org/applic/ihl/ihl.nsf/385ec082b509e76c41256739003e636d/53024c9c9b216ff2c125641e0035be1a?OpenDocument.
343
“attack or bombardment”: Regulations: Article 25, Convention (IV) respecting the Laws and Customs of War on Land and its annex: Regulations concerning the Laws and Customs of War on Land. The Hague, October 18, 1907, https://ihl-databases.icrc.org/applic/ihl/ihl.nsf/Article.xsp?action=openDocument&documentId=D1C251B17210CE8DC12563CD0051678F.
343
“the bomber will always get through”: “The bomber will always get through,” Wikipedia, https://en.wikipedia.org/wiki/The_bomber_will_always_get_through.
344
Nuclear Non-Proliferation Treaty: “Treaty on the Non-Proliferation of Nuclear Weapons (NPT),” United Nations Office for Disarmament Affairs, https://www.un.org/disarmament/wmd/nuclear/npt/text, accessed June 19, 2017.
344
Chemical Weapons Convention: “Chemical Weapons Convention,” Organisation for the Prohibition of Chemical Weapons, https://www.opcw.org/chemical-weapons-convention/, accessed June 19, 2017.
344
INF Treaty: “Treaty Between the United States of American and the Union of Soviet Socialist Republics on the Elimination of their Intermediate-Range and Shorter-Range Missiles.”
344
START: “Treaty Between the United States of American and the Union of Soviet Socialist Republics on the Reduction and Limitation of Strategic Offensive Arms,” U.S. Department of State, https://www.state.gov/www/global/arms/starthtm/start/start1.html, accessed June 19, 2017.
344
New START: “New Start,” U.S. Department of State, https://www.state.gov/t/avc/newstart/, accessed June 19, 2017.
344
Outer Space Treaty: Treaty on Principles Governing the Activities of States in the Exploration and Use of Outer Space, including the Moon and Other Celestial Bodies.
344
expanding bullets: Declaration (IV,3) concerning Expanding Bullets, The Hague, July 29, 1899, https://ihl-databases.icrc.org/applic/ihl/ihl.nsf/Article.xsp?action=openDocument&documentId=F5FF4D9CA7E41925C12563CD0051616B.
344
Geneva Gas Protocol: Protocol for the Prohibition of the Use of Asphyxiating, Poisonous or Other Gases, and of Bacteriological Methods of Warfare, Geneva, June 17, 1925, https://ihl-databases.icrc.org/ihl/INTRO/280?OpenDocument.
344
CCW: “The Convention on Certain Conventional Weapons,” United Nations Office for Disarmament Affairs, https://www.un.org/disarmament/geneva/ccw/, accessed June 19, 2017.
344
SORT: “Treaty Between the United States of America and the Russian Federation On Strategic Offensive Reductions (The Moscow Treaty),” U.S. Department of State, https://www.state.gov/t/isn/10527.htm, accessed June 19, 2017.
344
weapons of mass destruction (WMD) in orbit: Treaty on Principles Governing the Activities of States in the Exploration and Use of Outer Space, including the Moon and Other Celestial Bodies.
344
Environmental Modification Convention: Convention on the Prohibition of Military or Any Other Hostile Use of Environmental Modification Techniques.
344
Biological Weapons Convention: “The Biological Weapons Convention,” United Nations Office for Disarmament Affairs, https://www.un.org/disarmament/wmd/bio/, accessed June 19, 2017.
344
secret biological weapons program: Tim Weiner, “Soviet Defector Warns of Biological Weapons,” New York Times, February 24, 1998. Milton Leitenberg, Raymond A. Zilinskas, and Jens H. Kuhn, The Soviet Biological Weapons Program: A History (Cambridge: Harvard University Press, 2012). Ken Alibek, Biohazard: The Chilling True Story of the Largest Covert Biological Weapons Program in the World (Delta, 2000). Raymond A. Zilinskas, “The Soviet Biological Weapons Program and Its Legacy in Today’s Russia,” CSWMD Occasional Paper 11, July 18, 2016.
345
Other weapons can be: The lack of a verification regime has been a long-standing concern regarding the Biological Weapons Convention. “Biological Weapons Convention (BWC) Compliance Protocol,” NTI, August 1, 2001, http://www.nti.org/analysis/articles/biological-weapons-convention-bwc/. “The Biological Weapons Convention: Proceeding without a Verification Protocol,” Bulletin of the Atomic Scientists, May 9, 2011, http://thebulletin.org/biological-weapons-convention-proceeding-without-verification-protocol.
21 Are Autonomous Weapons Inevitable? The Search for Lethal Laws of Robotics
346
Convention on Certain Conventional Weapons (CCW): “2014 Meeting of Experts on LAWS,” The United Nations Office at Geneva, http://www.unog.ch/__80256ee600585943.nsf/(httpPages)/a038dea1da906f9dc1257dd90042e261?OpenDocument&ExpandSection=1#_Section1. “2015 Meeting of Experts on LAWS,” The United Nations Office at Geneva, http://www.unog.ch/80256EE600585943/(httpPages)/6CE049BE22EC75A2C1257C8D00513E26?OpenDocument; “2016 Meeting of Experts on LAWS,” The United Nations Office at Geneva, http://www.unog.ch/80256EE600585943/(httpPages)/37D51189AC4FB6E1C1257F4D004CAFB2?OpenDocument.
347
“appropriate human involvement”: CCW, “Report of the 2016 Informal Meeting of Experts on Lethal Autonomous Weapon Systems (LAWS), June 10, 2016.
348
“How near to a city is”: Schelling, Arms and Influence, 165.
349
“partition”: Article 36, “Autonomous weapons—the risks of a management by ‘partition,’ ” October 10, 2012, http://www.article36.org/processes-and-policy/protection-of-civilians/autonomous-weapons-the-risks-of-a-management-by-partition/.
349
Campaign to Stop Killer Robots has called: “A comprehensive, pre-emptive prohibition on the development, production and use of fully autonomous weapons.” The Campaign to Stop Killer Robots, “The Solution,” http://www.stopkillerrobots.org/the-solution/.
349
technology is too diffuse: Ackerman, “We Should Not Ban ‘Killer Robots,’ and Here’s Why.”
349
“not a wise campaign strategy”: Steve Goose, interview, October 26, 2016.
350
care more about human rights: Ian Vasquez and Tanja Porcnik, “The Human Freedom Index 2016,” Cato Institute, the Fraser Institute, and the
Friedrich Naumann Foundation for Freedom, 2016, https://object.cato.org/sites/cato.org/files/human-freedom-index-files/human-freedom-index-2016.pdf.
351
“You know you’re not”: Steve Goose, interview, October 26, 2016.
351
A few experts have presented: For example, Rickli, “Some Considerations of the Impact of LAWS on International Security: Strategic Stability, Non-State Actors and Future Prospects.”
351
“it’s not really a significant feature”: John Borrie, interview, April 12, 2016.
352
“offensive autonomous weapons beyond meaningful”: “Autonomous Weapons: An Open Letter From AI & Robotics Researchers.”
352
“will move toward some type”: Bob Work, interview, June 22, 2016.
355
no antipersonnel equivalents: Precision-guided weapons are evolving down to the level of infantry combat, including some laser-guided munitions such as the DARPA XACTO and Raytheon Spike missile. Because these are laser-guided, they are still remotely controlled by a person.
355
“let machines target machines”: Canning, “A Concept of Operations for Armed Autonomous Systems.”
356
“stopping an arms race”: Stuart Russell Walsh Max Tegmark and Toby, “Why We Really Should Ban Autonomous Weapons: A Response,” IEEE Spectrum: Technology, Engineering, and Science News, August 3, 2015, http://spectrum.ieee.org/automaton/robotics/artificial-intelligence/why-we-really-should-ban-autonomous-weapons.
357
focus on the unchanging element in war: The ICRC, for example, has called on states to “focus on the role of the human in the targeting process.” International Committee of the Red Cross, “Views of the International Committee of the Red Cross (ICRC) on autonomous weapon system,” paper submitted to the Convention on Certain Conventional Weapons (CCW) Meeting of Experts on Lethal Autonomous Weapons Systems (LAWS), April 11, 2016, 5, available for download at https://www.icrc.org/en/document/views-icrc-autonomous-weapon-system.
358
Phrases like . . . “appropriate human involvement”: Heather M. Roff and Richard Moyes, “Meaningful Human Control, Artificial Intelligence and Autonomous Weapons,” Briefing paper prepared for the Informal Meeting of Experts on Lethal Autonomous Weapons Systems, UN Convention on Certain Conventional Weapons, April 2016, http://www.article36.org/wp-content/uploads/2016/04/MHC-AI-and-AWS-FINAL.pdf. Human Rights Watch, “Killer Robots and the Concept of Meaningful Human Control,” April 11, 2016, https://www.hrw.org/news/2016/04/11/killer-robots-and-concept-meaningful-human-control. UN Institute for Disarmament Research, “The Weaponization of Increasingly Autonomous Technologies: Considering how Meaningful Human Control might move the discussion forward,” 2014, http://www.unidir.org/files/publications/pdfs/considering-how-meaningful-human-control-might-move-the-discussion-forward-en-615.pdf. Michael Horowitz and Paul Scharre, “Meaningful Human Control in Weapon Systems: A Primer,” Center for a New American Security, Washington DC, March 16, 2015, https://www.cnas.org/publications/reports/meaningful-human-control-in-weapon-systems-a-primer. CCW, “Report of the 2016 Informal Meeting of Experts on Lethal Autonomous Weapon Systems (LAWS).
358
“The law of war rules”: Department of Defense, “Department of Defense Law of War Manual,” 330.
Conclusion: No Fate but What We Make
360
Sarah and John are forever trapped: Kudos to Darren Franich for a mind-melting attempt to map the Terminator timelines: “ ‘Terminator Genisys’: The Franchise Timeline, Explained,” EW.com, June 30, 2015, http://ew.com/article/2015/06/30/terminator-genisys-franchise-timeline-explained/.
Acknowledgments
A book is a strange thing. It takes scores of people to make a book, but only one person’s name goes on the cover. This book began ten years ago, in a conversation with my good friend, Gene Tien. Along the way, many people have helped shape the ideas within. This book would not have been possible without their help.
I want to thank the executive team at the Center for a New American Security (CNAS): Michèle Flournoy, Richard Fontaine, Shawn Brimley, David Romley, and former CNAS CEO Bob Work. They have been incredibly supportive in the development of this book. My CNAS colleague Robert Kaplan has been an amazing mentor along the way, and I am very grateful for his advice and guidance.
Elements of this book draw upon my work for CNAS’s Ethical Autonomy Project, which was made possible by the John D. and Catherine T. MacArthur Foundation. I want to thank Jeff Ubois of MacArthur for his interest and support in our work. I owe a special thanks to Michael Horowitz, who led the Ethical Autonomy Project with me and has been a frequent coauthor on many reports. Kelley Sayler, Alex Velez-Green, Adam Saxton, and Matt Seeley also provided invaluable help in our research and publications. Melody Cook designed many of the graphics used in our project, which have been repurposed in this book with the permission of CNAS.
Many of the ideas in this book began years ago when I worked in the Pentagon. The patience and foresight of my leadership team—Leslie Hunter, Jen Zakriski, Todd Harvey, David Ochmanek, Kathleen Hicks, and James N. Miller—gave me the freedom to proactively engage on this issue, a rare occurrence in a bureaucracy. John Hawley and Bobby Junker, both top-notch scientists and amazing human beings, helped shape my thinking on the nature of autonomy and human control. Andrew May, Andy Marshall, and Bob Bateman were some of the earliest to recognize the importance of these issues and support work in this area.
I would like to thank the many individuals who agreed to be interviewed for this book: Ken Anderson, Peter Asaro, Ron Arkin, Stuart Armstrong, Paul Bello, John Borrie, Selmer Bringsjord, Brian Bruggeman, Ray Buettner, John Canning, Micah Clark, Jeff Clune, Kelly Cohen, Mary “Missy” Cummings, David Danks, Duane Davis, Neil Davison, Charles Dela Cuesta, Tom Dietterich, Bonnie Docherty, Charlie Dunlap, Peter Galluch, Steve Goose, John Hawley, Michael Horowitz, Christof Heyns, Frank Kendall, William Kennedy, Tom Malinowski, Bryan McGrath, Mike Meier, Heather Roff, Stuart Russell, Larry Schuette, Bradford Tousley, Brandon Tseng, Kerstin Vignard, Michael Walker, Mary Wareham, Steve Welby, Bob Work, and Jody Williams. Unfortunately, I was not able to include all of their interviews for reasons of space, but their ideas and insights helped shape the book in ways big and small. My thoughts on autonomous weapons have also been shaped over the years by many fellow travelers on this topic: Karl Chang, Rebecca Crootof, David Koplow, Kathleen Lawand, Patrick Lin, Matt McCormack, Brian Hall, Tim Hwang, David Simon, Shawn Steene, Noel Sharkey, Ryan Tewell, Alex Wagner, and Matt Waxman, among many others. I am grateful to Elbridge Colby and Shawn Steene for their feedback on drafts of the book. Maura McCarthy’s sharp eye as an editor helped ensure that this book’s proposal landed a publisher. My colleagues Neal Urwitz, JaRel Clay, and Jasmine Butler have been incredibly helpful in promotion and outreach. Thanks to Jennifer-Leigh Oprihory for suggesting the title.
The U.S. Department of Defense was very open and supportive throughout the development of this book. I would especially like to thank Jared Adams at the Defense Advanced Research Projects Agency (DARPA), and Kimberly Lansdale at the Aegis Training and Readiness Center, for their support and assistance in facilitating visits and interviews.
I’d like to thank my agent, Jim Hornfischer at Hornfischer Literary Management, and my editor, Tom Mayer at W. W. Norton, for taking a chance on me and this book. I owe a special thanks to Emma Hitchcock, Sarah Bolling, Kyle Radler, and the rest of the team at Norton for bringing this book to fruition.
My brother, Steve, has been incredibly patient over the years listening to me prattle on about autonomous weapons, and provided valuable feedback on early chapters. I owe everything to my parents, Janice and David. Nothing I’ve done would have been possible without their love and support.
Most of all, I am forever grateful to my wife Heather, who now knows more about autonomous weapons than she ever cared to. Her patience, love, and support made this p
ossible.
Abbreviations
AAA
antiaircraft artillery
ABM
Anti-Ballistic Missile
ACTUV
Anti-submarine warfare Continuous Trail Unmanned Vessel
AGI
artificial general intelligence
AGM
air-to-ground missile
AI
artificial intelligence
AMRAAM
Advanced Medium-Range Air-to-Air Missile
ARPA
Advanced Research Projects Agency
ASI
artificial superintelligence
ASW
anti-submarine warfare
ATR
automatic target recognition
BDA
battle damage assessment
BWC
Biological Weapons Convention
CCW
Convention on Certain Conventional Weapons
C&D
Command and Decision
CIC
combat information center
CIWS
Close-In Weapon System
CODE
Collaborative Operations in Denied Environments
DARPA
Defense Advanced Research Projects Agency
DDoS
distributed denial of service
DIY
do-it-yourself
DMZ
demilitarized zone
DoD
Department of Defense
FAA
Federal Aviation Administration
FIAC
fast inshore attack craft
FIS
Fire Inhibit Switch
