by Paul Scharre
102
Even Shiite militias in Iraq: David Axe, “An Iraqi Shi’ite Militia Now Has Ground Combat Robots,” War Is Boring, March 23, 2015, https://warisboring.com/an-iraqi-shi-ite-militia-now-has-ground-combat-robots-68ed69121d21#.hj0vxomjl.
103
armed uninhabited boat, the Protector: Berenice Baker, “No Hands on Deck—Arming Unmanned Surface Vessels,” naval-technology.com, November 23, 2012, http://www.naval-technology.com/features/featurehands-on-deck-armed-unmanned-surface-vessels/.
103
Singapore has purchased the Protector: “Protector Unmanned Surface Vessel, Israel,” naval-technology.com, http://www.naval-technology.com/projects/protector-unmanned-surface-vehicle/.
103
ESGRUM: “BAE ESGRUM USV,” NavalDrones.com, http://www.navaldrones.com/BAE-ESGRUM.html.
103
Only twenty-two nations have said they support: Campaign to Stop Killer Robots, “Country Views on Killer Robots,” November 16, 2017, http://www.stopkillerrobots.org/wp-content/uploads/2013/03/KRC_CountryViews_16Nov2017.pdf.
104
These include the United Kingdom’s Taranis: Nicholas de Larringa, “France Begins Naval Testing of Neuron UCAV,” IHS Jane’s Defence Weekly, May 19, 2016; archived athttps://web.archive.org/web/20161104112421/http://www.janes.com/article/60482/france-begins-naval-testing-of-neuron-ucav. “New Imagery Details Indian Aura UCAV,” Aviation Week & Space Technology, July 16, 2012, http://aviationweek.com/awin/new-imagery-details-indian-aura-ucav. “Israel Working on Low-Observable UAV,” FlightGlobal, November 28, 2012, https://www.flightglobal.com/news/articles/israel-working-on-low-observable-uav-379564/.
104
a handful of countries already possess the fully autonomous Harpy: Tamir Eshel, “IAI Introduces New Loitering Weapons for Anti-Radiation, Precision Strike.”
105
“the ultimate decision about shooting”: Jean Kumagai, “A Robotic Sentry For Korea’s Demilitarized Zone,” IEEE Spectrum: Technology, Engineering, and Science News, March 1, 2007, http://spectrum.ieee.org/robotics/military-robots/a-robotic-sentry-for-koreas-demilitarized-zone.
105
SGR-A1 cited as an example: Christopher Moyer, “How Google’s AlphaGo Beat a Go World Champion,” The Atlantic, March 28, 2016, https://www.theatlantic.com/technology/archive/2016/03/the-invisible-opponent/475611/. Adrianne Jeffries, “Should a Robot Decide When to Kill?,” The Verge, January 28, 2014, https://www.theverge.com/2014/1/28/5339246/war-machines-ethics-of-robots-on-the-battlefield. “Future Tech? Autonomous Killer Robots Are Already Here,” NBC News, May 16, 2014, http://www.nbcnews.com/tech/security/future-tech-autonomous-killer-robots-are-already-here-n105656. Sharon Weinberger, “Next Generation Military Robots Have Minds of Their Own,” accessed June 18, 2017, http://www.bbc.com/future/story/20120928-battle-bots-think-for-themselves. “The Scariest Ideas in Science,” Popular Science, accessed June 18, 2017, http://www.popsci.com/scitech/article/2007-02/scariest-ideas-science.
105
“WHY, GOD? WHY?”: “The Scariest Ideas in Science.”
105
cited the SGR-A1 as fully autonomous: Ronald C. Arkin, Governing Lethal Behavior in Autonomous Robots (Boca Raton, FL: Taylor & Francis Group, 2009), 10. Patrick Lin, George Bekey, and Keith Abney, “Autonomous Military Robotics: Risk, Ethics, and Design,” California Polytechnic State University, San Luis Obispo, CA, December 20, 2008, http://digitalcommons.calpoly.edu/cgi/viewcontent.cgi?article=1001&context=phil_fac; Hin-Yan Liu, “Categorization and legality of autonomous and remote weapons systems,” International Review of the Red Cross 94, no. 886 (Summer 2012), https://www.icrc.org/eng/assets/files/review/2012/irrc-886-liu.pdf.
105
In 2010, a spokesperson for Samsung: “Machine Gun-Toting Robots Deployed on DMZ,” Stars and Stripes, July 12, 2010, https://www.stripes.com/machine-gun-toting-robots-deployed-on-dmz-1.110809.
105
“the SGR-1 can and will prevent wars”: Ibid.
105
critics who have questioned whether it has too much: Markoff, “Fearing Bombs That Can Pick Whom to Kill.”
106
Brimstone has two primary modes of operation: MBDA, “Brimstone 2 Data Sheet November 2015,” accessed June 7, 2017, https://mbdainc.com/wp-content/uploads/2015/11/Brimstone2-Data-Sheet_Nov-2015.pdf.
106
“This mode provides through-weather targeting”: Ibid.
106
“It can identify, track, and lock on”: David Hambling Dec 4 and 2015, “The U.K. Fights ISIS With a Missile the U.S. Lacks,” Popular Mechanics, December 4, 2015, http://www.popularmechanics.com/military/weapons/a18410/brimstone-missile-uk-david-cameron-isis/.
107
“In May 2013, multiple Brimstone”: MBDA, “Brimstone 2 Data Sheet.”
107
reported range in excess of 20 kilometers: “Dual-Mode Brimstone Missile Proves Itself in Combat,” Defense Media Network, April 26, 2012, http://www.defensemedianetwork.com/stories/dual-mode-brimstone-missile-proves-itself-in-combat/.
107
Brimstone can engage these targets: I am grateful to MBDA Missile Systems for agreeing to an interview on background in 2016 to discuss the Brimstone’s functionality.
109
“1. Taranis would reach the search area”: BAE Systems, “Taranis: Looking to the Future,” http://www.baesystems.com/en/download-en/20151124120336/1434555376407.pdf.
109
BAE Chairman Sir Roger Carr: Sir Roger Carr, “What If: Robots Go to War?—World Economic Forum Annual Meeting 2016 | World Economic Forum,” video, accessed June 7, 2017, https://www.weforum.org/events/world-economic-forum-annual-meeting-2016/sessions/what-if-robots-go-to-war.
109
“decisions to release a lethal mechanism”: John Ingham, “WATCH: Unmanned Test Plane Can Seek and Destroy Heavily Defended Targets,” Express.co.uk, June 9, 2016, http://www.express.co.uk/news/uk/678514/WATCH-Video-Unmanned-test-plane-Taranis.
109
“The UK does not possess”: UK Ministry of Defence, “Defence in the Media: 10 June 2016,” June 10, 2016, https://modmedia.blog.gov.uk/2016/06/10/defence-in-the-media-10-june-2016/.
110
“must be capable of achieving”: UK Ministry of Defence, “Joint Doctrine Note 2/11: The UK Approach to Unmanned Aircraft Systems,” March 30, 2011, https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/33711/20110505JDN_211_UAS_v2U.pdf, 2-3.
110
“As computing and sensor capability increases”: Ibid, 2–4.
111
one short-lived effort during the Iraq war: “The Inside Story of the SWORDS Armed Robot ‘Pullout’ in Iraq: Update,” Popular Mechanics, October 1, 2009, http://www.popularmechanics.com/technology/gadgets/4258963.
112
“the military robots were assigned”: Alexander Korolkov and special to RBTH, “New Combat Robot Is Russian Army’s Very Own Deadly WALL-E,” Russia Beyond The Headlines, July 2, 2014, https://www.rbth.com/defence/2014/07/02/new_combat_robot_is_russian_armys_very_own_deadly_wall-e_37871.html.
112
“Platform-M . . . is used”: Ibid.
112
videos of Russian robots show soldiers: This video (https://www.youtube.com/watch?v=RBi977p0plA) is no longer available online.
114
According to David Hambling of Popular Mechanics: David Hambling, “Check Out Russia’s Fighting Robots,” Popular Mechanics, May 12, 2014, http://www.popularmechanics.com/technology/military/robots/russia-wants-autonomous-fighting-robots-and-lots-of-them-16787165.
114
amphibious Argo: “Battle Robotic complex ‘Argo’—Military Observer,” accessed June 7, 2017, http://warsonline.info/bronetechnika/boevoy-robotizirovanniy-kompleks-argo.html.
114
Pictures online show Russian soldiers: Tamir Eshel, “Russian Military to Test Combat Robots in 2016,” Defense Update, December 31, 2015, http://defense-update.com/20151231_russian-combat-robots.htm
l.
115
slo-mo shots of the Uran-9 firing: Rosoboronexport, Combat Robotic System Uran-9, n.d., https://www.youtube.com/watch?v=VBC9BM4-3Ek.
115
Uran-9 could make the modern battlefield a deadly place: Rich Smith, “Russia’s New Robot Tank Could Disrupt the International Arms Market,” The Motley Fool, February 7, 2016, https://www.fool.com/investing/general/2016/02/07/russias-new-robot-tank-could-disrupt-international.aspx.
115
Vikhr . . . “lock onto a target”: Simon Holmes, “Russian Army Puts New Remote-Controlled Robot Tank to Test,” Mail Online, April 29, 2017, http://www.dailymail.co.uk/~/article-4457892/index.html.
115
DJI’s base model Spark: “Spark,” DJI.com, http://www.dji.com/spark.
116
T-14 Armata: Alex Lockie, “Russia Claims Its Deadly T-14 Armata Tank Is in Full Production,” Business Insider, March 17, 2016, http://www.businessinsider.com/russia-claims-t14-armata-tank-is-in-production-2016-3.
116
“Quite possibly, future wars will be waged”: “Engineers Envisioned T-14 Tank ‘robotization’ as They Created Armata Platform,” RT International, June 1, 2015, https://www.rt.com/news/263757-armata-platform-remote-control/.
116
“fully automated combat module”: “Kalashnikov Gunmaker Develops Combat Module based on Artificial Intelligence,” TASS Russian News Agency, July 5, 2017, http://tass.com/defense/954894.
117
“Another factor influencing the essence of modern”: Coalson, “Top Russian General Lays Bare Putin’s Plan for Ukraine.”
117
Deputy Secretary of Defense Bob Work mentioned: “Remarks by Defense Deputy Secretary Robert Work at the CNAS Inaugural National Security Forum,” December 14, 2015, https://www.cnas.org/publications/transcript/remarks-by-defense-deputy-secretary-robert-work-at-the-cnas-inaugural-national-security-forum.
117
some have suggested, that a dangerous arms race: Markoff, “Fearing Bombs That Can Pick Whom to Kill.”
118
Policy discussions may be happening: Thanks to Sam Bendett for pointing out this important distinction.
118
NGO Article 36: Article 36, “The United Kingdom and Lethal Autonomous Weapon Systems,” April 2016, http://www.article36.org/wp-content/uploads/2016/04/UK-and-LAWS.pdf
8 Garage Bots: DIY Killer Robots
119
fifteen-second video clip . . . Connecticut teenager: Rick Stella, “Update: FAA Launches Investigation into Teenager’s Gun-Wielding Drone Video,” Digital Trends, July 22, 2015, https://www.digitaltrends.com/cool-tech/man-illegally-straps-handgun-to-a-drone/.
119
For under $500: “Spark,” DJI.com.
122
Shield AI: “Shield AI,” http://shieldai.com/
122
grant from the U.S. military: Mark Prigg, “Special Forces developing ‘AI in the sky’ drones that can create 3D maps of enemy lairs: Pentagon reveals $1m secretive ‘autonomous tactical airborne drone’ project,” DailyMail.com, http://www.dailymail.co.uk/sciencetech/article-3776601/Special-Forces-developing-AI-sky-drones-create-3D-maps-enemy-lairs-Pentagon-reveals-1m-secretive-autonomous-tactical-airborne-drone-project.html.
123
“Robotics and artificial intelligence are”: Brandon Tseng, email to author, June 17, 2016.
124
“fully automated combat module”: “Kalashnikov Gunmaker Develops Combat Module based on Artificial Intelligence.”
125
more possible positions in go: “AlphaGo,” DeepMind, accessed June 7, 2017, https://deepmind.com/research/alphago/.
125
“Our goal is to beat the best human players”: “AlphaGo: Using Machine Learning to Master the Ancient Game of Go,” Google, January 27, 2016, http://blog.google:443/topics/machine-learning/alphago-machine-learning-game-go/.
126
game 2, on move 37: Daniel Estrada, “Move 37!! Lee Sedol vs AlphaGo Match 2” video, https://www.youtube.com/watch?v=JNrXgpSEEIE.
126
“I thought it was a mistake”: Ibid.
126
“It’s not a human move”: Cade Metz, “The Sadness and Beauty of Watching Google’s AI Play Go,” WIRED, March 11, 2016, https://www.wired.com/2016/03/sadness-beauty-watching-googles-ai-play-go/.
126
1 in 10,000: Cade Metz, “In Two Moves, AlphaGo and Lee Sedol Redefined the Future,” WIRED, accessed June 7, 2017, https://www.wired.com/2016/03/two-moves-alphago-lee-sedol-redefined-future/.
126
“I kind of felt powerless”: Moyer, “How Google’s AlphaGo Beat a Go World Champion.”
126
“AlphaGo isn’t just an ‘expert’ system”: “AlphaGo,” January 27, 2016.
127
AlphaGo Zero: “AlphaGo Zero: Learning from Scratch,” DeepMind, accessed October 22, 2017, https://deepmind.com/blog/alphago=zero=learning=scratch/.
127
neural network to play Atari games: Volodymyr Mnih et al., “Human-Level Control through Deep Reinforcement Learning,” Nature 518, no. 7540 (February 26, 2015): 529–33.
127
deep neural network: JASON, “Perspectives on Research in Artificial Intelligence and Artificial General Intelligence Relevant to DoD.”
129
Inception-v3: Inception-v3 is trained for the Large Scale Visual Recognition Challenge (LSVRC) using the 2012 data. “Image Recognition,” TensorFlow, accessed June 7, 2017, https://www.tensorflow.org/tutorials/image_recognition.
129
one of 1,000 categories: The categories available for Inception-v3 are those from the Large Scale Visual Recognition Challenge (LSVRC) 2012, which is the same as the LSVRC 2014. “ImageNet Large Scale Visual Recognition Competition 2014 (ILSVRC2014),” accessed June 7, 2017, http://image-net.org/challenges/LSVRC/2014/browse-synsets.
129
Pascal Visual Object Classes database: “The PASCAL Visual Object Classes Challenge 2012 (VOC2012),” accessed June 7, 2017, http://host.robots.ox.ac.uk/pascal/VOC/voc2012/index.html.
130
FIRST Robotics Competition: “FIRST Robotics Competition 2016 Season Facts,” FirstInspires.org, http://www.firstinspires.org/sites/default/files/uploads/resource_library/frc-2016-season-facts.pdf.
131
“They can pretty much program in anything”: Charles Dela Cuesta, interview, May 20, 2016.
132
“The stuff that was impressive to me”: Ibid.
133
“I don’t think we’re ever going to give”: Brandon Tseng, email to author, June 17, 2016.
9 Robots Run Amok: Failure in Autonomous Systems
137
March 22, 2003: This account is based on official public records of the fratricide, including an investigation by the UK government in regard to the Tornado fratricide and a Defense Science Board report on both incidents. It is also based, in part, on individuals close to both incidents who asked to remain anonymous as they were not authorized to speak on the record about the incidents. Ministry of Defence, “Aircraft Accident to Royal Air Force Tornado GR MK4A ZG710,” https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/82817/maas03_02_tornado_zg710_22mar03.pdf. Defense Science Board, “Report of the Defense Science Board Task Force on Patriot Performance,” January 2005, http://www.acq.osd.mil/dsb/reports/2000s/ADA435837.pdf.
138
All they had was a radio: Ministry of Defence, “Aircraft Accident to Royal Air Force Tornado GR MK4A ZG710.”
138
attack on a nearby base: “Army: U.S. Soldier Acted Out of Resentment in Grenade Attack,” Associated Press, (March 24, 2003), http://www.foxnews.com/story/2003/03/24/army-us-soldier-acted-out-resentment-in-grenade-attack.html.
138
Tornado GR4A fighter jet: Staff and Agencies, “‘Glaring Failures’ Caused US to Kill RAF Crew,” The Guardian, October 31, 2006, https://www.theguardian.com/u
k/2006/oct/31/military.iraq.
138
It could be because the system simply broke: A UK board of inquiry discounted the possibility that Main and Williams had intentionally turned off the IFF, although without explaining why. In the absence of other possible explanations, it concluded the IFF system “had a fault.” Ministry of Defence, “Aircraft Accident to Royal Air Force Tornado GR MK4A ZG710,” 4–5.
139
Their screen showed a radar-hunting enemy missile: Ministry of Defence, “Aircraft Accident to Royal Air Force Tornado GR MK4A ZG710.”
139
parabolic trajectory: This is an approximation. Lior M. Burko and Richard H. Price, “Ballistic Trajectory: Parabola, Ellipse, or What?,” May 17, 2004, https://arxiv.org/pdf/physics/0310049.pdf.
139
The Tornado’s IFF signal: There are actually two relevant IFF modes of operation that might have prevented the Tornado fratricide, Mode 1 and Mode 4. More 4 is the standard encrypted military IFF used by coalition aircraft in theater. It was tested on the Tornado prior to starting engines on the ground and found functional, but there is no evidence it was broadcasting at any point during the flight. The reasons why are unclear. Mode 1 is an unencrypted mode that all coalition aircraft were supposed to be using as a backup. The UK accident investigation report does not specify whether this mode was broadcasting or not. In any case, the Patriot battery did not have the codes for Mode 1, so they couldn’t have received a Mode 1 signal in any case. This is likely because their equipment was not interoperable and so they were not on the network. The Mode 1 codes would have had to have been delivered by hand to be loaded, and apparently they were not. Ministry of Defence, “Aircraft Accident to Royal Air Force Tornado GR MK4A ZG710.”
139
They had seconds to decide: John K. Hawley, “Looking Back at 20 Years of MANPRINT on Patriot: Observations and Lessons,” Army Research Laboratory, September, 2007, 7, http://www.dtic.mil/docs/citations/ADA472740.
139
Main and Williams’ wingman landed in Kuwait: Stewart Payne, “US Colonel Says Sorry for Tornado Missile Blunder,” March 25, 2003, http://www.telegraph.co.uk/news/worldnews/middleeast/iraq/1425545/US-colonel-says-sorry-for-Tornado-missile-blunder.html.
