by Paul Scharre
258
“feasible” precautions: Anderson et al., “Adapting the Law of Armed Conflict to Autonomous Weapon Systems,” 403–405.
259
Lieber Code: “Article 71, General Orders No. 100: The Lieber Code,” The Avalon Project, http://avalon.law.yale.edu/19th_century/lieber.asp#art71.
259
“has been rendered unconscious”: “Article 41: Safeguard of an Enemy Hors de Combat,” Protocol Additional to the Geneva Conventions of 12 August 1949 (Protocol I).
259
“perfidy”: “Practice Relating to Rule 65: Perfidy,” Customary IHL, https://ihl-databases.icrc.org/customary-ihl/eng/docs/v2_rul_rule65.
261
“You’ve just been disarmed”: John S. Canning, “’You’ve just been disarmed. Have a nice day!’” IEEE Technology and Society Magazine (Spring 2009), 12–15.
261
“targeting either the bow or the arrow”: John Canning, interview, December 6, 2016.
261
ultra-precise weapons that would disarm: John S. Canning, “Weaponized Unmanned Systems: A Transformational Warfighting Opportunity, Government Roles in Making It Happen,” http://www.sevenhorizons.org/docs/CanningWeaponizedunmannedsystems.pdf
261
“let the machines target machines”: John S. Canning, “A Concept of Operations for Armed Autonomous Systems,” presentation, http://www.dtic.mil/ndia/2006disruptive_tech/canning.pdf.
261
“accountability gap”: Bonnie Docherty, “Mind the Gap: The Lack of Accountability for Killer Robots,” Human Rights Watch, April 9, 2015, https://www.hrw.org/report/2015/04/09/mind-gap/lack-accountability-killer-robots.
261
“fair nor legally viable”: Bonnie Docherty, interview, November 18, 2016.
261
“‘punishing’ the robot”: Docherty, “Mind the Gap.”
262
generally shielded from civil liability: Docherty notes: “Immunity for the US military and its defense contractors presents an almost insurmountable hurdle to civil accountability for users or producers of fully autonomous weapons. The military is immune from lawsuits related to: (1) its policy determinations, which would likely include a choice of weapons, (2) the wartime combat activities of military forces, and (3) acts committed in a foreign country. Manufacturers contracted by the military are similarly immune from suit when they design a weapon in accordance with government specifications and without deliberately misleading the military. These same manufacturers are also immune from civil claims relating to acts committed during wartime.” Ibid.
262
“dangerous combination”: Bonnie Docherty, interview, November 18, 2016.
262
“retributive justice”: Ibid.
262
“eliminate this accountability gap”: Docherty, “Mind the Gap.”
262
shootdown was a mistake: Rebecca Crootof, “War Torts: Accountability for Autonomous Weapons,” University of Pennsylvania Law Review 164, no. 6 (May 2016): 1347-1402, http://scholarship.law.upenn.edu/cgi/viewcontent.cgi?article=9528&context=penn_law_review.
262
U.S. government paid $61.8 million: Crootof, “War Torts.”
262
“resonates with everyone”: Bonnie Docherty, interview, November 18, 2016.
262
must be an individual to hold accountable: Charles Dunlap Jr., “Accountability and Autonomous Weapons: Much Ado About Nothing?” Temple International and Comparative Law Journal 30 (1), Spring 2016, 65–66.
263
“issue is not with autonomous weapons”: Ibid.
263
“In cases not covered by the law”: Protocol Additional to the Geneva Conventions of 12 August 1949, and relating to the Protection of Victims of Non-International Armed Conflicts (Protocol II), June 8, 1977, https://ihl-databases.icrc.org/applic/ihl/ihl.nsf/7c4d08d9b287a42141256739003e636b/d67c3971bcff1c10c125641e0052b545.
264
“There is no accepted interpretation”: Rupert Ticehurst, “The Martens Clause and the Laws of Armed Conflict,” International Review of the Red Cross 317 (April 30, 1997), https://www.icrc.org/eng/resources/documents/article/other/57jnhy.htm.
264
“priming”: Cengiz Erisen, Milton Lodge and Charles S. Taber, “Affective Contagion in Effortful Political Thinking,” Political Psychology 35, no. 2 (April 2014): 187–206.
264
Carpenter found: Charli Carpenter, “How Do Americans Feel About Fully Autonomous Weapons?” Duck of Minerva, June 19, 2013, http://duckofminerva.com/2013/06/how-do-americans-feel-about-fully-autonomous-weapons.html.
265
sharp arrow in the quiver of ban advocates: “Q&A on Fully Autonomous Weapons,” Human Rights Watch, October 21, 2013, https://www.hrw.org/news/2013/10/21/qa-fully-autonomous-weapons.
265
“it is too early to argue that”: Michael C. Horowitz, “Public Opinion and the Politics of the Killer Robots Debate,” February 16, 2016, http://rap.sagepub.com/content/3/1/2053168015627183.
265
“’Conscience’ has an explicitly moral inflection”: Peter Asaro, “Jus nascendi: Robotic Weapons and the Martens Clause,” http://www.peterasaro.org/writing/Asaro%20Jus%20Nascendi%20PROOF.pdf.
265
“disservice to reduce the ‘dictates of public conscience’”: Ibid.
265
“through public discussion, as well as academic scholarship”: Ibid.
265
“The bar for claiming to speak for humanity”: Horowitz, “Public Opinion and the Politics of the Killer Robots Debate.”
266
“the clearest manifestation of”: Steve Goose, email to author, November 22, 2016.
266
“emphasize effects rather than weapons”: Charles J. Dunlap et al., “Guest Post: To Ban New Weapons or Regulate Their Use?,” Just Security, April 3, 2015, https://www.justsecurity.org/21766/guest-post-ban-weapons-regulate-use/.
266
modern-day CS gas: Ibid.
267
legal for military use against civilians: “Riot Control Agents,” Organization for the Prohibition of Chemical Weapons, accessed June 16, 2017, https://www.opcw.org/about-chemical-weapons/types-of-chemical-agent/riot-control-agents/.
267
“smart mines”: Dunlap et al., “Guest Post.”
267
“the paradox that requires”: Ibid.
267
“Given the pace of accelerated scientific development”: Dunlap et al., “Is It Really Better to Be Dead than Blind?”
267
“strict compliance with the core principles”: Dunlap et al., “Guest Post.”
268
“even though there are no victims”: Bonnie Docherty, interview, November 18, 2016.
268
“grave concern”: Steve Goose, interview, October 26, 2016.
268
Protocol II: Protocol on Prohibitions or Restrictions on the Use of Mines, Booby-Traps and Other Devices as amended on 3 May 1996 (Protocol II to the 1980 CCW Convention as amended on 3 May 1996), https://ihl-databases.icrc.org/ihl/INTRO/575.
268
“The dangers just far outweigh”: Steve Goose, interview, October 26, 2016.
268
“where you stand depends on where you sit”: Rufus E. Miles, Jr., “The Origin and Meaning of Miles’ Law” Public Administration Review 38, no. 5 (September/October, 1978): 399-403. https://www.jstor.org/stable/975497?seq=1#page_scan_tab_contents.
268
“Denying such capabilities to nations”: Dunlap et al., “Guest Post.”
269
“The law of war rules on conducting attacks”: Department of Defense, “Department of Defense Law of War Manual,” 330.
269
“acts of violence against the adversary”: “Article 49: Definition of Attacks and Scope of Application,” Protocol Additional to the Geneva Conventions of 12 August 1949, and r
elating to the Protection of Victims of International Armed Conflicts (Protocol I), June 8, 1977, https://ihl-databases.icrc.org/ihl/WebART/470-750062?OpenDocument.
269
“the size of something that constitutes an attack”: Kenneth Anderson, interview, January 6, 2017.
270
“is a technical term relating to”: International Committee of the Red Cross, “Commentary of 1987 Protection of the Civilian Population,” https://ihl-databases.icrc.org/applic/ihl/ihl.nsf/Comment.xsp?action=openDocument&documentId=2C8494C2FCAF8B27C12563CD0043AA67.
270
“The notion of the launching of an attack”: Kenneth Anderson, interview, January 6, 2017.
17 Soulless Killers: The Morality of Autonomous Weapons
271
“morally reprehensible”: Jody Williams, interview, October 27, 2016.
272
combatants would have an ethical responsibility: Kenneth Anderson and Matthew Waxman, “Law and Ethics for Autonomous Weapon Systems Why a Ban Won’t Work and How the Laws of War Can,” Hoover Institution, http://media.hoover.org/sites/default/files/documents/Anderson-Waxman_LawAndEthics_r2_FINAL.pdf, 21-22.
272
detailed policy guidance: Department of Defense, “Department of Defense Directive Number 3000.09.”
273
“naked soldier”: Michael Walzer, Just and Unjust Wars: A Moral Argument With Historical Illustrations, 4th ed. (New York: Basic Books, 1977), 138–142.
273
“It is not against the rules of war”: Ibid, 142.
274
“War is cruelty”: “William Tecumseh Sherman,” Biography.com, https://www.biography.com/people/william-tecumseh-sherman-9482051.
274
sergeant chastised the soldiers: Walzer, Just and Unjust Wars, 141.
274
“He got him, but”: Ibid, 140.
274
They are the exception: It is also worth noting that this concern about the role of mercy only applies to antipersonnel autonomous weapons. All of the examples of these moments of mercy are ones where a single enemy individual is targeted. They do not appear to arise in situations where soldiers are targeting objects, even those that have people in them such as ships or tanks. Anti-vehicle or anti-material weapons, therefore, would not run afoul of this concern.
275
“posturing”: Dave Grossman, On Killing: The Psychological Cost of Learning to Kill in War and Society (Boston: Little, Brown and Company, 1996), 3–4.
275
evidence from a variety of wars: Ibid, 9–28.
275
innate biological resistance: Ibid, 177–178.
275
one animal submits first: Ibid, 5–6.
276
drone crews: Kelly Faircloth, “Everyone Names Their Roomba. What Would You Name Yours?,” http://jezebel.com/everyone-names-their-roomba-what-would-you-name-yours. James Dao, “Drone Pilots Are Found to Get Stress Disorders Much as Those in Combat Do,” New York Times, February 22, 2013, https://www.nytimes.com/2013/02/23/us/drone-pilots-found-to-get-stress-disorders-much-as-those-in-combat-do.html. Rebecca Hawkes, “Post-Traumatic Stress Disorder Is Higher in Drone Operators,” The Telegraph, May 30, 2015, http://www.telegraph.co.uk/culture/hay-festival/11639746/Post-traumatic-stress-disorder-is-higher-in-drone-operators.html. “Can Drone Pilots Be Diagnosed With Post-Traumatic Stress Disorder?,” NPR.org, accessed June 16, 2017, http://www.npr.org/2015/06/06/412525635/can-drone-pilots-be-diagnosed-with-post-traumatic-stress-disorder. Ed Pilkington, “Life as a Drone Operator: ‘Ever Step on Ants and Never Give It Another Thought?,’ ” The Guardian, November 19, 2015, https://www.theguardian.com/world/2015/nov/18/life-as-a-drone-pilot-creech-air-force-base-nevada.
276
firing rates: Grossman, On Killing, 153.
277
“if he can get others to share”: Another psychological factor that contributed to higher firing rates among machine gun crews, Grossman argues, was mutual accountability for their actions among the teammates. If one of them wasn’t performing his job properly, the others could immediately tell. Ibid, 149–154.
277
“We see some really dangerous behaviors”: Mary “Missy” Cummings, interview, June 1, 2016.
277
“[P]hysical and emotional distancing”: M. L. Cummings, “Creating Moral Buffers in Weapon Control Interface Design,” IEEE Technology and Society Magazine (Fall 2004), 29–30.
278
“[It] is more palatable”: Ibid, 31.
278
name their Roomba: Celeste Biever, “My Roomba’s Name Is Roswell,” Slate, March 23, 2014, http://www.slate.com/articles/health_and_science/new_scientist/2014/03/roomba_vacuum_cleaners_have_names_irobot_ceo_on_people_s_ties_to_robots.html.
278
“It is possible that without consciously”: Cummings, “Creating Moral Buffers in Weapon Control Interface Design,” 31.
278
“could permit people to perceive”: Ibid, 32.
278
“cheerful, almost funny graphic”: Ibid, 32–33.
278
Cummings criticized the Army’s decision: Cummings uses the terms “management by exception” to refer to a human-supervised autonomous control mode and “management by consent” to refer to a semi-autonomous control mode. Ibid, 33.
278
“[E]nabling a system to essentially fire at will”: Ibid, 33.
278
take a positive action before the weapon fires: Ibid, 33.
279
changed its marksmanship training: Grossman, On Killing, 35.
279
U.S. aerial firebombing killed: Robert S. McNamara in Fog of War: Eleven Lessons From the Life of Robert S. McNamara, documentary, directed by Errol Morris (2003; Sony Pictures Classics).
279
“were behaving as war criminals”: Ibid.
279
“the television coverage was starting”: Colin L. Powell with Joseph E. Persico, My American Journey (New York: Ballantine Books, 2003).
279
The dehumanization that enables killing: Grossman, On Killing, 156–164.
279
war crimes: Ibid, 210–211.
280
mental health surveys of deployed U.S. troops: Office of the Surgeon, Multi-National Force Iraq and Office of the Surgeon General, United States Army Medical Command, “Mental Health Advisory Team (MHAT) IV, Operation Iraqi Freedom 05-07, Final Report,” November 17, 2006, http://armymedicine.mil/Documents/MHAT-IV-Report-17NOV06-Full-Report.pdf, 34-42. Office of the Surgeon, Multi-National Force Iraq and Office of the Surgeon General, United States Army Medical Command, “Mental Health Advisory Team (MHAT) V, Operation Iraqi Freedom 06-08, Final Report,” February 14, 2008, http://armymedicine.mil/Documents/Redacted1-MHATV-OIF-4-FEB-2008Report.pdf, 30–32.
280
“What if [robotics] actually works?”: Ron Arkin, interview, June 8, 2016.
281
“I don’t care about the robots”: Ibid.
281
“ethical governor”: Ronald C. Arkin, “Governing Lethal Behavior: Embedding Ethics in a Hybrid Deliberative/Reactive Robot Architecture,” Technical Report GIT-GVU-07-11, http://www.cc.gatech.edu/ai/robot-lab/online-publications/formalizationv35.pdf.
281
“moral imperative to use this”: Ron Arkin, interview, June 8, 2016.
281
A typical bomb had only: “Accuracy and Employment of Air-Dropped Guided Munitions by the United States,” Center for a New American Security, https://s3.amazonaws.com/files.cnas.org/images/Accuracy-and-employment-air-dropped-guided-munitions.jpg.
281
More than 9,000 bombs: Richard P. Hallion, “Precision Guided Munitions and the New Era of Warfare,” APSC Paper Number 53, Air Power Studies Centre, http://fas.org/man/dod-101/sys/smart/docs/paper53.htm.
281
accurate to within five feet: “Accuracy and Employment of Air-Dropp
ed Guided Munitions by the United States.”
282
U.S. drone strikes: “Drone Wars: The Full Data,” The Bureau of Investigative Journalism, accessed June 16, 2017, https://www.thebureauinvestigates.com/stories/2017-01-01/drone-wars-the-full-data.
282
“the use of indiscriminate rockets”: “Ukraine: Unguided Rockets Killing Civilians,” Human Rights Watch, July 24, 2014, https://www.hrw.org/news/2014/07/24/ukraine-unguided-rockets-killing-civilians.
282
“some of it is quite dishonorable”:, Ron Arkin, interview June 8, 2016.
282
“utterly and wholly unacceptable”: Ronald Arkin, “The Case for Banning Killer Robots: Counterpoint,” accessed June 16, 2017, https://cacm.acm.org/magazines/2015/12/194632-the-case-for-banning-killer-robots/abstract.
283
“software safety”: Ron Arkin, interview, June 8, 2016.
283
“back in some general’s office”: Ibid.
283
“There is no doubt in my mind”: Jody Williams, interview, October 27, 2016.
283
“Should we create caged tigers”: Ron Arkin, interview, June 8, 2016.
284
“Where does the danger lurk?”: Ibid.
284
“we need to do the research”: Ibid.
284
“could possibly support”: Ibid.
285
“utilitarian, consequentialist”: Ibid.
285
“fundamentally inhuman”: Jody Williams, interview, October 27, 2016.
285
“fundamental question of whether it’s appropriate”: Peter Asaro, interview, December 19, 2016.
286
“What decisions require uniquely human judgment?”: Kenneth Anderson, email to author, January 4, 2016.
287
if a decision is made to take a human life: Duncan Purves, Ryan Jenkins & Bradley J. Strawser, “Autonomous Machines, Moral Judgment, and Acting for the Right Reasons” Ethical Theory and Moral Practice 18, no. 4 (2015): 851–872.
287
“the most fundamental and salient moral question”: Peter Asaro, interview, December 19, 2016.
287
Heyns called on states to declare: Christof Heyns, “Report of the Special Rapporteur on extrajudicial, summary or arbitrary executions,” United Nations Human Rights Council, April 9, 2013, http://www.ohchr.org/Documents/HRBodies/HRCouncil/RegularSession/Session23/A-HRC-23-47_en.pdf.
