by Paul Scharre
182
specific internal structure of the network: Szegedy et al., “Intriguing properties of neural networks.”
182
“huge, weird, alien world of imagery”: Jeff Clune, interview, September 28, 2016.
182
surreptitiously embedded into normal images: Ibid.
183
“hidden exploit”: Ibid.
183
linear methods to interpret data: Ian J. Goodfellow, Jonathan Shlens, and Christian Szegedy, “Explaining and Harnessing Adversarial Examples,” March 20, 2015, https://arxiv.org/abs/1412.6572; Ian Goodfellow, Presentation at Re-Work Deep Learning Summit, 2015, https://www.youtube.com/watch?v=Pq4A2mPCB0Y.
184
“infinitely far to the left”: Jeff Clune, interview, September 28, 2016.
184
“real-world images are a very, very small”: Ibid.
184
present in essentially every deep neural network: “Deep neural networks are easily fooled.” Goodfellow et al., “Explaining and Harnessing Adversarial Examples.”
184
specially evolved noise: Corey Kereliuk, Bob L. Sturm, and Jan Larsen, “Deep Learning and Music Adversaries,” http://www2.imm.dtu.dk/pubdb/views/edoc_download.php/6904/pdf/imm6904.pdf.
184
News-reading trading bots: John Carney, “The Trading Robots Really Are Reading Twitter,” April 23, 2013, http://www.cnbc.com/id/100666302. Patti Domm, “False Rumor of Explosion at White House Causes Stocks to Briefly Plunge; AP Confirms Its Twitter Feed Was Hacked,” April 23, 2013, http://www.cnbc.com/id/100646197.
185
deep neural networks to understand text: Xiang Zhang and Yann LeCun, “Text Understanding from Scratch,” April 4, 2016, https://arxiv.org/pdf/1502.01710v5.pdf.
185
Associated Press Twitter account was hacked: Domm, “False Rumor of Explosion at White House Causes Stocks to Briefly Plunge; AP Confirms Its Twitter Feed Was Hacked.”
186
design deep neural networks that aren’t vulnerable: “Deep neural networks are easily fooled.”
186
“counterintuitive, weird” vulnerability: Jeff Clune, interview, September 28, 2016.
186
“[T]he sheer magnitude, millions or billions”: JASON, “Perspectives on Research in Artificial Intelligence and Artificial General Intelligence Relevant to DoD,” 28–29.
186
“the very nature of [deep neural networks]”: Ibid, 28.
186
“As deep learning gets even more powerful”: Jeff Clune, interview, September 28, 2016.
186
“super complicated and big and weird”: Ibid.
187
“sobering message . . . tragic extremely quickly”: Ibid.
187
“[I]t is not clear that the existing AI paradigm”: JASON, “Perspectives on Research in Artificial Intelligence and Artificial General Intelligence Relevant to DoD,” Ibid, 27.
188
“nonintuitive characteristics”: Szegedy et al., “Intriguing Properties of Neural Networks.”
188
we don’t really understand how it happens: For a readable explanation of this broader problem, see David Berreby, “Artificial Intelligence Is Already Weirdly Inhuman,” Nautilus, August 6, 2015, http://nautil.us/issue/27/dark-matter/artificial-intelligence-is-already-weirdly-inhuman.
12 Failing Deadly: The Risk of Autonomous Weapons
189
“I think that we’re being overly optimistic”: John Borrie, interview, April 12, 2016.
189
“If you’re going to turn these things loose”: John Hawley, interview, December 5, 2016.
189
“[E]ven with our improved knowledge”: Perrow, Normal Accidents, 354.
191
“robo-cannon rampage”: Noah Shachtman, “Inside the Robo-Cannon Rampage (Updated),” WIRED, October 19, 2007, https://www.wired.com/2007/10/inside-the-robo/.
191
bad luck, not deliberate targeting: “ ‘Robotic Rampage’ Unlikely Reason for Deaths,” New Scientist, accessed June 12, 2017, https://www.newscientist.com/article/dn12812-robotic-rampage-unlikely-reason-for-deaths/.
191
35 mm rounds into a neighboring gun position: “Robot Cannon Kills 9, Wounds 14,” WIRED, accessed June 12, 2017, https://www.wired.com/2007/10/robot-cannon-ki/.
191
“The machine doesn’t know it’s making a mistake”: John Hawley, interview, December 5, 2016.
193
“incidents of mass lethality”: John Borrie, interview, April 12, 2016.
193
“If you put someone else”: John Hawley, interview, December 5, 2016.
194
“I don’t have a lot of good answers for that”: Peter Galluch, interview, July 15, 2016.
13 Bot vs. Bot: An Arms Race in Speed
199
May 6, 2010: U.S. Commodity Futures Trading Commission and U.S. Securities and Exchange Commission, “Findings Regarding the Market Events of May 6, 2010,” September 30, 2010), 2, http://www.sec.gov/news/studies/2010/marketevents-report.pdf.
199
“horrifying” and “absolute chaos”: Tom Lauricella and Peter A. McKay, “Dow Takes a Harrowing 1,010.14-Point Trip,” Wall Street Journal, May 7, 2010, http://www.wsj.com/articles/SB10001424052748704370704575227754131412596. Alexandra Twin, “Glitches Send Dow on Wild Ride,” CNN Money, May 6, 2010, http://money.cnn.com/2010/05/06/markets/markets_newyork/.
200
Gone are the days of floor trading: D. M. Levine, “A Day in the Quiet Life of a NYSE Floor Trader,” Fortune, May 20, 2013, http://fortune.com/2013/05/29/a-day-in-the-quiet-life-of-a-nyse-floor-trader/.
200
three-quarters of all trades: “Rocky Markets Test the Rise of Amateur ‘Algo’ Traders,” Reuters, January 28, 2016, http://www.reuters.com/article/us-europe-stocks-algos-insight-idUSKCN0V61T6. The percentage of trades that are automated has varied over time and is subject to some uncertainty. Estimates for the past several years have ranged from 50 percent to 80 percent of all U.S. equity trades. WashingtonsBlog, “What Percentage of U.S. Equity Trades Are High Frequency Trades?,” October 28, 2010, http://www.washingtonsblog.com/2010/10/what-percentage-of-u-s-equity-trades-are-high-frequency-trades.html.
200
sometimes called algorithmic trading: Tom C. W. Lin, “The New Investor,” SSRN Scholarly Paper (Rochester, NY: Social Science Research Network, March 3, 2013), https://papers.ssrn.com/abstract=2227498.
200
automated trading decisions to buy or sell: Some writers on automated stock trading differentiate between automated trading and algorithmic trading, using the term algorithmic trading only to refer to the practice of breaking up large orders to execute via algorithm by price, time, or volume, and referring to other practices such as seeking arbitrage opportunities as automated trading. Others treat algorithmic trading and automated trading as effectively synonymous.
200
Automated trading offers the advantage: Shobhit Seth, “Basics of Algorithmic Trading: Concepts and Examples,” Investopedia, October 10, 2014, http://www.investopedia.com/articles/active-trading/101014/basics-algorithmic-trading-concepts-and-examples.asp.
201
blink of an eye: “Average Duration of a Single Eye Blink—Human Homo Sapiens—BNID 100706,” accessed June 12, 2017, http://bionumbers.hms.harvard.edu//bionumber.aspx?id=100706&ver=0.
201
speeds measured in microseconds: Michael Lewis, Flash Boys: A Wall Street Revolt (New York: W. W. Norton, 2015), 63, 69, 74, 81.
201
shortest route for their cables: Ibid, 62–63.
201
optimizing every part of their hardware for speed: Ibid, 63–64.
201
test them against actual stock market data: D7, “Knightmare: A DevOps Cautionary Tale,” Doug Seven, April 17, 2014, https://dougseve
n.com/2014/04/17/knightmare-a-devops-cautionary-tale/.
202
“The Science of Trading, the Standard of Trust”: Jeff Cox, “ ‘Knight-Mare’: Trading Glitches May Just Get Worse,” August 2, 2012, http://www.cnbc.com/id/48464725.
202
Knight’s trading system began flooding the market: “Knight Shows How to Lose $440 Million in 30 Minutes,” Bloomberg.com, August 2, 2012, https://www.bloomberg.com/news/articles/2012-08-02/knight-shows-how-to-lose-440-million-in-30-minutes.
202
neglected to install a “kill switch”: D7, “Knightmare.”
202
executed 4 million trades: “How the Robots Lost: High-Frequency Trading’s Rise and Fall,” Bloomberg.com, June 7, 2013, https://www.bloomberg.com/news/articles/2013-06-06/how-the-robots-lost-high-frequency-tradings-rise-and-fall.
202
Knight was bankrupt: D7, “Knightmare.”
202
“Knightmare on Wall Street”: For a theory on what happened, see Nanex Research, “03-Aug-2012—The Knightmare Explained,” http://www.nanex.net/aqck2/3525.html.
203
Waddell & Reed: Waddell & Reed was not named in the official SEC and CFTC report, which referred only to a “large fundamental trader (a mutual fund complex).” U.S. Commodity Futures Trading Commission and U.S. Securities and Exchange Commission, “Findings Regarding the Market Events of May 6, 2010,” 2. However, multiple news sources later identified the firm as Waddell & Reed Financial Inc. E. S. Browning and Jenny Strasburg, “The Mutual Fund in the ‘Flash Crash,’ ” Wall Street Journal, October 6, 2010, http://www.wsj.com/articles/SB10001424052748704689804575536513798579500. “Flash-Crash Spotlight on Kansas Manager Avery,” Reuters, May 18, 2010, http://www.reuters.com/article/us-selloff-ivyasset-analysis-idUSTRE64H6G620100518.
203
E-minis: “What Are Emini Futures? Why Trade Emini Futures?,” Emini-Watch.com, accessed June 13, 2017, http://emini-watch.com/emini-trading/emini-futures/.
203
9 percent of the trading volume over the previous minute: U.S. Commodity Futures Trading Commission and U.S. Securities and Exchange Commission, “Findings Regarding the Market Events of May 6, 2010,” 2.
203
no instructions with regard to time or price: Both Waddell & Reed and the Chicago Mercantile Exchange contest the argument, made principally by the CFTF & SEC, that the large E-mini sale was a factor in precipitating the Flash Crash. The independent market research firm Nanex has similarly criticized the suggestion that the Waddell & Reed sell algorithm led to the crash. “Waddell & Reed Responds to ‘Flash Crash’ Reports—May. 14, 2010,” accessed June 13, 2017, http://money.cnn.com/2010/05/14/markets/flash_crash_waddell_reed/. “CME Group Statement on the Joint CFTC/SEC Report Regarding the Events of May 6 - CME Investor Relations,” October 1, 2010, http://investor.cmegroup.com/investor-relations/releasedetail.cfm?ReleaseID=513388. “Nanex - Flash Crash Analysis - Continuing Developments,” accessed June 13, 2017, http://www.nanex.net/FlashCrashFinal/FlashCrashAnalysis_WR_Update.html.
203
“unusually turbulent”: U.S. Commodity Futures Trading Commission and U.S. Securities and Exchange Commission, “Findings Regarding the Market Events of May 6, 2010,” 1.
203
“unusually high volatility”: Ibid, 2.
204
“hot potato” effect: Ibid, 3.
204
27,000 E-mini contracts: Ibid, 3.
204
automated “stop logic” safety: Ibid, 4.
204
“irrational prices”: Ibid, 5.
204
“clearly erroneous” trades: Ibid, 6.
205
Michael Eisen: Michael Eisen, “Amazon’s $23,698,655.93 book about flies,” it is NOT junk, April 22, 2011, http://www.michaeleisen.org/blog/?p=358.
205
Eisen hypothesized: “Amazon.com At a Glance: bordeebook,” accessed on October 1, 2016.
206
Navinder Singh Sarao: Department of Justice, “Futures Trader Pleads Guilty to Illegally Manipulating the Futures Market in Connection With 2010 ‘Flash Crash,’ ” November 9, 2016, https://www.justice.gov/opa/pr/futures-trader-pleads-guilty-illegally-manipulating-futures-market-connection-2010-flash.
206
By deliberately manipulating the price: Department of Justice, “Futures Trader Charged with Illegally Manipulating Stock Market, Contributing to the May 2010 Market ‘Flash Crash,’ ” April 21, 2015, https://www.justice.gov/opa/pr/futures-trader-charged-illegally-manipulating-stock-market-contributing-may-2010-market-flash. United States District Court Northern District of Illinois Eastern Division, “Criminal Complaint: United States of America v. Navinder Singh Sarao,” February 11, 2015, https://www.justice.gov/sites/default/files/opa/press-releases/attachments/2015/04/21/sarao_criminal_complaint.pdf.
206
pin the blame for the Flash Crash: “Post Flash Crash, Regulators Still Use Bicycles To Catch Ferraris,” Traders Magazine Online News, accessed June 13, 2017, http://www.tradersmagazine.com/news/technology/post-flash-crash-regulators-still-use-bicycles-to-catch-ferraris-113762-1.html.
206
spoofing algorithm was reportedly turned off: “Guy Trading at Home Caused the Flash Crash,” Bloomberg.com, April 21, 2015, https://www.bloomberg.com/view/articles/2015-04-21/guy-trading-at-home -caused-the-flash-crash.
206
exacerbated instability in the E-mini market: Department of Justice, “Futures Trader Charged with Illegally Manipulating Stock Market, Contributing to the May 2010 Market ‘Flash Crash,’ ” April 21, 2015, https://www.justice.gov/opa/pr/futures-trader-charged-illegally-manipulating-stock-market-contributing-may-2010-market-flash.
206
“circuit breakers”: The first tranche of individual stock circuit breakers, implemented in the immediate aftermath of the Flash Crash, initiated a five-minute pause if a stock’s price moved up or down more than 10 percent in the preceding five minutes. U.S. Commodity Futures Trading Commission and U.S. Securities and Exchange Commission, “Findings Regarding the Market Events of May 6, 2010,” 7.
206
Market-wide circuit breakers: A 7 percent or 13 percent drop halts trading for 15 minutes. A 20 percent drop stops trading for the rest of the day.
206
“limit up–limit down”: Securities and Exchange Commission, “Investor Bulletin: Measures to Address Market Volatility,” July 1, 2012, https://www.sec.gov/oiea/investor-alerts-bulletins/investor-alerts-circuitbreakersbulletinhtm.html.Specific price bands are listed here: Investopedia Staff, “Circuit Breaker,” Investopedia, November 18, 2003, http://www.investopedia.com/terms/c/circuitbreaker.asp.
207
An average day sees a handful of circuit breakers tripped: Maureen Farrell, “Mini Flash Crashes: A Dozen a Day,” CNNMoney, March 20, 2013, http://money.cnn.com/2013/03/20/investing/mini-flash-crash/index.html.
207
over 1,200 circuit breakers: Matt Egan, “Trading Was Halted 1,200 Times Monday,” CNNMoney, August 24, 2015, http://money.cnn.com/2015/08/24/investing/stocks-markets-selloff-circuit-breakers-1200-times/index.html. Todd C. Frankel, “Mini Flash Crash? Trading Anomalies on Manic Monday Hit Small Investors,” Washington Post, August 26, 2015, https://www.washingtonpost.com/business/economy/mini-flash-crash-trading-anomalies-on-manic-monday-hit-small-investors/2015/08/26/6bdc57b0-4c22-11e5-bfb9-9736d04fc8e4_story.html?utm_term=.749eb0bbbf5b.
207
simple human error: Joshua Jamerson and Aruna Viswanatha, “Merrill Lynch to Pay $12.5 Million Fine for Mini-Flash Crashes,” Wall Street Journal, September 26, 2016, http://www.wsj.com/articles/merrill-lynch-to-pay-12-5-million-fine-for-mini-flash-crashes-1474906677.
207
underlying conditions for flash crashes remain: Bob Pisani, “A Year after the 1,000-Point Plunge, Changes Cut Trading Halts on Volatile Days,” August 23, 2016, http://www.cnbc.com/2016/08/23/a-year-after-the-1000-point-plunge-changes-cut-trading
-halts-on-volatile-days.html.
207
“Circuit breakers don’t prevent”: Farrell, “Mini Flash Crashes.”
208
Gulf of Tonkin incident: Historians have since argued that there was ample evidence at the time that the purported naval battle on August 4, 1964, never happened, that Secretary of Defense Robert McNamara deliberately misled Congress in order to push for war in Vietnam, and that President Lyndon Johnson was aware that the battle never took place. Pat Paterson, “The Truth About Tonkin,” Naval History magazine, February 2008, https://www.usni.org/magazines/navalhistory/2008-02/truth-about-tonkin. John Prados, “The Gulf of Tonkin Incident, 40 Years Later,” The National Security Archive, August 4, 2004, http://nsarchive.gwu.edu/NSAEBB/NSAEBB132/. John Prados, “Tonkin Gulf Intelligence ‘Skewed’ According to Official History and Intercepts,” The National Security Archive, December 1, 2005. Robert J. Hanyok, “Skunks, Bogies, Silent Hounds, and the Flying Fish: The Gulf of Tonkin Mystery, 2–4 August 1964,” Cryptologic Quarterly, http://nsarchive.gwu.edu/NSAEBB/NSAEBB132/relea00012.pdf.
208
an “act of war”: Dan Gettinger, “ ‘An Act of War’: Drones Are Testing China-Japan Relations,” Center for the Study of the Drone, November 8, 2013, http://dronecenter.bard.edu/act-war-drones-testing-china-japan-relations/. “Japan to Shoot down Foreign Drones That Invade Its Airspace,” The Japan Times Online, October 20, 2013, http://www.japantimes.co.jp/news/2013/10/20/national/japan-to-shoot-down-foreign-drones-that-invade-its-airspace/. “China Warns Japan against Shooting down Drones over Islands,” The Times of India, October 27, 2013, http://timesofindia.indiatimes.com/world/china/China-warns-Japan-against-shooting-down-drones-over-islands/articleshow/24779422.cms.
208
broach other nations’ sovereignty: “Kashmir Firing: Five Civilians Killed after Drone Downed,” BBC News, July 16, 2015, sec. India, http://www.bbc.com/news/world-asia-india-33546468.
208
Pakistan shot down: “Pakistan Shoots Down Indian Drone ‘Trespassing’ into Kashmir,” AP News, accessed June 13, 2017, https://apnews.com/b67d689fb1f7410e9d0be01daaded3a7/pakistani-and-indian-troops-trade-fire-kashmir.
208
Israel has shot down drones: Gili Cohen, “Israeli Fighter Jet Shoots Down Hamas Drone Over Gaza,” Haaretz, September 20, 2016, http://www.haaretz.com/israel-news/1.743169.
