The Glass Cage: Automation and Us

by Nicholas Carr

  40. Mindell, Digital Apollo, 21.

  41. J. O. Roberts, “ ‘The Case against Automation in Manned Fighter Aircraft,” SETP Quarterly Review 2, no. 3 (Fall 1957): 18–23.

  42. Quoted in Mindell, Between Human and Machine, 77.

  43. Harris, Human Performance on the Flight Deck, 221.

  Chapter Four: THE DEGENERATION EFFECT

  1. Alfred North Whitehead, An Introduction to Mathematics (New York: Henry Holt, 1911), 61.

  2. Quoted in Frank Levy and Richard J. Murnane, The New Division of Labor: How Computers Are Creating the Next Job Market (Princeton: Princeton University Press, 2004), 4.

  3. Raja Parasuraman et al., “Model for Types and Levels of Human Interaction with Automation,” IEEE Transactions on Systems, Man, and Cybernetics—Part A: Systems and Humans 30, no. 3 (2000): 286–297. See also Nadine Sarter et al., “Automation Surprises,” in Gavriel Salvendy, ed., Handbook of Human Factors and Ergonomics, 2nd ed. (New York: Wiley, 1997).

  4. Dennis F. Galletta et al., “Does Spell-Checking Software Need a Warning Label?,” Communications of the ACM 48, no. 7 (2005): 82–86.

  5. National Transportation Safety Board, Marine Accident Report: Grounding of the Panamanian Passenger Ship Royal Majesty on Rose and Crown Shoal near Nantucket, Massachusetts, June 10, 1995 (Washington, D.C.: NTSB, April 2, 1997).

  6. Sherry Turkle, Simulation and Its Discontents (Cambridge, Mass.: MIT Press, 2009), 55–56.

  7. Jennifer Langston, “GPS Routed Bus under Bridge, Company Says,” Seattle Post-Intelligencer, April 17, 2008.

  8. A. A. Povyakalo et al., “How to Discriminate between Computer-Aided and Computer-Hindered Decisions: A Case Study in Mammography,” Medical Decision Making 33, no. 1 (January 2013): 98–107.

  9. E. Alberdi et al., “Why Are People’s Decisions Sometimes Worse with Computer Support?,” in Bettina Buth et al., eds., Proceedings of SAFECOMP 2009, the 28th International Conference on Computer Safety, Reliability, and Security (Hamburg, Germany: Springer, 2009), 18–31.

  10. See Raja Parasuraman et al., “Performance Consequences of Automation-Induced ‘Complacency,’ ” International Journal of Aviation Psychology 3, no. 1 (1993): 1–23.

  11. Raja Parasuraman and Dietrich H. Manzey, “Complacency and Bias in Human Use of Automation: An Attentional Integration,” Human Factors 52, no. 3 (June 2010): 381–410.

  12. Norman J. Slamecka and Peter Graf, “The Generation Effect: Delineation of a Phenomenon,” Journal of Experimental Psychology: Human Learning and Memory 4, no. 6 (1978): 592–604.

  13. Jeffrey D. Karpicke and Janell R. Blunt, “Retrieval Practice Produces More Learning than Elaborative Studying with Concept Mapping,” Science 331 (2011): 772–775.

  14. Britte Haugan Cheng, “Generation in the Knowledge Integration Classroom” (PhD thesis, University of California, Berkeley, 2008).

  15. Simon Farrell and Stephan Lewandowsky, “A Connectionist Model of Complacency and Adaptive Recovery under Automation,” Journal of Experimental Psychology: Learning, Memory, and Cognition 26, no. 2 (2000): 395–410.

  16. I first discussed van Nimwegen’s work in my book The Shallows: What the Internet Is Doing to Our Brains (New York: W. W. Norton, 2010), 214–216.

  17. Christof van Nimwegen, “The Paradox of the Guided User: Assistance Can Be Counter-effective” (SIKS Dissertation Series No. 2008-09, Utrecht University, March 31, 2008). See also Christof van Nimwegen and Herre van Oostendorp, “The Questionable Impact of an Assisting Interface on Performance in Transfer Situations,” International Journal of Industrial Ergonomics 39, no. 3 (May 2009): 501–508; and Daniel Burgos and Christof van Nimwegen, “Games-Based Learning, Destination Feedback and Adaptation: A Case Study of an Educational Planning Simulation,” in Thomas Connolly et al., eds., Games-Based Learning Advancements for Multi-Sensory Human Computer Interfaces: Techniques and Effective Practices (Hershey, Penn.: IGI Global, 2009), 119–130.

  18. Carlin Dowling et al., “Audit Support System Design and the Declarative Knowledge of Long-Term Users,” Journal of Emerging Technologies in Accounting 5, no. 1 (December 2008): 99–108.

  19. See Richard G. Brody et al., “The Effect of a Computerized Decision Aid on the Development of Knowledge,” Journal of Business and Psychology 18, no. 2 (2003): 157–174; and Holli McCall et al., “Use of Knowledge Management Systems and the Impact on the Acquisition of Explicit Knowledge,” Journal of Information Systems 22, no. 2 (2008): 77–101.

  20. Amar Bhidé, “The Judgment Deficit,” Harvard Business Review 88, no. 9 (September 2010): 44–53.

  21. Gordon Baxter and John Cartlidge, “Flying by the Seat of Their Pants: What Can High Frequency Trading Learn from Aviation?,” in G. Brat et al., eds., ATACCS-2013: Proceedings of the 3rd International Conference on Application and Theory of Automation in Command and Control Systems (New York: ACM, 2013), 64–73.

  22. Vivek Haldar, “Sharp Tools, Dull Minds,” This Is the Blog of Vivek Haldar, November 10, 2013, blog.vivekhaldar.com/post/66660163006/sharp-tools-dull-minds.

  23. Tim Adams, “Google and the Future of Search: Amit Singhal and the Knowledge Graph,” Observer, January 19, 2013.

  24. Betsy Sparrow et al., “Google Effects on Memory: Cognitive Consequences of Having Information at Our Fingertips,” Science 333, no. 6043 (August 5, 2011): 776–778. Another study suggests that simply knowing an experience has been photographed with a digital camera weakens a person’s memory of the experience: Linda A. Henkel, “Point-and-Shoot Memories: The Influence of Taking Photos on Memory for a Museum Tour,” Psychological Science, December 5, 2013, pss.sagepub.com/content/early/2013/12/04/0956797613504438.full.

  25. Mihai Nadin, “Information and Semiotic Processes: The Semiotics of Computation,” Cybernetics and Human Knowing 18, nos. 1–2 (2011): 153–175.

  26. Gary Marcus, Guitar Zero: The New Musician and the Science of Learning (New York: Penguin, 2012), 52.

  27. For a thorough description of how the brain learns to read, see Maryanne Wolf, Proust and the Squid: The Story and Science of the Reading Brain (New York: HarperCollins, 2007), particularly 108–133.

  28. Hubert L. Dreyfus, “Intelligence without Representation—Merleau-Ponty’s Critique of Mental Representation,” Phenomenology and the Cognitive Sciences 1 (2002): 367–383.

  29. Marcus, Guitar Zero, 103.

  30. David Z. Hambrick and Elizabeth J. Meinz, “Limits on the Predictive Power of Domain-Specific Experience and Knowledge in Skilled Performance,” Current Directions in Psychological Science 20, no. 5 (2011): 275–279.

  31. K. Anders Ericsson et al., “The Role of Deliberate Practice in the Acquisition of Expert Performance,” Psychological Review 100, no. 3 (1993): 363–406.

  32. Nigel Warburton, “Robert Talisse on Pragmatism,” Five Books, September 18, 2013, fivebooks.com/interviews/robert-talisse-on-pragmatism.

  33. Jeanne Nakamura and Mihaly Csikszentmihalyi, “The Concept of Flow,” in C. R. Snyder and Shane J. Lopez, eds., Handbook of Positive Psychology (Oxford, U.K.: Oxford University Press, 2002), 90–91.

  Interlude, with Dancing Mice

  1. Robert M. Yerkes, The Dancing Mouse: A Study in Animal Behavior (New York: Macmillan, 1907), vii–viii, 2–3.

  2. Ibid., vii.

  3. Robert M. Yerkes and John D. Dodson, “The Relation of Strength of Stimulus to Rapidity of Habit-Formation,” Journal of Comparative Neurology and Psychology 18 (1908): 459–482.

  4. Ibid.

  5. Mark S. Young and Neville A. Stanton, “Attention and Automation: New Perspectives on Mental Overload and Performance,” Theoretical Issues in Ergonomics Science 3, no. 2 (2002): 178–194.

  6. Mark W. Scerbo, “Adaptive Automation,” in Raja Parasuraman and Matthew Rizzo, eds., Neuroergonomics: The Brain at Work (New York: Oxford University Press, 2007), 239–252.

  Chapter Five: WHITE-COLLAR COMPUTER

  1. “RAND Study Says Computerizing Medical Records Could Save $81 Billion Annually and Improve the Quality of Medical Care,�
� RAND Corporation press release, September 14, 2005.

  2. Richard Hillestad et al., “Can Electronic Medical Record Systems Transform Health Care? Potential Health Benefits, Savings, and Costs,” Health Affairs 24, no. 5 (2005): 1103–1117.

  3. Reed Abelson and Julie Creswell, “In Second Look, Few Savings from Digital Health Records,” New York Times, January 10, 2013.

  4. Jeanne Lambrew, “More than Half of Doctors Now Use Electronic Health Records Thanks to Administration Policies,” The White House Blog, May 24, 2013, whitehouse.gov/blog/2013/05/24/more-half-doctors-use-electronic-health-records-thanks-administration-policies.

  5. Arthur L. Kellermann and Spencer S. Jones, “What It Will Take to Achieve the As-Yet-Unfulfilled Promises of Health Information Technology,” Health Affairs 32, no. 1 (2013): 63–68.

  6. Ashly D. Black et al., “The Impact of eHealth on the Quality and Safety of Health Care: A Systematic Overview,” PLOS Medicine 8, no. 1 (2011), plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.1000387.

  7. Melinda Beeuwkes Buntin et al., “The Benefits of Health Information Technology: A Review of the Recent Literature Shows Predominantly Positive Results,” Health Affairs 30, no. 3 (2011): 464–471.

  8. Dean F. Sittig et al., “Lessons from ‘Unexpected Increased Mortality after Implementation of a Commercially Sold Computerized Physician Order Entry System,’ ” Pediatrics 118, no. 2 (August 1, 2006): 797–801.

  9. Jerome Groopman and Pamela Hartzband, “Obama’s $80 Billion Exaggeration,” Wall Street Journal, March 12, 2009. See also, by the same authors, “Off the Record—Avoiding the Pitfalls of Going Electronic,” New England Journal of Medicine 358, no. 16 (2008): 1656–1658.

  10. See Fred Schulte, “Growth of Electronic Medical Records Eases Path to Inflated Bills,” Center for Public Integrity, September 19, 2012, publicintegrity.org/2012/09/19/10812/growth-electronic-medical-records-eases-path-inflated-bills; and Reed Abelson et al., “Medicare Bills Rise as Records Turn Electronic,” New York Times, September 22, 2012.

  11. Daniel R. Levinson, CMS and Its Contractors Have Adopted Few Program Integrity Practices to Address Vulnerabilities in EHRs (Washington, D.C.: Office of the Inspector General, Department of Health and Human Services, January 2014), oig.hhs.gov/oei/reports/oei-01-11-00571.pdf.

  12. Danny McCormick et al., “Giving Office-Based Physicians Electronic Access to Patients’ Prior Imaging and Lab Results Did Not Deter Ordering of Tests,” Health Affairs 31, no. 3 (2012): 488–496. An earlier study tracked the treatment of diabetes patients over five years at two clinics, one that had installed an electronic medical record system and one that hadn’t. It found that physicians at the clinic with the EMR system ordered more tests but did not achieve better glycemic control in their patients. “The data suggest that despite the substantial cost and increasing technical sophistication of EMRs, EMR use failed to achieve desirable levels of clinical improvement,” wrote the researchers. Patrick J. O’Connor et al., “Impact of an Electronic Medical Record on Diabetes Quality of Care,” Annals of Family Medicine 3, no. 4 (July 2005): 300–306.

  13. Timothy Hoff, “Deskilling and Adaptation among Primary Care Physicians Using Two Work Innovations,” Health Care Management Review 36, no. 4 (2011): 338–348.

  14. Schulte, “Growth of Electronic Medical Records.”

  15. Hoff, “Deskilling and Adaptation.”

  16. Danielle Ofri, “The Doctor vs. the Computer,” New York Times, December 30, 2010.

  17. Thomas H. Payne et al., “Transition from Paper to Electronic Inpatient Physician Notes,” Journal of the American Medical Information Association 17 (2010): 108–111.

  18. Ofri, “Doctor vs. the Computer.”

  19. Beth Lown and Dayron Rodriguez, “Lost in Translation? How Electronic Health Records Structure Communication, Relationships, and Meaning,” Academic Medicine 87, no. 4 (2012): 392–394.

  20. Emran Rouf et al., “Computers in the Exam Room: Differences in Physician-Patient Interaction May Be Due to Physician Experience,” Journal of General Internal Medicine 22, no. 1 (2007): 43–48.

  21. Avik Shachak et al., “Primary Care Physicians’ Use of an Electronic Medical Record System: A Cognitive Task Analysis,” Journal of General Internal Medicine 24, no. 3 (2009): 341–348.

  22. Lown and Rodriguez, “Lost in Translation?”

  23. See Saul N. Weingart et al., “Physicians’ Decisions to Override Computerized Drug Alerts in Primary Care,” Archives of Internal Medicine 163 (November 24, 2003): 2625–2631; Alissa L. Russ et al., “Prescribers’ Interactions with Medication Alerts at the Point of Prescribing: A Multi-method, In Situ Investigation of the Human–Computer Interaction,” International Journal of Medical Informatics 81 (2012): 232–243; M. Susan Ridgely and Michael D. Greenberg, “Too Many Alerts, Too Much Liability: Sorting through the Malpractice Implications of Drug-Drug Interaction Clinical Decision Support,” Saint Louis University Journal of Health Law and Policy 5 (2012): 257–295; and David W. Bates, “Clinical Decision Support and the Law: The Big Picture,” Saint Louis University Journal of Health Law and Policy 5 (2012): 319–324.

  24. Atul Gawande, The Checklist Manifesto: How to Get Things Right (New York: Henry Holt, 2010), 161–162.

  25. Lown and Rodriguez, “Lost in Translation?”

  26. Jerome Groopman, How Doctors Think (New York: Houghton Mifflin, 2007), 34–35.

  27. Adam Smith, The Wealth of Nations (New York: Modern Library, 2000), 840.

  28. Ibid., 4.

  29. Frederick Winslow Taylor, The Principles of Scientific Management (New York: Harper & Brothers, 1913), 11.

  30. Ibid., 36.

  31. Hannah Arendt, The Human Condition (Chicago: University of Chicago Press, 1998), 147.

  32. Harry Braverman, Labor and Monopoly Capital: The Degradation of Work in the Twentieth Century (New York: Monthly Review Press, 1998), 307.

  33. For a succinct review of the Braverman debate, see Peter Meiksins, “Labor and Monopoly Capital for the 1990s: A Review and Critique of the Labor Process Debate,” Monthly Review, November 1994.

  34. James R. Bright, Automation and Management (Cambridge, Mass.: Harvard University, 1958), 176–195.

  35. Ibid., 188.

  36. James R. Bright, “The Relationship of Increasing Automation and Skill Requirements,” in National Commission on Technology, Automation, and Economic Progress, Technology and the American Economy, Appendix II: The Employment Impact of Technological Change (Washington, D.C.: U.S. Government Printing Office, 1966), 201–221.

  37. George Dyson, comment on Edge.org, July 11, 2008, edge.org/discourse/carr_google.html#dysong.

  38. For a lucid explanation of machine learning, see the sixth chapter of John MacCormick’s Nine Algorithms That Changed the Future: The Ingenious Ideas That Drive Today’s Computers (Princeton: Princeton University Press, 2012).

  39. Max Raskin and Ilan Kolet, “Wall Street Jobs Plunge as Profits Soar,” Bloomberg News, April 23, 2013, bloomberg.com/news/2013-04-24/wall-street-jobs-plunge-as-profits-soar-chart-of-the-day.html.

  40. Ashwin Parameswaran, “Explaining the Neglect of Doug Engelbart’s Vision: The Economic Irrelevance of Human Intelligence Augmentation,” Macroresilience, July 8, 2013, macroresilience.com/2013/07/08/explaining-the-neglect-of-doug-engelbarts-vision/.

  41. See Daniel Martin Katz, “Quantitative Legal Prediction—or—How I Learned to Stop Worrying and Start Preparing for the Data-Driven Future of the Legal Services Industry,” Emory Law Journal 62, no. 4 (2013): 909–966.

  42. Joseph Walker, “Meet the New Boss: Big Data,” Wall Street Journal, September 20, 2012.

  43. Franco “Bifo” Berardi, The Soul at Work: From Alienation to Automation (Los Angeles: Semiotext(e), 2009), 96.

  44. A. M. Turing, “Systems of Logic Based on Ordinals,” Proceedings of the London Mathematical Society 45, no. 2239 (1939): 161–228.

  45. Ibid.

  46. Hector J. Levesque, “On Our Best Behaviour,” lecture delivered a
t the International Joint Conference on Artificial Intelligence, Beijing, China, August 8, 2013.

  47. See Nassim Nicholas Taleb, Antifragile: Things That Gain from Disorder (New York: Random House, 2012), 416–419.

  48. Donald T. Campbell, “Assessing the Impact of Planned Social Change,” Occasional Paper Series, no. 8 (December 1976), Public Affairs Center, Dartmouth College, Hanover, N.H.

  49. Viktor Mayer-Schönberger and Kenneth Cukier, Big Data: A Revolution That Will Transform How We Live, Work, and Think (New York: Houghton Mifflin Harcourt, 2013), 166.

  50. Kate Crawford, “The Hidden Biases in Big Data,” HBR Blog Network, April 1, 2013, hbr.org/cs/2013/04/the_hidden_biases_in_big_data.html.

  51. In a 1968 article, Weed wrote, “If useful historical data can be acquired and stored cheaply, completely and accurately by new computers and interviewing technics without the use of expensive physician time, they should be seriously considered.” Lawrence L. Weed, “Medical Records That Guide and Teach,” New England Journal of Medicine 278 (1968): 593–600, 652–657.

  52. Lee Jacobs, “Interview with Lawrence Weed, MD—The Father of the Problem-Oriented Medical Record Looks Ahead,” Permanente Journal 13, no. 3 (2009): 84–89.

  53. Gary Klein, “Evidence-Based Medicine,” Edge, January 14, 2014, edge.org/responses/what-scientific-idea-is-ready-for-retirement.

  54. Michael Oakeshott, “Rationalism in Politics,” Cambridge Journal 1 (1947): 81–98, 145–157. The essay was collected in Oakeshott’s 1962 book Rationalism in Politics and Other Essays (New York: Basic Books).

  Chapter Six: WORLD AND SCREEN

  1. William Edward Parry, Journal of a Second Voyage for the Discovery of a North-West Passage from the Atlantic to the Pacific (London: John Murray, 1824), 277.

  2. Claudio Aporta and Eric Higgs, “Satellite Culture: Global Positioning Systems, Inuit Wayfinding, and the Need for a New Account of Technology,” Current Anthropology 46, no. 5 (2005): 729–753.