The Formula_How Algorithms Solve All Our Problems... and Create More

by Luke Dormehl

  A PERIGEE BOOK

  Published by the Penguin Group

  Penguin Group (USA) LLC

  375 Hudson Street, New York, New York 10014

  USA • Canada • UK • Ireland • Australia • New Zealand • India • South Africa • China

  penguin.com

  A Penguin Random House Company

  THE FORMULA

  Copyright © 2014 by Itzy, Kickass.so

  Penguin supports copyright. Copyright fuels creativity, encourages diverse voices, promotes free speech, and creates a vibrant culture. Thank you for buying an authorized edition of this book and for complying with copyright laws by not reproducing, scanning, or distributing any part of it in any form without permission. You are supporting writers and allowing Penguin to continue to publish books for every reader.

  PERIGEE is a registered trademark of Penguin Group (USA) LLC.

  The “P” design is a trademark belonging to Penguin Group (USA) LLC.

  ISBN: 978-0-698-15884-9

  First American edition: November 2014

  Previously published in the UK in 2014 by Virgin Books, an imprint of Ebury Publishing.

  While the author has made every effort to provide accurate telephone numbers, Internet addresses, and other contact information at the time of publication, neither the publisher nor the author assumes any responsibility for errors, or for changes that occur after publication. Further, the publisher does not have any control over and does not assume any responsibility for author or third-party websites or their content.

  Most Perigee books are available at special quantity discounts for bulk purchases for sales promotions, premiums, fund-raising, or educational use. Special books, or book excerpts, can also be created to fit specific needs. For details, write: Special.Markets@us.penguingroup.com.

  To my friend, Tim Plester

  Contents

  Title Page

  Copyright

  Dedication

  Acknowledgments

  An Explanation of the Title, and Other Cyberbole

  1 The Quantified Selves

  2 The Match & the Spark

  3 Do Algorithms Dream of Electric Laws?

  4 The Machine That Made Art

  Conclusion: Predicting the Future

  A Note on Author Interviews

  Notes

  Index

  Acknowledgments

  Writing a book is almost always a solitary activity, but I was fortunate enough to be surrounded by a group of people whose love and/or support made The Formula a pleasure to work on. Thanks first of all to Clara, Tim and Celia Lunt, as well as members of my family. I could not have completed this project without the invaluable aid of Ed Faulkner, while it would never have got through the door in the first place were it not for my agent Maggie Hanbury, Henry de Rougemont, Simon Garfield and Jake Lingwood. Many thanks to Marian Lizzi, my U.S. editor. Appreciative nods also go in the direction of all those who spent countless hours speaking with me as part of my research (a full list of their names is printed on page 243), in addition to my FastCo.Labs editor Chris Dannen, Cult of Mac’s Leander Kahney, the excellent Nicole Martinelli, Karl French, tech comms guru Alice Bonasio-Atkinson, Tim Matts, Alex Millington, Michael Grothaus, Tom Atkinson, Simon Callow, and my brothers-from-other-mothers, Andre and Nathan Trantraal. All helped this book along in one way or another. All I can take full credit for are the (hopefully few) mistakes.

  An Explanation of the Title, and Other Cyberbole

  At their root, algorithms are little more than a series of step-by-step instructions, usually carried out by a computer. However, if their description is straightforward, their inner workings and impact on our lives are anything but.

  Algorithms sort, filter and select the information that is presented to us on a daily basis. They are responsible for the search results shown to us by Google, the information about our friends that is highlighted by Facebook, and the type of products Amazon predicts we will be most likely to buy. Increasingly, they will also be responsible for what movies, music and other entertainment look like, which people we are partnered with in predictive relationships, and even the ways in which laws are enforced and police operate. An algorithm can scan through your metadata and recommend that you will likely make a hardworking employee, just as one could accuse you of a crime, or determine that you are unfit to drive a car. In the process, algorithms are profoundly changing the way that we view (to quote Douglas Adams) life, the universe and everything.

  One of my favorite observations about technology is the one often attributed to the cultural theorist Paul Virilio: “The invention of the ship was also the invention of the shipwreck.” One could, of course, turn this around and say that the inventor of the shipwreck was also the person that invented the ship. Algorithms have had their fair share of shipwrecks (which I will discuss during the course of this book), but they also perform incredibly useful functions: allowing us to navigate through the 2.5 quintillion bytes of data that are generated each day (a million times more information than the human brain is capable of holding) and draw actionable conclusions from it.

  As with the old adage about how to carve an elephant statue (you chip away everything that isn’t an elephant), I will start out by explaining what this book is not. It is not, for one thing, a computer science textbook about algorithms. There are far better books (and, indeed, far more qualified writers) to achieve this task.

  Neither is it a history of the algorithm as a concept. While I considered attempting such a thing, I was put off by both the sheer scale of the project and the fact that its end result—while no doubt fascinating under the stewardship of the right author—would be not entirely dissimilar to the textbook I also shied away from. By this I do not mean that a history book and a textbook are necessarily the same thing, but rather that a history of a once-niche mathematical concept would likely appeal only to those mathematicians or computer scientists already familiar with it.

  Instead, I want to tell the story of the myriad ways (some subtle, others less so) that algorithms affect all of our lives: from the entertainment we enjoy to the way we think about human relationships. What do scoring hot dates, shooting Hollywood turkeys, bagging your own poo, and cutting opportunities for lawyers’ fees have in common? This is a book about the algorithmization of life as we know it.

  In my day job, writing about a field known as the “digital humanities” for Fast Company, I’m constantly considering the implications of “algorithmic” culture and the idea (not always a misguided one) that no matter what the problem, it can be solved with the right algorithm.

  A typical illustration of what I mean can be seen in Bill Tancer’s 2009 book Click: What We Do Online and Why It Matters. Tancer—described in at least one place as “the world’s preeminent expert on online [behavior]”—begins his book by describing a radio interview he listened to in the car one day. Being interviewed was a British psychologist referring to a mathematical formula he had developed to determine the most objectively depressing week of the year. After much work, he had discovered that this was the third week of January, a feat brought about by the convergence of failed New Year’s resolutions, credit-card debt accumulated over the holiday season, and the usual dismal weather patterns. Tancer notes that he remained unconvinced: a perspective that was later backed up when the formula was severely criticized for its lack of scientific rigor. However, his lack of conviction has nothing to do with the suggestion that a reducti
ve formula could possibly provide answers on a topic as complex and multifaceted as depression, but rather because he believes that he had come up with a better formula.1

  In other words, his problem wasn’t with the existence of the sum, but rather with its working.

  This book was spurred by years of hearing similar observations, all claiming that there is no problem technology cannot reduce to its most formulaic level and thereby determine objective answers in response to. This thinking is the reason “The Formula” is in upper case rather than simply existing as a catchall for the various technical processes I describe. It implies an ideological element, and that ideology is evidenced by the more expansive view I take of algorithms and their associated technological apparatus: conceiving of them as the embodiment of a particular form of techno-rationality, symptomatic of a type of social ordering built around the promise of objectivity. In this way I use The Formula much as the late American political scientist and communications theorist Harold Lasswell used the word “technique”: referring, in Lasswell’s words, to “the ensemble of practices by which one uses available resources to achieve values.” It is about both the application and the scope of application, as well as the existence of objective truths lurking just beneath the surface—to be teased out with the right data-mining tools.

  Writers on technology tend, with a few notable exceptions, to be overwhelmingly utopian in their outlook. To them, all progress is positive. As a result, there is a tendency among technology writers to christen each new invention as the totemic figurehead of its own “era”—something that has led to the disdainful term “cyberbole.” While this book could very well join the number of volumes about algorithms and big data already lining the shelves, what I am interested in goes back much further than simply the birth of the Internet or the age of the personal computer.

  Writing during the first half of the 1960s, the French sociologist (and Christian anarchist!) Jacques Ellul described a creature known as the Technical Man, an individual “fascinated by results, by the immediate consequences of setting standardized devices into motion . . . committed to the never-ending search for ‘the one best way’ to achieve any designated objective.” This objective could occasionally be clouded (or else speeded up) by a naive enthusiasm for the means of getting there: not by anything so unquantifiable as ethical concerns, but rather by an enthusiasm for the ingenuity, elegance and “spectacular effectiveness” of man’s ability to dream up solutions.

  As Ellul’s observation proves, this approach is not therefore a new one, and the founders of Google and the heads of the various high-tech companies I discuss are not the first people to display what the late American sociologist Lewis Mumford called the “will-to-order”—meaning the desire to make formulaic sense of the world. Writing in the 1930s, long before the birth of the modern computer, Mumford noted that automation was simultaneously for “enlarging the mechanical or sensory capacities of the human body” and “for reducing to a measurable order and regularity the processes of life.” To make sense of a big picture, we reduce it, he suggested. To take an abstract concept such as human intelligence and turn it into something quantifiable, we abstract it further, stripping away complexity and assigning it a seemingly arbitrary number, which becomes a person’s IQ.

  What is new is the scale that this idea is now being enacted upon, to the point that it is difficult to think of a field of work or leisure that is not subject to algorithmization and The Formula. This book is about how we reached this point, and how the age of the algorithm impacts and shapes subjects as varied as human creativity, human relationships (and, more specifically, romantic relationships), notions of identity and matters of law.

  Algorithms are very good at providing us with answers in all of these cases.

  The real question is whether they give the answers we want.

  CHAPTER 1

  The Quantified Selves

  Larry Smarr weighed close to 200 pounds when he arrived in La Jolla, California, in 2000. The photograph from his driver’s license at the time depicts an overweight 51-year-old with a soft, round face and the fleshy ripple of a double chin. Although he regularly tested his mind as a leading physicist and expert in supercomputing, Smarr had not exercised his body properly in years. He regularly drank Coke and enjoyed chowing down on deep-fried, sugar-coated pastries.1 Moving to the West Coast to run a new institute at the University of California called the California Institute for Telecommunications and Information Technology, he was suddenly confronted with a feeling he hadn’t experienced in years: a sense of deep inadequacy. “I looked around and saw all these thin, fit people,” he remembers. “They were running, bicycling, looking beautiful. I realized how different I was.”

  Smarr’s next question was one shared by scientists and philosophers alike: Why? He visited his local bookshop and bought “about a zillion” diet books. None of them was deemed satisfactory to a man whose brain could handle the labyrinthine details of supernovas and complex star formations, but for whom healthy eating and regular exercise seemed, ironically enough, like complex astrophysics. “They all seemed so arbitrary,” he says. It wasn’t until he discovered a book called The Zone, written by biochemist Barry Sears, that Smarr found what it was that he was looking for. Sears treated the body as a coupled nonlinear system in which feedback mechanisms like the glucose-insulin and immune systems interact with one another. That was an approach that Smarr could relate to. Inspired, he started measuring his weight, climbing naked onto a pair of scales each day and writing down the number that appeared in front of him. Next, he hired a personal trainer and began keeping track of the amount of exercise he participated in on a daily basis. After that it was on to diet—breaking food down into its “elemental units” of protein, fat, carbohydrates, fiber, sugar and salt, and modifying his diet to remove those inputs that proved detrimental to well-being. “Think of it like being an engineer, reverse-engineering the subsystems of a car,” Smarr says. “From that you can derive that you need a certain level of petrol in order to run, and that if you put water in your gas tank you will tear apart the car. We don’t think that way about our bodies, but that is the way we ought to think.”

  It didn’t take long until Smarr turned to more complex forms of technology to help him lose weight. He purchased and began wearing Polar WearLink heart-rate straps, FitBits, BodyMedia, and other pieces of wearable tech that use algorithms to convert body metrics into data. Wanting to check his progress, Smarr started paying for blood tests at a private laboratory and then—in a quest for yet more numbers to pore over—began FedEx-ing off his stool for regular analysis. “I didn’t have a biology or medical background, so I had to teach myself,” Smarr says of his process.

  One of the numbers he wound up fixating on related to complex reactive proteins, which act as a direct measure of inflammation in the body. In a normal human body this number should be less than one. In Smarr’s case it was five. Over time it rose to 10, then 15. As a scientist, he had discovered a paradox. “How was it that I had reduced all of the things that normally drive inflammation in terms of my food supply, but the numbers were growing and growing with this chronic inflammation?” he muses. “It didn’t make any sense.”

  At that point, Smarr decided to visit his doctor to present the findings. The appointment didn’t go as planned.

  “Do you have any symptoms?” the doctor asked.

  “No,” Smarr answered. “I feel fine.”

  “Well, why are you bothering me, then?”

  “Well, I’ve got these great graphs of my personal data.”

  “Why on earth are you doing that?” came the response.

  The doctor told Smarr that his data was too “academic” and had no use for clinical practice. “Come back when there’s something actually wrong with you, rather than just anomalies in your charts,” the doctor said.

  Several weeks later, Smarr felt a severe pain in the left side of hi
s abdomen. He went back to the doctor’s and was diagnosed with diverticulitis, a disease caused by acute inflammation. It was the perfect illustration of Smarr’s problem: doctors would deal only in clinical symptoms, unwilling to delve into the data that might actually prove preventative. Having learned an important lesson, Smarr decided to take over his own health tracking.

  “People have been brainwashed into thinking that they have no responsibility for the state of their bodies,” he says. “I did the calculation of the ratio of two 20-minute doctor visits per year, compared to the total number of minutes in the year, and it turns out to be one in 10,000. If you think that someone is going to be able to tell you what’s wrong with you and fix the problem in one 10,000th of the time that you have available to do the same, I’d say that’s the definition of insanity. It just doesn’t make any sense.”

  In the aftermath of the doctor’s visit, Smarr began obsessively tracking any and all symptoms he noticed and linking each of these to fluctuations in his body data. He also upped the amount of information he was looking at, and started using complex data-mining algorithms to sift through it looking for irregularities. Another high number he zeroed in on referred to lactoferrin, an antibacterial agent shed by white blood cells when they are in attack mode, a bit like a canister of tear gas being dispersed into a crowd of people. This number was meant to be less than seven. It was 200 when Smarr first checked it, and by May 2011 had risen to 900. Searching through scientific literature, Smarr diagnosed himself as having a chronic autoimmune disorder, which he later narrowed down to something called Crohn’s disease. “I was entirely led there by the biomarkers,” he notes.

  In this sense, Smarr is the epitome of algorithmic living. He tracks 150 variables on a constant basis, walks 7,000 steps each day, and has access to millions of separate data points about himself. As such, both his body and his daily life are divided, mathematized and codified in a way that means that he can follow what is happening inside him in terms of pure numbers. “As our technological ability to ‘read out’ the state of our body’s main subsystems improves, keeping track of changes in our key biochemical markers over time will become routine, and deviations from the norm will more easily reveal early signals of disease development,” Smarr argues in an essay entitled “Towards Digitally Enabled Genomic Medicine: A 10-Year Detective Story of Quantifying My Body.”2