Weapons of Math Destruction

by Cathy O'Neil

  A potential student’s first click on a for-profit college website comes only after a vast industrial process has laid the groundwork. Corinthian, for example, had a thirty-person marketing team that spent $120 million annually, much of it to generate and pursue 2.4 million leads, which led to sixty thousand new students and $600 million in annual revenue. These large marketing teams reach potential students through a wide range of channels, from TV ads and billboards on highways and bus stops to direct mail, search advertising on Google, and even recruiters visiting schools and knocking on doors. An analyst on the team designs the various promotions with the explicit goal of getting feedback. To optimize recruiting—and revenue—they need to know whom their messages reached and, if possible, what impact they had. Only with this data can they go on to optimize the operation.

  The key for any optimization program, naturally, is to pick an objective. For diploma mills like the University of Phoenix, I think it’s safe to say, the goal is to recruit the greatest number of students who can land government loans to pay most of their tuition and fees. With that objective in mind, the data scientists have to figure out how best to manage their various communication channels so that together they generate the most bang for each buck.

  The data scientists start off with a Bayesian approach, which in statistics is pretty close to plain vanilla. The point of Bayesian analysis is to rank the variables with the most impact on the desired outcome. Search advertising, TV, billboards, and other promotions would each be measured as a function of their effectiveness per dollar. Each develops a different probability, which is expressed as a value, or a weight.

  It gets complicated, though, because the various messaging campaigns all interact with each other, and much of their impact can’t be measured. For example, do bus advertisements drive up the probability that a prospect will take a phone call? It’s hard to say. It’s easier to track online messaging, and for-profits can gather vital details about each prospect—where they live and what web pages they’ve surfed.

  That’s why much of the advertising money at for-profit universities goes to Google and Facebook. Each of these platforms allows advertisers to segment their target populations in meticulous detail. Publicists for a Judd Apatow movie, for example, could target males from age eighteen to twenty-eight in the fifty richest zip codes, perhaps zeroing in on those who have clicked on or “liked” links to Apatow’s hit movie Trainwreck, have mentioned him on Twitter, or are friends with someone who has. But for-profit colleges hunt in the opposite direction. They’re more likely to be targeting people in the poorest zip codes, with special attention to those who have clicked on an ad for payday loans or seem to be concerned with post-traumatic stress. (Combat veterans are highly recruited, in part because it’s easier to get financing for them.)

  The campaign proceeds to run an endless series of competing ads against each other to see which ones bring in the most pros pects. This method, based on so-called A/B testing, is one that direct-mail marketers have been using for decades. They send a plethora of come-ons, measure the responses, and fine-tune their campaigns. Every time you discover another credit card offer in your mailbox, you’re participating in one of these tests. By throwing out the letter unopened, you’re providing the company with a valuable piece of data: that campaign didn’t work for you. Next time they’ll try a slightly different approach. It may seem fruitless, since so many of these offers wind up in the trash. But for many direct marketers, whether they’re operating on the Internet or through the mail, a 1 percent response rate is the stuff of dreams. After all, they’re working with huge numbers. One percent of the US population is more than three million people.

  Once these campaigns move online, the learning accelerates. The Internet provides advertisers with the greatest laboratory ever for consumer research and lead generation. Feedback from each promotion arrives within seconds—a lot faster than the mail. Within hours (instead of months), each campaign can zero in on the most effective messages and come closer to reaching the glittering promise of all advertising: to reach a prospect at the right time, and with precisely the best message to trigger a decision, and thus succeed in hauling in another paying customer. This fine-tuning never stops.

  And increasingly, the data-crunching machines are sifting through our data on their own, searching for our habits and hopes, fears and desires. With machine learning, a fast-growing domain of artificial intelligence, the computer dives into the data, following only basic instructions. The algorithm finds patterns on its own, and then, through time, connects them with outcomes. In a sense, it learns.

  Compared to the human brain, machine learning isn’t especially efficient. A child places her finger on the stove, feels pain, and masters for the rest of her life the correlation between the hot metal and her throbbing hand. And she also picks up the word for it: burn. A machine learning program, by contrast, will often require millions or billions of data points to create its statistical models of cause and effect. But for the first time in history, those petabytes of data are now readily available, along with powerful computers to process them. And for many jobs, machine learning proves to be more flexible and nuanced than the traditional programs governed by rules.

  Language scientists, for example, spent decades, from the 1960s to the early years of this century, trying to teach computers how to read. During most of this time, they programmed definitions and grammatical rules into the code. But as any foreign-language student discovers all too quickly, languages teem with exceptions. They have slang and sarcasm. The meaning of certain words changes with time and geography. The complexity of language is a programmer’s nightmare. Ultimately, coding it is hopeless.

  But with the Internet, people across the earth have produced quadrillions of words about our lives and work, our shopping, and our friendships. By doing this, we have unwittingly built the greatest-ever training corpus for natural-language machines. As we turned from paper to e-mail and social networks, machines could study our words, compare them to others, and gather something about their context. The progress has been fast and dramatic. As late as 2011, Apple underwhelmed most of techdom with its natural-language “personal assistant,” Siri. The technology was conversant only in certain areas, and it made laughable mistakes. Most people I know found it near useless. But now I hear people talking to their phones all the time, asking for the weather report, sports scores, or directions. Somewhere between 2008 and 2015, give or take, the linguistic skills of algorithms advanced from pre-K to middle school, and for some applications much higher.

  These advances in natural language have opened up a mother lode of possibilities for advertisers. The programs “know” what a word means, at least enough to associate it with certain behaviors and outcomes, at least some of the time. Fueled in part by this growing linguistic mastery, advertisers can probe for deeper patterns. An advertising program might start out with the usual demographic and geographic details. But over the course of weeks and months it begins to learn the patterns of the people it’s targeting and to make predictions about their next moves. It gets to know them. And if the program is predatory, it gauges their weaknesses and vulnerabilities and pursues the most efficient path to exploit them.

  In addition to cutting-edge computer science, predatory advertisers often work with middlemen, who use much cruder methods to target prospects. In 2010, one effective ad featured a photo of President Obama and said: “Obama Asks Moms to Return to School: Finish Your Degree—Financial Aid Available to Those Who Qualify.” The ad suggested that the president had signed a new bill aimed at getting mothers back in school. This was a lie. But if it spurred people to click, it served its purpose.

  Behind this misleading headline, an entire dirty industry was beavering away. When a consumer clicked on the ad, according to a ProPublica investigation, she was asked a few questions, including her age and phone number, and was immediately contacted by a for-profit school. These callers didn’t give her any more information about
President Obama’s new bill, because it never existed. Instead they offered to help her borrow money for enrollment.

  This kind of online targeting is called “lead generation.” Its goal is to come up with lists of prospects, which can be sold—in this case, to for-profit universities. According to the ProPublica report, between 20 and 30 percent of the promotional budgets at for-profit colleges go to lead generation. For the most promising leads, colleges will pay as much as $150 each.

  One lead generator, Salt Lake City–based Neutron Interactive, posted fake jobs at websites like Monster.​com, as well as ads promising to help people get food stamps and Medicaid coverage, according to David Halperin, a public policy researcher. Using the same optimization methods, they would roll out loads of different ads, measuring their effectiveness for each demographic.

  The purpose of these ads was to lure desperate job seekers to provide their cell phone numbers. In follow-up calls, only 5 percent of the people showed interest in college courses. But those names were valuable leads. Each one was worth as much as $85 to for-profit colleges. And they would do everything in their power to make that investment pay off. Within five minutes of signing up, according to a US Government Accountability Office report, prospective students could expect to begin receiving calls. One target received more than 180 calls in a single month.

  The for-profit colleges, of course, have their own methods for generating leads. One of their most valuable tools is the College Board website, the resource that many students use to sign up for SAT tests and research the next step in their lives. According to Mara Tucker, a college preparedness counselor for the Urban Assembly Institute of Math and Science for Young Women, a public school in Brooklyn, the search engine on the website is engineered to direct poor students toward for-profit universities. Once a student has indicated in an online questionnaire that she’ll need financial aid, the for-profit colleges pop up at the top of her list of matching schools.

  For-profit colleges also provide free services in exchange for face time with students. Cassie Magesis, another readiness counselor at the Urban Assembly, told me that the colleges provide free workshops to guide students in writing their résumés. These ses sions help the students. But impoverished students who provide their contact information are subsequently stalked. The for-profit colleges do not bother targeting rich students. They and their parents know too much.

  Recruiting in all of its forms is the heart of the for-profit business, and it accounts for far more of their spending, in most cases, than education. A Senate report on thirty for-profit systems found that they employed one recruiter for every forty-eight students. Apollo Group, the parent company for the University of Phoenix, spent more than a billion dollars on marketing in 2010, almost all of it focused on recruiting. That came out to $2,225 per student on marketing and only $892 per student on instruction. Compare that to Portland Community College in Oregon, which spends $5,953 per student on instruction and about 1.2 percent of its budget, or $185 per student, on marketing.

  Math, in the form of complex models, fuels the predatory advertising that brings in prospects for these colleges. But by the time a recruiter is hounding prospective students on their cell phones, we’ve left the world of numbers behind. The sales pitches, with their promises of affordable tuition, bright career prospects, and upward mobility, aren’t that different from the promotions for magic elixirs, baldness cures, and vibrating belts that reduce waistline fat. They’re not new.

  Yet a crucial component of a WMD is that it is damaging to many people’s lives. And with these types of predatory ads, the damage doesn’t begin until students start taking out big loans for their tuition and fees.

  The crucial metric is the so-called 90-10 rule, included in the Higher Education Act of 1965. It stipulates that colleges cannot get more than 90 percent of their funding from federal aid. The thinking was that as long as the students had some “skin in the game” they would tend to take their education more seriously. But for-profit colleges quickly worked this ratio into their business plan. If students could scrape together a few thousand dollars, either from savings or bank loans, the universities could line them up for nine times that sum in government loans, making each student incredibly profitable.

  To many of the students, the loans sound like free money, and the school doesn’t take pains to correct this misconception. But it is debt, and many of them quickly find themselves up to their necks in it. The outstanding debt for students at the bankrupt Corinthian Colleges amounted to $3.5 billion. Almost all of it was backed by taxpayers and will never be repaid.

  Some people no doubt attend for-profit colleges and emerge with knowledge and skills that serve them well. But do they fare better than graduates from community colleges, whose degrees cost a fraction as much? In 2014, investigators at CALDER/American Institutes for Research created nearly nine thousand fictitious résumés. Some of their fake job applicants held associate degrees from for-profit universities, others had similar diplomas from community colleges, while a third group had no college education at all. The researchers sent their résumés to job postings in seven major cities and then measured the response rate. They found that diplomas from for-profit colleges were worth less in the workplace than those from community colleges and about the same as a high school diploma. And yet these colleges cost on average 20 percent more than flagship public universities.

  The feedback loop for this WMD is far less complicated than it is nefarious. The poorest 40 percent of the US population is in desperate straits. Many industrial jobs have disappeared, either replaced by technology or shipped overseas. Unions have lost their punch. The top 20 percent of the population controls 89 percent of the wealth in the country, and the bottom 40 percent controls none of it. Their assets are negative: the average household in this enormous and struggling underclass has a net debt of $14,800, much of it in extortionate credit card accounts. What these people need is money. And the key to earning more money, they hear again and again, is education.

  Along come the for-profit colleges with their highly refined WMDs to target and fleece the population most in need. They sell them the promise of an education and a tantalizing glimpse of upward mobility—while plunging them deeper into debt. They take advantage of the pressing need in poor households, along with their ignorance and their aspirations, then they exploit it. And they do this at great scale. This leads to hopelessness and despair, along with skepticism about the value of education more broadly, and it exacerbates our country’s vast wealth gap.

  It’s worth noting that these diploma mills drive inequality in both directions. The presidents of the leading for-profit universities make millions of dollars every year. For example, Gregory W. Cappelli, CEO of Apollo Education Group, the parent company of the University of Phoenix, took home $25.1 million in total compensation in 2011. At public universities, which have their own distortions, only football and basketball coaches can hope to make that much.

  For-profit colleges, sadly, are hardly alone in deploying predatory ads. They have plenty of company. If you just think about where people are hurting, or desperate, you’ll find advertisers wielding their predatory models. One of the biggest opportunities, naturally, is for loans. Everyone needs money, but some more urgently than others. These people are not hard to find. The neediest are far more likely to reside in impoverished zip codes. And from a predatory advertiser’s perspective, they practically shout out for special attention with their queries on search engines and their clicks on coupons.

  Like for-profit colleges, the payday loan industry operates WMDs. Some of them are run by legal operations, but the industry is fundamentally predatory, charging outrageous interest rates that average 574 percent on short-term loans that are flipped on average eight times—making them much more like long-term loans. They are critically supported by legions of data brokers and lead generators, many of them scam artists. Their advertisements pop up on computers and phones, offering fast access to cash. W
hen the prospects fill out the applications, often including their bank information, they open themselves to theft and abuse.

  In 2015, the Federal Trade Commission charged two data brokers for selling the loan applications of more than half a million consumers. According to the suit, the companies, Sequoia One of Tampa, Florida, and Gen X Marketing Group of nearby Clearwater, made off with customers’ phone numbers, employer details, social security numbers, and bank account information—and then sold them for about fifty cents each. The companies that bought the information, according to the regulators, raided the consumers’ bank accounts for “at least” $7.1 million. Many of the victims were subsequently charged bank fees for emptying out their account or bouncing checks.

  If you think about the numbers involved, they’re almost pathetically low. Spread over a half million accounts, $7.1 million comes to barely $14 each. Even if the thieves failed to access many of these accounts, much of the money they stole was no doubt in small numbers, the last $50 or $100 that some poor people keep in their accounts.

  Now regulators are pushing for new laws governing the market for personal data—a crucial input for all sorts of WMDs. To date, a couple of federal laws, such as the Fair Credit Reporting Act and the Health Insurance Portability and Accountability Act, or HIPAA, establish some limits on health and credit data. Maybe, with an eye on lead generators, they’ll add more.

  However, as we’ll see in coming chapters, some of the most effective and nefarious WMDs manage to engineer work-arounds. They study everything from neighborhoods to Facebook friends to predict our behavior—and even lock us up.