by Olaf Groth
Ultimately, though, identifying and monitoring that thin and blurry line between benefit and harm will require greater transparency and public scrutiny of new algorithms, data sets, and platforms. Despite the open atmosphere across much of the AI field, proprietary code and data often make it difficult to ascertain the point at which systems cross boundaries. That uncertainty will lead to problems in legal proceedings, financial transactions, and virtually every other field that AI touches, but the concerns it raises for education are especially critical. We can barely agree on the line between education and brainwashing in our analog textbooks and longstanding curricula, let alone in areas where AI systems provide far more subtle nudges toward learning outcomes.
By keeping a human in the loop as part of the peer-to-peer tutoring platform he’s developing, Patrick Poirier hopes to install a crucial guardrail against AI manipulation. Poirier founded Erudite AI, a Montreal start-up that uses machine learning to increase the effectiveness of students tutoring other students. The application, still in the pilot stages in early 2018, will help guide students as they work with a fellow classmate, analyzing their interaction, integrating a variety of best learning practices, and guiding the tutor toward the best educational pathways for the student with whom they’re working. For example, if the tutoring student tries to provide an answer immediately, the system would interject and help the tutor provide feedback in a way that helps his fellow pupil learn, Poirier explains. As he and his colleagues train the system, a professional educator will monitor the interactions. At the outset, Erudite needs the trained teacher as a proxy for the AI system they’re developing. Those interactions begin to prime the AI, and then educators can help refine its capabilities as it takes on more of the interaction. Ultimately, Poirier says, the largely automated system will guide peers as they help each other learn, but it also will remain auditable by teachers or educational agencies.
That initial idea drew the attention of the IBM Watson XPRIZE officials, who named it a Top Ten team, but what makes the start-up especially unique is Poirier’s broader outlook on the future of education. Rather than grading school exams down from 100 percent, the “disheartening” current standard, Poirier says, Erudite will work on a “grading up” scale, much like a video game might. Students start at a baseline and work their way higher as they gain more knowledge. “With technology, we can see more personalized learning, more interactive learning,” Poirier says. “The theory will be integrated into the act of learning.” Whereas students today come to professors wondering where they lost points—as if they’d ever earned them in the first place—they will have to work up to those scores in the future. “You can learn by doing tasks, but if something isn’t validating that you’re learning the task, you don’t know. You need to test to measure that,” he says. “With technology, you can start grading up because as you do things you get points.”
Such approaches won’t guarantee the validity and veracity of AI systems, but resetting goals to incentivize gains over penalties could help enhance trust in thinking machines until fuller transparency and “explainability” emerges. In the meantime, evolving AI technologies will improve our ability to analyze algorithms and data sets. Already, researchers have developed AI models that test one another, pushing toward higher quality output (at least as defined by the developers behind the code). Only public scrutiny can assure the quality of those algorithms and data sets from a broader societal perspective. Continued efforts to establish bodies that can audit AI systems and balance corporate interests with community values could help safeguard against the manipulation of learning minds.
SAVANT MACHINES OR SYMBIOTIC COGNITION?
Companies collect, analyze, and sell massive streams of data on each of us. Walk into the grocery store, and security cameras track the route you take around the aisles. Swipe your credit and loyalty club cards, and the store compares what you purchased and in what combination with your previous trips. Pull that all together, and the store knows precisely what product to put where and at what price to make you (and other similar shoppers) more likely to grab an extra pack or two. Yet, even the grocery store, let alone a huge Internet company, has far more data about you than it would ever care to use. Amazon might not need to know a customer is a physically fit, middle-aged, cheese-eating, blues-loving, job-hopping, Latino fly fisherman with ancestral roots in Costa Rica—unless that combination of attributes make them more likely to buy a certain product.
The depth of awareness that large companies could compile differs from what they actually compile, and that might help quell some of our anxieties. Much like the blurry lines of confidentiality and privacy, we don’t have a clear sense of where the balance tips on the equality of power between the system and the person. If companies begin to generate ever-finer grains of truth about our life patterns, do we really want to see them in their raw form? Does that knowledge begin to undercut the tenuous sense of parity that relies, in part, on the willful ignorance of consumers? After all, under the EU’s data-protection rules, people retain control over the primary data companies collect on them, but the companies appear to maintain control over the insights they derive from that data. We might not really know how they judge us.
In 2017, we visited the IBM Watson West center with a delegation of executive MBA students from the creative industries, during which the IBMers there demonstrated a new application that analyzes personalities based on Twitter feeds. One of the students, an outgoing studio-musician-turned-entrepreneur from Texas named John Anthony Martinez, bravely volunteered his feed. Watson promptly concluded that Martinez was “melancholy” and “guarded and particular,” but also “energetic” with a fast-paced and busy schedule. And while Martinez preferred to listen rather than talk, Watson concluded, he’s “driven by a desire for self-expression.” The results didn’t surprise Martinez, but he didn’t fully agree, either, perhaps because Watson doesn’t discriminate between tweets from retweets, which probably skewed the report a bit. People with more restrained sensibilities might object to many of those labels, especially when based on a data stream that captures just one slice of their identity (and one that’s often fabricated, to boot). Sometimes, that unvarnished feedback provides valuable insight, but experienced coaches and psychologists say that works only if the feedback is delivered properly. Could Watson gain enough emotional intelligence to share its awareness fruitfully, in a way that maintains a healthy, equitable relationship between coach and individual, machine and human?
It’s an entirely different matter when it’s another person, not a machine, making such judgments. Plenty of people are inept when it comes to emotional intelligence, empathy, and awareness of emotionally dicey situations. But we know that and, absent evidence of emotional cruelty or intent to injure or slander, we accept a fair amount of ambiguity in our human interactions. We know and trust that humans, in general, possess the sensory and evaluative capabilities to pull back from or ease explosive situations.
We don’t yet know if that sensibility will exist in cognitive computers. How do we assure awareness parity for especially sensitive social questions, such as sexuality, identity, and personal health? Most of us might readily accept the medical advances powered by IBM Watson if they can provide doctors a clearer picture of our well-being, even if that means the machine knows aspects of our minds and bodies better than we do ourselves. And we might want an AI-powered system to provide us sound financial advice and nudge us toward better use of our hard-earned money. But if they do know us that intimately, don’t we want to know how deep that insight goes?
Imagine the machine developing a constantly evolving analysis of you and the people with whom you relate, but never communicating that view back to you or them. Years from now, an AI system might help “George” prepare for the day ahead by collecting personality profiles on all the people he expects to meet. The platform wouldn’t need access to the various individuals’ data streams, having already scraped enough information from publicl
y available social media and its analysis of any relevant voice recordings, personal chats, or photos on George’s smartphone. Now, imagine you’re sitting across the table from George, getting his feedback about your ongoing job search in the United States. While you’re asking George for advice, his AI is processing the discussion in real time. It’s analyzing your chances of landing a US-based job and correlating that with data gleaned from other workers in related industries. Does George use that feedback to mentor and advise you? Or does he realize your prospects are dim and dismiss you because you offer little benefit as a member of his business network? Meanwhile, having come to the United States from Europe, your AI assistant might be bound by the EU’s stricter privacy protection laws, limiting the depth of the analysis you receive and leaving you with far less insight about George and his motivations.
Cognition is power, and a range of individual and societal factors will influence how we balance that power. Different regulatory regimes, willingness to participate in the digital economy, and the ability to afford better products will affect the unequal distribution of cognitive power. None of these imbalances will be easy to correct—and, as a society, we might decide some should remain—but we could start by taking a global view of AI and the flow of personal data. Perhaps fifteen to twenty years from now, we might sign an accord to limit asymmetric awareness in AIs designed or hosted in another region, starting an era of cognitive balance in which the power of knowledge is constantly renegotiated, tested, violated, verified, and reset.
TRUTH AND TRANSPARENCY FOR TRUST
Human values and trust each rely on a certain understanding between people, one that requires a minimum threshold of trust. Those levels can vary, of course. We might not need to know the mechanic who fixes our car, and we probably don’t worry about whether he signed an oath to uphold the best auto maintenance practices, but we certainly expect both from the surgeon about to operate on us. Regardless of whether we want to know precisely what the doctor will do as she digs around our innards, we need the assurance that she’s subject to a higher requirement for transparency and explicability. Absent that, we lose trust in the process and the person.
To maintain similar levels of trust and shared values—and, by extension, to retain an appropriate balance of human-machine power—the increasingly pervasive role of AI will raise the threshold we demand of transparency and “explainability,” as it’s referred to by developers. The use of AI-powered policing in Los Angeles and probation systems in US courts has raised objections from constituents who demand a right to know how they are being ranked, rated, and assessed and what the logic was in the assessment. Yet, as Wade Shen noted in Chapter 4, finding a way to make an AI system explain the how and why of its decision is a difficult challenge, one in which DARPA has invested millions of dollars. The project has significant ramifications for military leaders who seek to understand why a system recommends an action that will have deadly consequences. But it also plays out in day to day life, for example as federal investigators and Uber’s scientists try to figure out why an autonomous car hit a pedestrian in Phoenix. As self-driving vehicles become more commonplace on streets around the globe, will the systems that operate them be able to explain why they made decisions that led to damage or death? If they can’t, would we ever have enough trust to remove the steering wheel entirely?
Given its June 2018 effective date, the EU’s data-protection directive looked much further into the future than most existing AI-related regulation. Its one-size-fits-all approach ruffled feathers, as many applications don’t need perfect explanations. In fact, society might prefer less-than-perfect explanations at times—say, if a sophisticated medical diagnostic AI works better than any other system to identify an illness, but is so complex we don’t fully understand why it’s so effective. We might want it to enhance our well-being, perhaps even save lives, but we can’t be certain of the precise reasoning that gets us there. Legitimate questions about how much trust we place in that system, how sound it is over the long term, and its unintended consequences might become moot, because the regulations threaten to stunt it from the start.
Yet, Europe also stepped up as a white knight against the massive market power of the Microsoft, Google, and others. While the region might not produce one of its own Digital Barons given its regulatory leanings, it has started to unify a single digital market with 500 million people and has set out plans that could produce a sort of “digital-commons baron.” The EU’s latest goal to create a collection of large, public, and open data sets for training AI systems could provide the fuel for a vast array of new models and applications, including many we have yet to even imagine. Even the most AI-savvy companies have more data than they know what to do with, much of it irrelevant to their business mode. (In 2018, one online service was asking prospective hires how they might think about putting its superfluous data to good use.)
Creating and controlling those widely available data sets will only expand the EU’s existing influence on transatlantic mindsets and business models, and many observers expect markets around the world to adopt forms of its data-protection regulations. Furthermore, without the transparency and explainability at the heart of the EU rules, we have a much more difficult task ascertaining a system’s veracity. We can gauge the outputs and measure how they change with different inputs, essentially distilling a better sense for the integrity of the final results. Yet, one wonders how policy makers and regulators might seek to put boundaries on processes that can only be measured in hindsight, especially when those systems influence our health, guide the distribution of government resources, or conflict with societal and cultural values. For applications that don’t have a direct influence on people’s lives, a retrospective explanation of the machine’s decision-making process might be enough. We might not care at all about some things—how much does it matter if an AI-powered guidance system tells us to take Park Avenue instead of Madison?—but others may get under our skin in a deeper way.
As these applications play a deeper role in individual lives, they require deeper levels of trust, and thus confront us with a series of trade-offs, a continuum along which benefits are measured against ignorance. In 2018, millions of people put their trust in online dating sites, trusting the algorithms of Match.com and others to find compatible mates. They trust the process, in part because it’s not any more error-prone than the old-fashioned method of randomly meeting someone and getting to know them better. Fast forward a couple decades, when matchmaking systems have more data and deeper training in human compatibility, and we might want a little more explanation about how the systems make those matches—or, perhaps more realistically, why it didn’t make a certain connection.
Perhaps by then our AI assistants will pick up on imperceptible cues and data during a first date and identify it as an inevitably flawed pairing. He might display subtle physical cues of compulsive lying; she might have left faint data trails of an unfaithful past. Yet, the date went well, and they felt like it scored on all the personal, physical, and emotional attractions they hoped for. If they decide to ignore the machine, they might head down a path doomed to heartbreak. If they follow its advice and decline the second date, they might miss out on a serendipitous connection that defies the odds. Absent a clear explanation of why the machine questions the connection, how do they trust it to make that call? And what if the company’s AI platform begins to identify all their frailties and decides they’re too risky to match with anyone? Deep learning algorithms might consider matchmaking in a neutral, transactional activity—a statistically determined gateway to a first date, rather than a headlong plunge into a relationship with all its potential for self-improvement and growth.
Of course, nothing is certain in life. Some relationships work, some don’t. Even machine learning and neural networks generate probabilities based on past patterns, not perfect predictability. So, we might shrug off a first date that doesn’t work out. However, we will demand greater insight when, say,
military or political leaders make decisions based on an AI analysis of intelligence. A soldier called to war might reasonably question why political and military leaders made the decision to put them in the line of fire. A soldier’s family and a nation’s citizenry might reasonably demand better explanations and less ambiguity about how and why the AI recommended lethal and dangerous action. (This might be especially true in the United States, after faulty intelligence about weapons of mass destruction launched the second American invasion of Iraq.)
As we approach greater risks to our emotional and physical well-being, we heighten the levels of trust we require in what we can’t understand. The same will hold true in an era of thinking machines. Until AI systems can explain their decisions well enough to foster a deeper sense of trust, people will naturally err on the side of caution and limit the benefits artificial intelligence could deliver. Over time, we will experiment our way toward explainability, feeling out our comfort level with the lack of certainty in one application or another. In some cases, we will take comfort from other situations in which we don’t care about perfect clarity, such has traffic routings. But we also will double down on efforts to enhance our understanding of trust ratings, medical diagnoses, and the many other areas where explainability is essential.
DAWN OF A NEW DEMOCRACY OR DIGITAL FEUDALISM?
Because they rely on a constant diet of data, thinking machines threaten to expand the digital divide between the connected and disconnected. People in developing countries, especially in rural areas, will reap fewer of the benefits offered by AI-powered systems. Those who can afford greater access or deeper interactions stand to build on their advantage, as those who produce an economically valuable footprint are granted the option to barter their data for ever-expanding access. At worst, they at least have the choice to opt out. Even in affluent, data-rich places, artificial intelligence could polarize societies by pushing people further into bubbles of like-mindedness, reinforcing beliefs and values and limiting the sorts of interactions and friction that force a deeper consideration of those principles, something we in American societies have already witnessed. Taken a step further, AI systems could facilitate digital social engineering, creating parallel microsocieties, allowing the broader ability of companies to target job seekers of certain demographics and exclude others, as we’ve already seen on Facebook.‡
