by Olaf Groth
The similarities with China’s social credit system go far beyond scores, though. According to research by University of Texas at Austin sociology professor Sarah Brayne, who embedded with the LAPD to study its predictive policing systems, the databases incorporate a dizzying array of data on individuals and the community alike, everything from neighborhood crime data to family and friend networks to names, addresses, and phone numbers from Papa John’s and Pizza Hut. And yet, few Angelinos understand how much data the LAPD’s systems brings into play, how police deploy it, and how vaguely its use is bounded by a lack of policies and court precedents on the use of these powerful new technologies.
“We believe that there is such a thing as an unreasonable search and seizure” in the United States, Brayne said in an interview. The idea of a police investigator rummaging through paper copies of a suspect’s past receipts, family interactions, and pizza purchases would strike most Americans as a formal search, she suggests. Yet, the same information emerges when authorities look through digital copies of the same, and they can peek in there now with far fewer restrictions. “But just like clearing someone (of suspicion) in the system,” she says, “it’s invisible.”#
Both predictive policing and social credit systems raise all kinds of concerns about AI-fueled abuses of power, and both US and Chinese citizens have pushed back on certain aspects of both. Without the proper guardrails in place, thinking machines could help companies or governments manipulate individual citizens, but the analytical and predictive capabilities of the cognitive machines that underpin predictive policing and social credit can also keep streets safer or establish financial trust in a country without an existing system of financial credit scores. For patients who want to participate, it could help nudge diabetics toward healthier lifestyles and reduce relapse among opioid addicts, as two of the ten finalists for the IBM Watson AI XPrize are setting out to do. It can help process the massively complex atmospheric problems and shore up existing climate change simulations to help us better understand the danger human activity poses to the environment and how to mitigate it. It can help reduce deforestation and illegal logging, as the Rainforest Connection does with solar-powered smartphones, called guardian devices, that it mounts in trees. The devices monitor the forests by sending sounds up through the local mobile-phone network to its AI system in the cloud, which can identify the sound of trees being felled and alert a local response.**
It can even help eliminate so much of the tedium that clogs our everyday lives. After selling his last company in 2013, Dennis Mortensen went back through his calendar to see how many meetings he’d scheduled the prior year. All told, he’d arranged 1,019 meetings, and he had to reschedule more than 672 of them. “I’m 45 years old, in the workforce about 20 years,” Mortenson says. “This view of the future where I’m doing another 20 years of sitting around in some version of an inbox playing email ping-pong, it didn’t seem real to me.” So, he founded x.ai, a startup that developed an AI scheduling assistant that can schedule meetings on your behalf. He used it to set up his interview for this book, and it worked seamlessly—a little notation on the email address noting the difference between a human response and an interaction with the bot. “The more I looked at this particular chore, the more I realized it shouldn’t have been a human job to begin with, but we do plenty of those,” he says. Finding ways to automate those tedious tasks and free our potential “is the only way we can move forward.”
With quickly growing deployment of more sophisticated AI systems, we might gain a vast new control over the tedium of our lives, the threats to the environment, and the level of danger on our streets. Yet, those same systems can go too far and threaten our individual agency and lives. A new power balance will emerge in the coming era of thinking machines—between countries, companies, people, and machines. Already, the world’s two leading AI superpowers, the United States and China, are providing a glimpse of how these different power balances might shape up in the years to come.
SOCIAL COHESION AND SOCIAL CREDIT
For years, John Fargis and his wife, Vida, would return to Shandong to visit her family and, as is customary in one of the most-traditional regions of the country, celebrate the Chinese New Year with veneration for those who came before them. Families welcomed their ancestors on the first night. They ate dumplings with wheat grown from the soil of their predecessors’ graves. A visit to a friend’s or neighbor’s home elicited an enthusiastic recitation of the host family’s lineage. And on it went, until the third night popped with fireworks, ushering the ancestors back to their graves. To this day, the region’s culture remains deeply rooted in Confucian ideals of filial piety and structured society. “It’s deeply traditional, powerful, and rich,” says Fargis, a professor at the Hult International Business School who’s spent much of the past twenty-five years in China, including as Henry Luce Scholar and the first foreigner to gain permission to teach in a Chinese reform school setting.††
Confucianism, that “absolutely exquisite, rich and opaque tradition,” spun the essential threads of Chinese culture for more than 2,500 years. It held up for millennia, against the influence of nearly every foreign philosophy of life or authority. Buddhist thought migrated in from India during the Han Dynasty, around the first century BCE, but it did little to change the central cultural principles of Confucianism. In fact, Buddhism quickly got “China-ized,” Fargis says, and the conventional wisdom came to regard it as a philosophy seeded by Chinese philosopher Lao-Tzu, the creator of Taoism. Centuries later, around the time of the US Civil War, the Qing dynasty put down the Taiping Rebellion (1850–1864), which was sparked by a Christian millenarian sect whose leader, Hong Xiuquan, considered himself the younger brother of Jesus Christ. Throughout its existence, Chinese civilization did a more thorough job of either resisting foreign cultural impregnation, or culturally appropriating those external influences, than virtually any other civilization on the planet, with Confucianism as the bonding internal force.
Yet, by the early 1900s, a growing chorus of criticism began to question the Confucian notion of virtue as the central element of orderly society. The last Confucian exam took place in 1906, seen as a relic of a bygone era and a vestige of a system that couldn’t quite explain a world order in which China was down with the world’s foot on its throat. These imperial examinations, which had been in place for centuries, served as a gatekeeper to government jobs. However, they mainly tested knowledge of Chinese classics and literary style, doing more to maintain the country’s shared cultural, intellectual, and political sensibilities than to ensure an appropriate level of technical and practical expertise in the state bureaucracy. The conclusion of these official examinations and the subsequent end of the Qing Dynasty in 1911 deepened the erosion of China’s Confucian construct and opened the country to the embrace of a new import of foreign thought: Communism. But even then, the deeply rooted traditional ideals of a well-ordered society based on hierarchy, orthodoxy, obedience to authority, and discipline were fortified in a cryptic way by the introduction of Leninism, says Orville Schell, director of the Center on US-China Relations at the Asia Society.‡‡ Just as the imperial dynasty remained the organizing principle of a highly striated society during traditional times, the Party became the preeminent centralizing force during China’s revolutionary Communist period and remains so today during its more recent emergence as a geopolitical and economic powerhouse. And in both the ancient and modern scheme of things, the notion of the importance of a “big leader” at the heart of China’s political and societal environment and a strong unified one-party leadership has endured, Schell says.
More recently, President Xi Jinping’s rise to power re-enforced these notions of the centrality of a single power center guided by an authoritarian “big leader,” a role that Xi has more than fulfilled since his reign began in 2014, Schell explains. With major programs such as the Belt and Road Initiative (BRI), which envisions trillions of dollars of investment in seaports, roads
, railways, and other major infrastructure project across Asia, Europe, and Africa, Xi has written himself even larger than Mao and Mao’s later Party successors. Xi has reinserted more government control and influence in both political and private-sector life as China seeks to enlarge its global footprint and become a new kind of great power “with Chinese characteristics.”
Yet, as Fargis notes, within the bounds of Party authority, Chinese citizens enjoy a certain freedom of activity and, for so many, a soaring standard of living that most couldn’t have dreamed of just a couple decades ago. And that has started to change some of the most deeply held Chinese beliefs, even in traditional Shandong. When Fargis and his wife inquired about the usual holiday visit in 2017, his brother-in-law said he planned to take his family on a vacation to Hainan island in the South China Sea, far away from their home village, instead. “If you think about what the missionaries couldn’t achieve with their efforts to convert folks in Shandong to Christianity; what the Japanese military couldn’t achieve; [and] what Maoism couldn’t achieve to erode Confucian tradition,” he says, “arguably, technology-driven prosperity is now achieving.”
This comparatively new prosperity is chipping away at a deep cultural foundation, even as it is expanding upon an incomplete economic foundation—one that presents possible drawbacks but opens even more tremendous opportunities. For example, most urban Chinese use Alipay and similar mobile phone apps to pay for things, substituting the touch of a button for cash and the credit cards that most citizens never had in the first place. China already has moved well beyond most of the Western world when it comes to mobile payments. Viewed from the perspective of a nation rushing rapidly forward while adhering to the Party’s vision of its tradition, culture, and economy, the country’s emerging social credit system makes more sense.
CREDIT WHERE CREDIT IS DUE
Alibaba’s Zhima Credit (or “Sesame Credit”) emerged as one of several pilot programs for what’s expected to eventually develop into a national social-credit program that would include more than just financial information. The real push behind the initiative began in January 2015, when the People’s Bank of China issued eight provisional licenses to private companies, hoping to encourage more widespread access to and use of credit scores. While the companies said they’d made progress, the PBOC had not fully licensed any of the eight systems three years later, and more than half of China’s citizens still did not have an extensive enough personal financial history to borrow from formal financial institutions, according to a policy brief from the Peterson Institute for International Economics.§§ That left in place a vast, informal lending network throughout the country, and Beijing had little insight into how indebted its citizens were and whether repayment trends were improving or worsening. So, absent the benefit of an established credit rating system like the one in the United States, they sought to create one by scraping e-commerce, social media, and other online data from applications such as WeChat and Alipay.
Yet, the service aggressively expanded beyond mere financial credit “to serve as a stand-in for an individual’s trustworthiness,” the Peterson Institute brief noted. Zhima Credit gathered a wide array of financial and personal data to generate a score on an 800-point scale for individuals. In addition to information about someone’s online shopping habits and ability to pay bills on time, it collects demographic data and associations gleaned from social networks. So, a twenty-eight-year-old pregnant woman might receive a better “rating” than an eighteen-year-old man who buys a motorbike. Someone who has 700 Sesame Credits would be considered extremely respectable, whereas a score of 300 could trigger a range of social repercussions—anything from restrictions on first-class train and international air travel, to acquaintances ostracizing them to avoid being tainted by association. It eventually became one of the most extensive individual data-collection efforts the world had ever seen, with almost 200 million Chinese residents enrolling and being scored on their trustworthiness in transactions and relationships to date.¶¶ And that’s before the system even becomes mandatory in 2020.
These and other smaller social credit programs pulled in a remarkable array of data types as well. Even before any purchasing or lending activity takes place, factors such as age, parenting, and social network would yield differences. One local program in Zhejiang even encouraged people to report their neighbors’ breeches of social norms. The program, called “Safe Zhejiang,” awarded discounts at high-end coffee shops or other perks in exchange for reports on anything from traffic violations to domestic disputes. Launched in August 2016, it reportedly had about 5 million users by the end of the following year, but it had met with stiff resistance from most residents who didn’t care to be forced into surveillance of their neighbors or feared retribution for reporting concerns. Even some of the authorities in the province balked, worried it might undermine them.##
Despite occasional resistance to some of the early initiatives, the push to develop the national social credit system continued unabated into 2018. Mainstream Western media portrayed the system as a means to monitor and control the population, especially minorities like the Uyghurs, a Turkic people of mostly Muslim faith in Xinjiang, a semiautonomous province in China’s outer northwestern region. Reports have shown that conspicuous activities like congregating groups leads to deductions in the point system. Local law-enforcement officials can access an individual’s status through smart eyeglasses that identify citizens through facial recognition and, thus, facilitate more arrests.***
Yet Western concerns about the invasiveness of these technologies and the government’s use of them to nudge people’s behaviors don’t raise widespread concerns among the Chinese populace. For one, the social contract underlying this use of smarter technologies is different in China, where the long-standing Confucian tradition of deference to authority in the name of stability still holds sway. And as multiple Chinese academics, AI developers, and entrepreneurs noted in interviews, today’s residents are more optimistic about the power and potential beneficial aspects of advanced technologies. “If you look at the Chinese history, just in the last 40 years, . . . the people who embraced change benefitted the most,” said Ya-Qin Zhang, president of Baidu and the former head of Microsoft Research Asia. “Plus, there’s a constancy of direction from government, so even people who come back to the [technological] change later are winners.” And because of that embrace of technology—along with a general sense that the most troubling aspects of AI remain distant possibilities—little discussion of the potential downsides occurs in China’s mainstream media or public discourse, although fluent global information flows will soon bring more of those issues to the forefront, says Hsiao-Wuen Hon, the head of Microsoft Research Asia.
Yet, even for those who accept the concept of the social credit score, unresolved questions remain. For one, experts suspect a social credit system could lead to the formation of new social classes, as people with good ratings shy away from those with poor ratings. More immediately, though, what sort of recourse do citizens have to rectify mistakes or otherwise challenge their scores? Even in the United States, where credit agencies Experian, TransUnion, and Equifax hold oligopoly positions, recourse to address poor or erroneous credit scores is spotty at best. In fact, simply requesting a copy of one’s credit report more than three times in one year triggers a score reduction. And, contrary to the complex procedures and opaque rating systems employed by the US systems, the Chinese government has openly stated its approach and philosophy: Counteract corruption and other untrustworthy patterns of behavior while fostering greater reliability in economic and interpersonal transactions.
But, viewed through the lens of western ideals of civil rights, as the AI-powered social credit system closely examines, quantifies, and makes public not just criminal or financial activity, but demographic characteristics and the “rough edges” of interpersonal relationships, how close does it get to becoming a digital branding iron? Participation currently is voluntary, but the go
vernment already has put restrictions on travel for those who fail to pay bills or engage in other unsavory activity. If it becomes mandatory in 2020, as initially planned, how will it affect the economic existence and social dynamics of almost 1.5 billion Chinese citizens? At some point, too much digital control might trigger unintended consequences that decrease, rather than increase, stability.
PREDICTIVE POLICING
In late 2017, hoping to illustrate just how vast a surveillance network Chinese authorities have built, BBC reporter John Sudworth requested and was granted rare access to one of China’s high-tech law-enforcement control rooms. Officials agreed to temporarily mark Sudworth as a person of interest in their database and then, loose on the streets of Guiyang without being physically shadowed, he set out to see how long he could avoid detection and apprehension. In a city of more than 4.3 million residents, he lasted just seven minutes until police nabbed him.†††
Dozens of countries around the world have started deploying a range of AI-powered technologies to monitor people, everything from facial recognition to gait analysis to speech-pattern analysis. While most Chinese citizens expect and even shrug off widespread surveillance initiatives there, few US residents realize just how personal and how deeply integrated AI-backed surveillance has become in many of the country’s largest cities. In some parts of Los Angeles, for example, street cameras capture the faces of people in the vicinity when crimes were committed, logging innocent passersby into the system. Cross paths with criminal activity more than once, and the score behind a person’s name in the system goes up, heightening their potential interest to police. This kind of correlation is helpful for police, who don’t allege that presence means involvement in illegal activity. But it can lead to troubling situations. In one case in Fresno, California, an AI system used by authorities assigned threat levels to residents based on social media posts and billions of other commercial records. Officers responding to incidents would see conclusions drawn by algorithms that public citizens couldn’t review, according to Matt Cagle, an attorney at the American Civil Liberties Union (ACLU), which is suing the LAPD. Speaking before a California Assembly committing hearing, Cagle said: “When one City Council member found himself flagged as an elevated threat, he had no way to determine the basis for that decision. There were no rules governing the use of that system.” The lack of clear guidelines allows officers to label or stigmatize residents, he and the ACLU argue, and no regulations exist to require the LAPD and other forces to expunge records, anonymize faces, or label individuals as mere contextual elements in a crime area, like buildings or trees instead of potential suspects.‡‡‡
