Mind in Motion

by Barbara Tversky

  Recognizing faces is fundamentally different from recognizing objects. For one thing, whose face is important, but which chair or which hammer is usually not. Chairs and hammers are typically interchangeable, but individuals, for better or worse, are not. Silhouettes work for recognizing chairs and giraffes and shirts and bananas, but shape won’t do for faces. All faces have more or less the same shape. Recognizing faces depends on what’s inside the shape, but not just the features themselves, again because we all have those, and so do dogs and monkeys. Key for faces is the qualities of those internal features, eyes, nose, mouth, and how they are arrayed. That’s a hard problem, one that the fusiform gyrus, tucked underneath the cortex, is good at solving. Other areas of the brain, the occipital face area (OA) and the superior temporal sulcus (fSTS), respond to face parts in any configuration, the kinds kids have fun making, but the fusiform face area is sensitive to properly configured faces.

  Now a few interesting facts about memory for faces. Memory for faces is specific to faces, independent of other abilities. It is stable over time, and doesn’t seem to be trainable. Perhaps not surprisingly, it turns out to have a strong genetic base. Recognizing faces improves with age, especially through adolescence (ten to twenty years), and peaks at thirty-two.

  Despite the importance of individual faces in our lives, not everyone’s brain can distinguish and recognize individuals. What do Brad Pitt, Oliver Sacks, and Chuck Close have in common? Sure, they’re famous. And supremely talented. But the exotic feature they share is prosopagnosia. Practice a bit and the syllables will roll off your tongue. Yes, it’s Greek; prosopon is Greek for “face”; agnosia, for “not knowing.” Face blindness. A problem in the fusiform gyrus. People with prosopagnosia can sense they are looking at a face but do not know whose. Painfully, some can’t even recognize their spouses. If you suspect prosopagnosia, don’t take it personally. It’s the brain’s fault; apparently, the neural circuitry that responds to faces in prosopagnosics differs wildly from the neural circuitry in those who recognize faces effortlessly.

  Not recognizing faces is not just embarrassing but a serious problem. Prosopagnosics develop compensatory mechanisms, analyzing and memorizing critical features, attending to other aspects of individuality like voice, body, and clothing. Fortunately, more wonders of the brain, the inability to identify individual faces doesn’t interfere with the ability to detect emotional states. Fascinating, isn’t it, that recognizing identity and recognizing emotion are computed independently, in different areas of the brain?

  Faces and emotion

  If cooperation is key to the success of our species, emotion is key to cooperation. To work with you, I need to trust you, to like you, to think positively of you. There are widely diverse views of emotion, but there is one thing on which they agree: emotions can be readily divided into positive or negative. Like/dislike. And quickly expressed in behavior. Positive means approach; negative, avoid. Emotions come on quickly, expressed by the face, the body, the voice. Emotions are rapidly recognized and hard to suppress. e e cummings famously opened a poem, “since feeling comes first,” an insight later supported by research (and contributing the Third Law of Cognition). People viewing random meaningless forms have gut feelings about them before they know whether the forms are familiar. And they feel more positively toward the forms they’ve seen even when they fail to recognize them. Recognition and emotion are separate systems that are slow to talk to each other. The essence of emotion, positive or negative, is a rapid distillation of the essence of social glue, approach or avoid. This fundamental dimension has been anointed with a term from chemistry, valence. In chemistry, valence underlies the bonding of molecules; in psychology, the bonding of people. Emotional valence is the great reducer, the bottom line. Third Law of Cognition: Feeling comes first.

  Emotions not only come on quickly in ourselves and are sensed quickly by others, they are also contagious. As are yawns, even for dogs. Remarkably, dogs are more likely to catch yawns from familiar people than unfamiliar, strong evidence for social empathy. Both people and dogs seem to catch stress from others, for example, we and they respond similarly to crying babies by increasing cortisol levels, a response to stress. It’s not just emotions that are contagious but also emotional states, like stress. If experiencing others’ emotions is the starting point for empathy and if emotions are contagious, then we seem to be wired for empathy. It can even be located in the brain, in the catchy TPJ (temporal parietal junction, just behind the ear). Note the qualifications: the ifs and the seem. Not all emotions, not all people, not all the time. And not with the same intensity.

  Here’s something else to try: walk down the streets looking at the faces coming toward you. Smile. Most likely, your smile will be returned. Like many people, I squint when there’s sun in my eyes. Squinting makes me smile, though I’m not aware of it and often wonder why so many people walking toward me are smiling at me.

  Given the crucial role of emotion in social interactions, the diversity of views of emotion is not surprising. As we noted earlier, people like to categorize and even need to categorize; there are simply too many different things in the world. One venerable view, personified and immortalized in the animated film Inside Out, is that there is a small set of basic emotions: anger, fear, disgust, sadness, happiness, surprise. That these are universally recognized from facial expressions, that they have an evolutionary history evident in other species as social signals, and that the basic facial expressions are distinguished by the activity of specific and specifiable muscles. Experiments worldwide using photos of faces frozen in these poses have shown that judgments of these emotions are rapid and that deliberation does not improve judgments—strong evidence that judgments of basic emotions, like recognition of people and things, are direct and unmediated.

  Others agree that emotions are rapidly expressed and recognized but claim that, like flavors of food, there are countless nuanced emotions. That they are hard to name. That appraisal of them is tempered by context and culture. That we can and do experience and can and do express more than one emotion at the same time, disgust and surprise, sadness and anger, fear and happiness. That emotions are expressed in the body as well as the face and that, not infrequently, the emotions displayed by the body override those on the face.

  Taste, too, is said to have five basic categories: sweet, sour, salty, bitter, and a recent addition, umami (savory or brothy or meaty), with several vying to be sixth or seventh. Now, we must explode an urban myth: these basic tastes are not sensed by different parts of the tongue. Even more significant, these tastes appear in combinations, and those combinations don’t begin to capture the myriad different flavors people can sense, as chocolate lovers and wine tasters and mole poblano makers and curry consumers will attest.

  Categories are so much easier than continua. They are so much simpler, there’s so much less to be kept in mind. Distinct and different groups rather than many subtle gradations. This tension between a few basic categories and the myriad variations that we can sense in the world holds even for a far simpler quality, color. There is evidence for a basic set of colors whose centers or best examples are agreed on across cultures and whose names enter languages more or less systematically, though there are disputes about the order: first dark and light; then red is added, then green and yellow followed by blue, then brown, then purple, pink, orange or gray. In the case of color, there is evidence for heightened sensitivity for the basic colors. Basic colors notwithstanding, people recognize many subtle shades of colors whose names carry neither transparency nor agreement. Crayola crayons, used by generations of children and nostalgic adults, has expanded its box of colors with these: melon, mahogany, and manatee, and also wild blue yonder, bittersweet shimmer, purple mountains’ majesty, and unmellow yellow, the last barely distinguishable from laser lemon.

  Returning to emotion, there are the Ekman-Friesen six (+/–): anger, fear, disgust, sadness, happiness, surprise. Yet, our faces and bodies and words and to
nes of voice express far more emotions and more articulated emotions than the six proverbial basic emotions (recently recognized by Ekman, who has expanded the list). Think sympathetic, apprehensive, annoyed, disappointed, remorseful, suspicious, anxious, proud, pleased, bored, hostile; the list is long and the list is only words. Describing emotions is not always easy or straightforward. Appraisal of emotions depends not just on the face but also on how they emerge and fade on the face and the body and the voice. And, as we said, appraisal of emotion also depends on the larger context and on the culture.

  Film lore has it that Kuleshov, an early Soviet filmmaker, paired a frame of the expressionless face of a heartthrob film star in a sequence with a frame of either an attractive woman reclining seductively on a sofa or a bowl of soup or a child in a coffin. Viewers interpreted the same face as desire, hunger, or sorrow depending on the juxtaposition. The effect (alas, like so many) sometimes replicates, sometimes not, but there is plenty of other research showing that interpretation of emotions from faces, both frozen and animated, is tempered by culture, by voice, by words, by the body, by others, by setting. Long as this list is, it’s undoubtedly incomplete.

  Do the eyes have it?

  Some old proverbs just might be true, in this case the one about the eyes being the window to your soul. If you are interested in how well you can read faces, there’s a five-minute test you can take and that thousands of others have taken, the Mind in the Eyes test. It shows you photos of only the eyes and eyebrows of real people and asks you to select which of four emotional states the eyes express. On average, more-educated people outperform less-well-educated people, women outperform men but only slightly, and neurotypical people outperform people with conditions that compromise recognition of emotion such as Asperger’s, schizophrenia, and anorexia. An analysis of more than eighty-nine thousand people who completed the test and volunteered their genetic information confirmed a genetic basis. A separate study showed that performance of identical twins is more similar than performance of unrelated people, more evidence for a genetic basis.

  What is it about the eyes? There is still much to learn. And, in fact, the eyes learn, by scrutinizing the world. When eyes open wide, they take in a wide range of information. When eyes narrow, they achieve sharper acuity, but for only the small region in focus. Squinting is all too familiar to those of us who are near-sighted. Try both. New research connects the information-gathering functions of the eyes to emotional expression. Wider eyes appear for such opposing emotional states as fear and surprise. Both fear and surprise naturally stimulate a broad search for information. Other emotions associated with wide eyes include awe, puzzlement, cowardice, anticipation, and interest. By contrast, the unrelated emotions of disgust and anger are associated with narrowing of the eyes; the sources of disgust and anger are quite clear, in focus. Other emotions associated with narrow eyes include annoyance, disapproval, suspicion, and pride. When people viewed photos of faces in which the expressions of the mouths were discordant with those of the eyes, people’s judgments of emotional state followed the eyes, though the effects were reduced slightly due to the discordance.

  Faces and traits

  In a surprising and disturbing series of experiments, students were shown a series of pairs of photos of faces for a fraction of a second and asked to judge which of the two people was more competent. The photos were the candidates for eighty-nine previous gubernatorial races. Students were also asked whether they recognized any of the faces, and if they did, those data were dropped. Remarkably, the candidate whose face was judged more competent won better than chance, around 55 to 58 percent of the time. These judgments were rapid, as requested, and a replication asking participants to carefully deliberate their judgments actually reduced predictability. Competence judgments don’t just account for election results after the fact, they also predicted more than 68 percent of the gubernatorial and 72 percent of the senatorial races in 2006. And this happens in other countries, not just the United States. Rapid social judgments of competence or dominance (the judgments of these traits turn out to be related) from faces appear to have broad consequences on our lives; they correlate with sentencing decisions, hiring decisions, salary, and military rank. Our brains seem to be wired to make these evaluations automatically, as we casually go about our business, without an explicit intention to judge.

  That was the surprising part. Now the disturbing part. These judgments have no validity. That is, people judged to be more competent or dominant are not necessarily more competent or dominant. The judgments are not borne out in behavior. There is some good news, however. The biases can be countered by other information. Remember that these are studies on frozen faces. In real life we can hope to have more information about people than a single facial expression for important decisions and other information can counter the biased judgments. However, we must be wary and remember the facts. Rapid judgments of frozen faces by participants in academic laboratories do predict voting decisions made by other people, and seem to predict other life-changing decisions like employment and promotions.

  Bodies and emotion

  Bodies, too, express emotion. Bodies have large advantages over faces: they are larger. Bodies can make bigger expressions and they can be seen from a greater distance. Normally, bodies and faces work together, as integrated units. Conveniently, experiments can separate and realign face and body. When face and body are congruent, expressing the same emotion, appraisals are more accurate. If face and body are incongruent, expressing different emotions, the body carries more weight than the face in judging emotions.

  When they conflict, emotion expressed by the body can override and even reverse emotion expressed by the face. A striking example comes from competitive tennis matches. Players typically react strongly to points they win or lose. When a winning body is paired with a losing face, people see the reaction as positive. And vice versa: when a losing body is paired with a winning face, people interpret the reaction as negative. Impressions go with the body when the face and the body conflict. In these cases, the face alone, without the body, even when viewed close up in a photograph, is not reliably judged for positive or negative affect. For this situation, what comes through from the face is intensity, but not necessarily valence.

  Faces cannot be discounted as carriers of emotion. Tennis matches require forceful actions of the body and are played to a crowd, so the dominance of the body over the face is natural. Tennis matches are only one of a vast range of situations that elicit emotion. Many encounters, both intense and casual, involve people sitting or standing face-to-face, where emotion is carried primarily by the face and voice and less by the rest of the body. Human communication is typically redundant, presumably at least in part to reduce error; if you miss part of a message, say, in a noisy restaurant, you might get another part, enough to understand. Like many expressions and communications, emotion is expressed in the face, voice, and body and redundantly even within each mode. People who are blind can grasp emotion from voice, both prosody and words; people who are deaf can grasp emotion from face and body.

  Bodies and action

  Bodies do more than express emotion. As we saw in Chapter One, bodies gesture, creating countless communications. They also act in the world, doing the myriad things that need to get done, also discussed in Chapter One. Of course, the brain is involved: the EBA, the extrastriate body area, that is, the lateral occipitotemporal cortex, responds selectively to bodies and body parts, but not to faces and other things such as objects and animals. Significantly, still photos of bodies caught in motion, such as throwing a javelin, also activate an area known to respond to actual motion, MT/MST, or medial temporal/medial superior temporal cortex. Photos of static poses do not activate that area, a strong indication that moving bodies have a different and special status from stationary ones.

  Bodies convey action, but they also convey intentions of action, and in exquisite detail. In one set of studies, people were shown videos of the hand of a
nother person reaching to pick up a block for different reasons: to take it for themselves, to give it to someone else, or to grab it before a competitor grabbed it. The videos were stopped before the hand actually reached the block. Despite the subtle differences in the actions of the body, viewers could reliably distinguish those three possibilities. The differing intentions were apparently evident from the shape and action of the hand and the direction of the actor’s gaze.

  In continuous natural actions such as making a sandwich or washing dishes, the head and eyes of the actor shift to the next object to be acted on before the hand begins to reach. The hand is often busy completing the previous action while the actor is already preparing the next action. So, the head and eyes of an actor show the actor’s intent and provide good clues to the next action.

  Just as for the face, people, tiny ones and adults alike, quickly process the direction of eye gaze and the actions of hands of others and use that information to infer and understand what others are thinking, doing, or intending to do. Yet another example, an important one, of information conveyed by the body that others absorb and even act on without the need for language.