Mind in Motion

by Barbara Tversky

  MEANING

  I’ve shown you some of the amazing mental gymnastics our minds can perform. Those mental gymnastics transform what we see in the world and what we imagine in our minds into countless ideas, from the elementary and mundane needed to catch a ball, cross the street, or pack a suitcase to the spectacular and arcane used to create magnificent buildings or fantastic football plays or theories of particle physics. Marvelous as they are—and they are marvelous—buildings and football plays and zooming particles have a physical presence of one sort or another. But spatial thinking has even more wonders to reveal. Spatial thinking underlies how we talk and how we think, about space to be sure but also about time, emotions, social relations, and much more. Turn the page.

  CHAPTER FIVE

  The Body Speaks a Different Language

  In which we consider how actions of the body, especially the hands, turn into gestures that act on thought, our own and others, and provide the social glue underlying cooperation.

  The Winter’s Tale act 5, scene 2, First gentleman: “There was speech in their dumbness, language in their very gesture.”

  WATCHING PEOPLE, EVEN FROM AFAR, YOU KNOW WHAT IS HAPPENING. You know what they are doing and you know what they are feeling, happy, angry, energetic, anxious—no need to hear. You know their intentions. You know their relations. One couple huddled arm-in-arm, another erect and apart. You see people in conversation. One head tilts, quizzically. Someone leans forward, in confidence. Another leans back, yielding the floor. One shakes a fist at another, the other steps back. One minute the exchanges are slow and relaxed; another they are rapid and intense. What others are doing tells you what to do. You go to the back of the line at the theater, skirt around teams fixing potholes, and cross the street to avoid a fight. These coordinated actions of bodies are often as subtle, as highly articulated, as exquisitely timed as a virtuoso string quartet. They are actions, but not actions on things, like making dinner or getting dressed. They are unlike the other myriad actions that we perform throughout the day that change things in the world.

  Our bodies perform an astonishing assortment of actions. We prepare food and consume it; we dress and undress; we arrange books, clothing, food in cabinets and closets; we assemble furniture and sew clothing; play pianos, flutes, and drums; operate vacuum cleaners, cars, and bicycles; we walk, run, dance, climb trees, chase dogs, shoot hoops and rapids, do yoga, and ski. Some are actions by hands that can even change the world, like assassinations; others are actions by feet that simply change where we are in the world. But there’s another set of actions that neither change the world nor our locations in it. These actions change thoughts, our own or those of others. These are gestures. Intriguingly, many gestures are abbreviations of the very actions that change the world or our locations in it: putting, taking, raising, pushing, turning, splitting, mixing, and countless more. As gestures, they express actions on ideas rather than on objects. We talk that way, too, as if ideas were objects and thinking were action on objects. We pull ideas together, put them aside, tear them apart, turn them over or inside out.

  Despite the extraordinary expressiveness of the body, the face, and the hands, when we think about thinking, we typically think of words. We teach them to our children, we write them to our friends, we post them on refrigerators, we speak them with strangers. We learn the rules of grammar and composition for organizing words into sentences and sentences into discourse of all kinds. We consult dictionaries for meanings of words and style manuals for techniques of composition. Not so for gesture. There isn’t an authoritative dictionary for the meanings of gestures as there is for the meanings of words. There aren’t rules of grammar for organizing gestures into sentences; there aren’t sentences.

  Gestures come first, before words, both in evolution and in development. An insightful, if speculative, theory of the evolution of language from monkeys to people begins with actions that are significant in the lives of monkeys, such as throwing and tearing. A remarkable experimental program has found single neurons in monkey motor cortex that fire when the monkey performs one of those actions and when the monkey sees someone, even a human, perform the same action. Mirror neurons they are called. They unite the doing of action and the seeing of action in single neurons, different neurons for different actions. The brain basis for understanding action. Some speculate that actions are also the foundation for language, for expressing action. A truncated version of an action like throwing or tearing could signal an intention to perform that action. The truncated action becomes a gesture. The area in monkey cortex representing the hand overlaps an area in humans that represents spoken language. The theory then speculates that the voice came to take over for the hand, both because it has greater articulatory power and because it can project across distance.

  If gesture precedes language in evolution, then perhaps it can be seen in primates. The trick is to find it in the wild, not the laboratory where “natural” behavior has been contaminated by interactions with humans. In fact, painstaking observation has shown many cases of communicative gestures by chimpanzees and bonobos in the wild. The intentions of ape gestures seem to be requests for attention, sex, grooming, or companionship. Requests to stop some behavior have also been observed. So far, no one has found apes counting or giving directions. Given that there is cultural transmission of tool use and food foraging in apes, it would be exciting if continued observation found cases of using gesture to teach or explain.

  Communicating with the body is ubiquitous but usually implicit. You don’t have to think about it; it happens by itself. Somebody asks you a question you can’t answer. You shrug your shoulders. “How’s school?” I ask D., a five-year-old granddaughter. Her answer: one thumb up, the other down. Body-to-body communication is more direct than word-to-word, performed by one body and understood viscerally by another, often without awareness. I glance at the door, your head and eyes follow my glance. I cross my legs; soon you do the same. As we converse, we increasingly use each other’s words and gestures, a phenomenon called entrainment. Entrainment undoubtedly serves to make sure we understand each other, to create mutual understanding or common ground. It is also a form of social mimicry.

  When we imitate each other, we like each other more. It goes both ways: we’re more likely to imitate someone we like. Mutual imitation encourages cooperation. Social mimicry is social glue.

  But there’s even more to imitation, explicit or implicit. You smile or wince and I feel your pleasure or your pain. I might even smile or wince, automatically, mirroring your emotion. Even babies do this. Emotional mirroring is fundamental to empathy.

  Body-to-body communication goes far beyond mirroring. It is often complementary. At a cocktail party, you spot a group of friends chatting. You approach. The circle widens to include you and you enter. During a seated discussion, one person stands, ending the meeting. At a disturbing presidential debate in 2016, the larger of the candidates added drama to a wordy event by circling in place like a lion about to pounce on a prey, a show of power to the audience and a threat to the smaller candidate. What’s gesture, what’s body language—there’s no way to separate the hands from the head from the body, they’re all connected. Bodies can be seen from afar; whether the person approaching is young or old, drunk or sober, friendly or aggressive is evident from a distance. Faces and hands require closer viewing. We talked about faces in Chapter Two, when we considered the world the body enters. Hands are especially agile, their many joints and muscles performing remarkable feats on pianos and surgery tables and cutting boards and weaving looms. Those supremely articulated movements of hands and fingers also participate in subtle gestures that express subtle meanings. We turn to those now.

  THE HANDS SPEAK

  No less impressive than the large performances of the body, small gestures of the hands turn out to be chockfull of meaning. Even in babies, or maybe especially in babies. Babies gesture to communicate before they speak. Many parents complain of becoming slaves to th
eir babies’ points, requests to take me there or bring me that. Some gestures by babies are less demanding. C., at eighteen months, is exploring her grandmother’s (that’s me) overnight kit. She pulls out a toothbrush and then a small tube she thinks is toothpaste. She tries to open the tube but fails. She hands it to me, that action a request to open it. I say, “C., that’s not toothpaste, it’s lotion.” C. looks at me and rubs her arm up and down as if putting on lotion to show she understands. Another example. A., at the same age, spots a small decal of an airplane on a motorcycle. She makes sure I am watching, points to the picture of the airplane, and then points emphatically to the sky as if to say, airplanes go in the sky. Two-“word” sentences, where one is uttered and the other gestured or where both are gestured, are common in babies just learning to talk. They are also an invitation to an adult to provide the words: “Right, A., airplanes go in the sky.” In fact, such multimedia productions are a harbinger of spoken language. Babies who gesture to communicate early usually speak early.

  Now consider B., an adult who has been blind since birth. She’s been asked for directions, how to get from one place to another. As she speaks, her hands show each part of the route in sequence. She can’t see her gestures and can’t know if you are looking, or know if they help you understand.

  Another example—you see this every day: people walking down the street jabbering, one hand holding a small, flat rectangle, the other one gesturing emphatically in the air. Although we are not party to the conversation, we can see the gestures, but the conversation partner cannot. And we no longer regard such behavior as loony.

  Why do people gesture? The answer is simple. Gestures express so many meanings directly; words take time to find and to assemble. Words are arbitrary. Except for a few onomatopoeic words, words like buzz, hiccup, and gurgle, words bear no relationship to their meanings. It’s all the more remarkable that we learn so many of them so early and so quickly when they are only arbitrarily connected to the meanings they express. Gestures, by contrast, more often than not bear immediate relations to their meanings. C. expressed lotion by pretending to put lotion on her arm. A. expressed “airplane” by pointing to a picture of an airplane. She then expressed “in the sky” by pointing to the sky. What could be more direct than conveying an object by pointing to it or showing how it’s used? These gestures seem to do what words or short phrases do in spoken language. The gestures essentially substitute for words; they’re easier to produce than words certainly at that age, before the fluency that allows words to pop out before the thought has finished—often with regret. In fact, for babies, many gestures of this kind will eventually drop out and be replaced by words.

  The gestures made by B., the blind adult, are different. They accompanied her speech; they expanded her speech, presenting more or less the same information but in a more natural format. Her gestures worked with her speech or more likely her thinking. Her gestures didn’t substitute for words. As she spoke, her gestures sketched the route segment by segment, drawing straight lines for the streets and bending her hand for turns. Strung together, her gestures formed a map of the route. Did her gestures serve her own thinking, or were they meant for her unseen and unseeable listener?

  On the one hand, gestures can represent thoughts that can be conveyed by single words, as when C. rubbed her arm to represent lotion. On the other hand, gestures can serve to create an overall structure in space, as in B.’s gestural sketch of a route. That spatial structure, unlike airplane, can’t be expressed in a single word; unlike in the sky, it is even difficult to express in several words. Her gestures followed a logic quite different from the logic of language. They created a continuous diagram that organized and expressed an integrated set of thoughts. Their structure was not the structure of language. Gestures don’t follow the rules of grammar. It probably hasn’t escaped you that the expression I used to create a dimension of the breadth of meanings for gestures was a set of words describing a pair of gestures: on the one hand, on the other. That pair of gestures creates a virtual diagram in space, a horizontal line representing a continuum of expansiveness of meaning.

  THE HANDS DRAW

  Gestures do so many things. One of them is to draw in the air, and there are fundamental similarities between gestures and graphics such as sketches, drawings, diagrams, charts, paintings, and models. Both gestures and graphics are created by actions in space. Both are used to represent something other than themselves, though on occasion—think painting and dance—they have dual roles: they both represent and are objects of contemplation in and of themselves. Both follow a different representational logic, a more direct one, from that of language. Crucially, both resemble what they represent. Of course, there are differences between gestures and graphics as well. Gestures draw, but in broad strokes, strokes of the fingers or the hands or the body, not of a pencil or fine brush. They necessarily lack the refinement of paintings or sketches. And they quickly disappear. Graphics stay there but stay still—except for animated graphics, and those have problems of their own. Yet another significant difference: gestures are performed in the here and now. Depictions and graphics of all sorts are free of the momentary context of here and now; crucially, they can represent things and events not in the present, things and events that are in the past or future, an advantage they share with language.

  That gestures can show so little and show it so imprecisely forces abstraction. At a minimum, abstraction entails slimming the information, not uniformly but by selecting the essential features of the ideas and eliminating the irrelevant (I hear you saying, But so do words). For gesture, that also means selecting features that can be enacted or spatialized. Graphics, too, force selecting what to show and what to ignore, but they can show far more, sometimes too much, overwhelming viewers and forcing them to search and further select. In contrast to graphics, gestures are fleeting, they don’t hang around to be explored. But graphics require implements, pencil and paper; gestures require no more in the way of implements than the body we carry with us. And quite frequently, the surrounding world. Finally, gestures are actions, often abbreviated actions in the world, and as such better suited to show action than static graphics. These features of gestures—that they use actions in space to create meaning, that they represent something other than themselves, that they can resemble what they represent, that they are abstract and schematic, that they are fleeting, and that they are in and of themselves actions—all these features help us to understand what they communicate and how. And how they affect thought both in those who create them and in those who see them.

  KINDS OF GESTURES

  Everyone likes to put things into bins, to make piles of like things and separate them from unlike things. That is, taxonomies, dictionaries, catalogs, categories. They’re so useful. Putting information into bins and the bins into bigger bins makes everything simpler. But there’s no way to produce a neat catalog of gestures much less of all the ways that our actions in space create meanings. Except for a small set of frozen gestures like “okay,” “thumbs up,” and “high five,” gestures are constantly—indeed, typically—invented on the fly and adapted to the situation. Of course, words, too, can be invented on the fly—when did email and spam become nouns and then verbs? But words tend to be invented from other words, and invented words conform to parts of speech; they’re nouns or verbs or adjectives. There is no syntax for gestures, no grammar, nothing that corresponds to parts of speech. Sentences are almost always invented on the fly in conversation or carefully crafted in poetry, and there is no comprehensive catalog of sentences. There are, however, typologies of utterances and of discourse, and even of gestures. These typologies aren’t rigidly defined; many gestures fall into more than one category. Nevertheless, they are useful. For gestures, the commonly accepted types are emblem, beat, deictic, iconic, and metaphoric. The features that distinguish the types are partly form, partly function, partly semantic, partly a combination.

  Emblems are frozen gestures that are wo
rd-like: the signs for “okay” or “thumbs up” or “peace.” Nodding the head sideways back and forth for “no” and up and down for “yes.” Waving to say hello or goodbye. Emblems typically serve as crisp replies or greetings. As such, they usually stand alone; they rarely combine with other gestures or words to form longer utterances.

  Beats are rhythmic gestures that accompany speech, typically at phrase or clause breaks. They can serve to structure the discourse and advance it; they can serve to emphasize. Although they are regarded as not having semantic content, they often do. The repeated pounding on a lectern timed with each of the faults of the opponent in a political debate are beats. The emphatic hand slices that accompany a list, first, second, third… are also beats, but because those beats usually proceed along a horizontal line in space, they carry semantic meaning by establishing a dimension along which a set of things is ordered, events in time, teams by order in a league, movies by ticket sales. The human mind does like to order and to rank.

  Deictic gestures point. The word deictic and the noun form deixis derive from a Greek word that means to show or demonstrate or prove or point to. Oddly, despite its origin, deixis was first used with respect to words, not gestures. Deixis refers to words like here, there, me, this, that, next, and now, words that rely on the current context, the here and the now, to be understood. The now in the previous sentence is no longer now.