Mind in Motion

by Barbara Tversky

  REPRESENTATIONS AND TRANSFORMATIONS

  Now some jargon. What I’ve been calling ideas psychologists often call representations. Representations are in the mind rather than in the world, though they often come from the world. Representations are regarded as something static, something you can look at and contemplate, something you can alter in your mind. Of course, there aren’t really representations in the brain; it’s just a useful way of talking. A representation captures the information that is central for an idea or a problem. Representations are schematic, like maps. Aerial photographs don’t make good maps; good maps select the information that’s important for the task at hand, say, driving, hiking, or bicycling, and they simplify, amplify, and even distort that information. Roads, for example, wouldn’t be visible at scale on many maps, so they are enlarged. Small bends in roads don’t appear on maps. And maps often add information like names of streets and cities, borders between states or countries, coloring for altitude or depth.

  Representations in the mind can come from outside the mind, from perception, the view you have of the city you’re exploring or the face in the distance you think you recognize or the chess board on the table. What you put in your mind isn’t all that you can see; rather, what’s in your mind is abstracted from what you see, usually embellished with interpretations from the inside, like the name of the person or the roles of the buildings, bank, department store, church. Representations can also come entirely from the inside; you might be thinking about chess moves or rearranging the furniture in your living room or the route you want to take home to do your errands. Representations come in many formats: Some lean toward depictive, like the chess board or music score or scene in front of your eyes or the living room in your mind. Others lean toward descriptive, like the words of a song or your lines in a play or a list of errands. Still others might be musical, the song’s melody, or motoric, the finger movements to play a piece on the piano or open a lock or the body movements to perform a dive or a tennis serve. Many representations are essentially mixed media.

  Although the number of representations is boundless, the number of kinds of representations is not. And there are varieties and combinations of each kind. Representations can be regarded as internalized perceptions, so visual representations carry some visual properties like color, spatial representations carry some spatial properties like composition or size or distance, auditory representations carry some acoustic properties like pitch, and verbal representations carry semantic and syntactic properties.

  What I’ve called actions on ideas psychologists often call transformations. Sometimes we use the term operation, lingo borrowed from computer science. Just as there are countless real-life actions on real objects, there are countless mental actions on ideas or transformations of representations. Recall the list, a partial one: pull together, raise, toss out, arrange, and so on. Some transformations are loosely tied to domains like arithmetic or cooking or music or language or gene splicing or chess, but many are generic. And so very many of them are based on actions by the body in space, whether actual or imagined. In fact, a useful way to think about mental transformations is as internalized actions. Just as representations can be regarded as internalized perceptions.

  MENTAL ROTATION

  Now decide whether the Fs and Rs and 5s in Figure 4.1 are the same or mirror images of each other. Then think for a minute how you decided.

  FIGURE 4.1.

  Mental rotation entered the world in 1971 to great fanfare. You got a feel for it when you decided whether the Fs or the Rs or 5s were the same or mirror images of each other. Doing so is a very different kind of thinking from the thinking needed to decide that because Socrates is a man and all men are mortal, Socrates is mortal. It is entirely different from deciding which movie to see or whether or not to buy a dog. It is not so different from adding 5 + 7 to get 12 because it turns out that mathematical thinking has a layer of spatial thinking. You might have done what I did, imagine a mental ruler with numbers going horizontally from 1 on the left, each marked with a vertical tick, longer ticks for 5s and 10s. My mind slid to the 5 tick, then to 10, then 2 more to get 12.

  Mental rotation is a distinctly visual-spatial transformation. It has been likened to watching something actually rotate in space. Now some gritty detail on the first dramatic study. Rather than letters or numbers, that study used pairs of ten linked cubes with two bends, each a different direction. Neither easy to describe nor easy to mentally rotate. The figures appeared at different angles, either in the picture plane or in depth. When rotated to the same angle, half were the same and half were mirror images. Participants were shown many pairs of those figures, half same, half mirror image, over many days. The angular difference between the pairs varied from 0 to 180 degrees. With so much practice, some participants became highly skilled at the task. Their performance became quite regular and they made very few errors. The data of interest were the times to decide whether the two figures of each pair were the same or mirror images as a function of the difference in angle. One way to decide is to mentally rotate the pairs to the same angle and to “see” if they overlap or diverge. If people do mentally rotate the figures into correspondence, then the greater the difference in the orientation of the two figures, the longer it should take to respond. Exactly that happened. Twelve disparities in orientation, varying from 0 to 180 degrees, yielded twelve data points lying on a straight line, showing increasing response time with increasing disparity in orientation. It was as if the brain had a turntable that continuously rotated and you simply put a shape on it and turned it on. That striking claim about the mind, however, turned out to be an oversimplification.

  The discovery of mental rotation was especially striking on the background of the field at the time. The cognitive revolution that began in the late fifties and sixties had released the shackles of behaviorism and allowed entry into the mind. The challenge was (and still is) to show what was happening in the mind by putting the mind into the world. Yet the field of thinking was (and in many ways still is) dominated by language. It’s easy to get people to read or listen and to talk or write and then to make inferences about their thinking from their words. Also, when people think about thinking, they think they are thinking in words. So how to study imagery, spatial or visual? Or auditory or olfactory or tactile? How to get it out of the head and into the world where it can be objectively observed? Words can’t do justice to images. Drawing won’t work, it’s too hard for many to draw their imagery. Then there are so many people who say they have no imagery, so spatial and visual thinking can’t be tied to subjective experience. Reaction times to perform spatial or visual or acoustic tasks provided a way to put the mind into the world. In the case of mental rotation, a continuous spatial process perfectly predicted the times.

  These remarkable findings seemed to imply that people smoothly mentally rotate the figures into congruence as if they were watching the figures rotate into congruence. However, there are perfectly intelligent people who can’t perform the task easily; several of them dropped out of the experiment. Others say that they don’t smoothly mentally rotate; rather they look back and forth part-by-part. Eye movements confirm those reports, going back and forth between segments of the figures as if checking part-by-part. Of course, this is still visual-spatial reasoning even if it’s piecemeal rather than holistic. Those whose eye movements and subjective experience indicated holistic mental rotation performed better on a battery of tests for spatial ability than those whose eye movements and subjective experience suggested part-by-part comparison. Since then, a variant of the mental rotation task has become one of the major measures of spatial ability. More on spatial ability soon.

  Mental rotation, imagining something in front of the eyes in a different orientation, is more than an arcane skill studied in the laboratory. We use it when we are lying down, when we recognize objects that aren’t upright, and when we read at an odd angle. We use it when we solve puzzles or organize shelves and drawer
s or pack a suitcase or put together a bicycle or assemble a piece of furniture or put a key in a lock. Surgeons, plumbers, electricians, football coaches, mathematicians, fashion designers, urban planners, gardeners, physicists, fire fighters, architects, basketball players, interior designers, dentists, and so many more use mental rotation and other forms of spatial reasoning regularly in their work. Or their play. But not to worry if you’re at the low end. Remember that mental rotation can be done in many ways, piecemeal or by trial and error. What’s more, although some fortunate folk seem to be born with the skill, it can be acquired, in the usual way: practice. Moreover, lawyers and journalists and historians and accountants and executives and philosophers and poets and translators don’t seem to need mental rotation in their work.

  The intimate connection between mental actions and physical ones is evident watching people attempt mental rotation tasks. When they try to solve mental rotation problems, many people spontaneously rotate their hands as if rotating an object. When they do, their performance is faster and more accurate. As they practice and get better at the mental rotation task, hand rotations tend to drop out. Presumably, physical rotation helps to internalize mental rotation. In other studies, participants rotated a wheel clockwise or counterclockwise while solving mental rotation problems. Mental rotation was faster and more accurate when the direction of manual rotation was the same as the optimal direction for mental rotation. But when the direction of manual rotation was in the opposite direction, mental rotation times and errors increased. More evidence that mental actions resemble physical ones comes from neuroimaging studies showing that mental rotation activates motor areas of the brain. Mental actions do not merely resemble physical ones; it turns out that making the parallel physical actions helps perform the mental ones.

  TWO PERSPECTIVES: OUTSIDER AND INSIDER

  When we solve mental rotation problems, we are outside looking at an object. We can mentally rotate all sorts of objects, familiar and unfamiliar, meaningful ones like letters and chairs, meaningless ones like shapes, 2D or 3D. They vary in difficulty and in patterns of reaction times. For example, deciding whether pairs of asymmetric letters like R and G at different orientations are the same or mirror images doesn’t give linear reaction times. Rotating a letter to 90 degrees from upright hardly increases reaction time, but turning it upside down slows time considerably. Apparently, we get pretty good at reading sideways.

  But when we imagine our own bodies in different orientations, we take an insider perspective. Look at the body in Figure 4.2 to decide which arm is extended, right or left.

  If you are like most people, you imagined your own body in that position and then figured out whether your left (correct) or right arm would be extended. Many everyday situations require that kind of thinking, such as when you tell someone how to get from your office to your home or figure out a route from a map. At each choice point, you have to decide whether to turn left or right. Doing this is more like spatial-motor imagination than visual-spatial imagination. You can feel it in your body as you think. If you’re like me, you might even turn your body a bit. Think back to being a child and figuring out how to put on a jacket or sweater. Or which way to turn a lid. Or which hand is your right one, still a problem for many grown-ups. Then there are soccer moves and tennis serves and dance and yoga and gymnastics. Playing the piano or violin, using your hands to raise clay on a potter’s wheel or do calligraphy. How to twist your body and your shoulder and your hand to reach an object that found its way under the bed. Or how to throw your opponent off in a combat sport.

  FIGURE 4.2. Which hand is outstretched?

  Thinking about how your body moves and turns in space can be brought into the laboratory as a visual-spatial task, no gymnastics required. Like the one you just tried. On each of many trials, people view bodies like the one above in various orientations with one arm outstretched; their job is to say whether the right or left arm is outstretched. Or they view hands in various orientations and say whether it’s the right or left hand. Even though the stimuli are visual, deciding whether an arm or a hand is right or left seems to rely primarily on spatial-motor imagination. That is, people imagine themselves or their hands in those orientations in order to make the right-left judgment. Just as for mental rotation of objects, the times to perform these tasks yield regular patterns of reaction times but quite different patterns from those of imagining rotation of objects. The patterns of response times reflect imagining motor actions rather than watching spatial transformation. For hands, it takes people longer to make left-right judgments when the hand positions are awkward than when they are comfortable. Both tasks, mental rotation of objects and mental rotation of bodies, have been brought into the scanner. The two tasks activate partly overlapping but partly different brain areas. Somewhat the same, somewhat different.

  Intriguingly, people who have lost an arm can perform these tasks, that is, they can decide which arm is outstretched and whether the depicted hand is right or left, but they are slower than people who have all their limbs. Presumably, the loss of physical motion diminishes imaginal motion, further support for the close relation between imagination and action. More to come.

  Just as actually rotating a hand can facilitate mental rotation, actually turning the body can facilitate imagined turns of the body. In one set of experiments, people made or imagined a short route with two turns. Then they were asked to point back to where they started. When they only imagined the turns, they made large errors, but if they actually turned, even blindfolded, they were far more accurate. In both cases of mental rotation, rotation of objects or rotation of bodies, actual actions facilitate imagined ones. The actual actions need not be identical to the imagined ones, but they are congruent with the imagined ones, rotating the hand for imagining rotating an object and rotating the body for imagining rotating the body.

  Although actual rotations help imagined ones, actually moving straight forward or backward doesn’t seem to help imagining moving forward or backward. Rotation in space leads to dramatic changes in the spatial relations of the things around us to us: what was in front might now be to the right, what was on the left might now be in back. Mimicking the motion apparently helps us update those spatial relations. Moving forward or backward, translation in space, can change what’s in front and back but doesn’t change what’s right and left. Updating spatial relations when imagining translation in space is apparently easy enough that it doesn’t benefit from the support of actual movement.

  All of this points to the difficulty of mental rotation, whether of our own bodies or of objects in front of our eyes, and to the roles congruent actions of the body can play in supporting that thinking.

  CREATING IMAGES: DRAWING IN THE MIND

  Mental rotation generated tremendous excitement and sparked more exciting research in spatial thinking. If the mind can imagine mental rotation, what other wonders can the mind perform? Perhaps we can imagine things changing in size, location, shape. Or adding parts, taking them away, reorganizing them. Perhaps we can scan them to make judgments, like size and distance. Yes, people can do those mental manipulations and more, with greater or lesser ease. Try this one. Imagine half a grapefruit, dome side up, flat side down. Now imagine hanging a capital J from the middle of the flat side. What do you have?

  You just built something in your head from a description in language, without any visual input. Mental construction, like physical construction, appears to be a step-by-step process. Consequently, the greater the number of parts, the longer it takes. For example, the figure in Figure 4.3 can be described as having two parts, the intersection of two rectangles, or as having five parts, five squares in a specific array. Same figure, but when described and conceived of as two parts, it takes less time for people to create an image in the mind than when described and conceived of as five parts.

  FIGURE 4.3. Image to be formed from a description with two or five parts.

  Mental construction mimics physical constructi
on in that it’s step-by-step from parts. But the analogy goes deeper. First, another task, one familiar from grade school, geometric analogies. Try the analogy in Figure 4.4.

  FIGURE 4.4. Geometric analogy requiring two spatial transformations for solution. From Novick and Tversky, 1987.

  The answer requires moving the small upper figure (the circle or triangle) inside the lower larger one (the rectangle or trapezoid) and enlarging the smaller one. Or changing the size of the smaller one and then moving it into the larger one. The order of moving or changing size doesn’t matter. Just like adding a set of numbers.

  We asked people to solve geometric analogies like this one. Each required two or three transformations drawn from a larger set of possible transformations. After solving each, people told us the order in which they did the transformations. Although they were free to choose any order, the order they chose agreed, the same order for nearly everyone. We then asked another group of people to do the transformations in that preferred order or in some other order. When the new group of students used the preferred order, they were faster and more accurate. Because there are no mathematical constraints on the order of applying transformations, the constraints must be cognitive, and we puzzled over what they could be. Maybe people did the harder ones first, while they had something to look at, and then did the easier ones that had to be done entirely in the mind. So, we asked people which transformations were harder. We also determined which ones took more time, another measure of difficulty. It was a good idea, but the data didn’t support it. People preferred first to move, then to rotate or reflect, then to remove a small part, then to add a half or change size, then to add shading, and finally to add a small part. The fastest and easiest transformation was the first, move, but the slowest and hardest was the second, rotate or reflect. So, neither time nor difficulty explained the order. We continued to puzzle.