Mind in Motion

by Barbara Tversky

  Animation

  You might be thinking, as many have, why bother with arrows, just use animation to convey processes that unfold over time. After all, using change in time to show change in time follows directly from the Principle of Correspondence. You wouldn’t be alone in that thought, and that thought led to the creation of educational animations for just about anything you can imagine, so many processes take place in time. The trouble was, when they were carefully and appropriately compared to static graphics, animations were no better for understanding and learning. This puzzled us, and we tried various simplified animations, but still found no advantages to animated graphics. We did find, as have many others, that good diagrams were better for learning than good descriptions. After looking at many animations, we realized that they violate the Principle of Use. Things happen too quickly to make sense of them. Sometimes we didn’t even know where to look. In addition, animations just show, they don’t explain. Good explanations segment processes by actions on objects, causes, and outcomes. Animations typically unfold in real time, but causes and outcomes don’t necessarily occur in equal time segments. This is not to say animations can’t be effective. They have proved themselves in helping people keep track of simple changes in time or space. However, creating effective explanatory animations takes thoughtful design—and checking to make sure they do what they are supposed to do.

  Interacting with interfaces: Gestures

  Gesture interfaces have caught on so quickly, undoubtedly because they take advantage of the natural urge of humans to use their hands to express meaning. And that’s the trick, to make the gestures correspond to the meaning. One example. We found that young kids performed discrete math problems, namely, addition, better with discrete than continuous gestures on an iPad. Conversely, they performed estimation tasks better with a continuous gesture.

  Memorability

  So many of the graphs and charts in textbooks and the media look alike. It’s no wonder: they all spew forth from the same graphics software. To make something memorable, make it distinctive. Unadorned points and lines and boxes are lifeless. One way to give them life is by turning them into pictograms or icons. Sometimes graphics of networks of politicians or actors use their faces as points. Hand-drawn graphs and charts can be distinctive, beautiful, and memorable. In the optimistic international spirit that tragically preceded World War II, philosophers and linguists of the Vienna Circle worked to create universal linguistic and depictive communications, Esperanto and Isotype (International System of Typographic Picture Education) among them. Otto Neurath, who later escaped to England, invented Isotypes, simple pictograms, for example, of tractors or factories. Neurath’s diagrams filled the bars of bar graphs with them, making it easy for viewers to see what variables were being compared in the bar graphs.

  Diagramming is blossoming everywhere. One interesting use is in conflict resolution or conflict prevention. A facilitator gathers different groups, listens to their issues, and sketches them on a huge white board, altering them as the discussion progresses. In articulating their issues to the facilitator, people work through them and clarify them for themselves and for the others. Seeing the points of convergence and divergence on the white board often helps the interested parties see solutions. It can be quite magical!

  FIGURE 8.18. Visual notes of a talk of the author’s by Yoon Bahk.

  Here’s another, a new job for artists and cartoonists: visual scribe. Figure 8.18 is a lovely example of visual notes sketched by Yoon Bahk from a talk I gave a few years back at an awesome drawing-art-cognition gathering at the Metropolitan Museum in New York City. Some of the content previews the next chapter.

  GOOD DIAGRAMS WORK

  An enormous amount of research over many years has showed that diagrams are effective for learning, teaching, memory, and even persuasion, typically far more effective than unadorned prose. This holds for teaching, learning, and explaining a wide range of topics, STEM importantly among them. The advantage of diagrams over text is easily explained: diagrams are a more direct mapping of meaning than words. That makes diagrams a boon in showing what something looks like, how to do something, or how anything works. As always, good design is important; we hope we are giving tips on doing just that. Same, of course, holds for prose; good design is crucial.

  If you prefer stories to data, I have a good one: a very simple diagram that has saved millions of lives. In 1997, the New York Times columnist Nicholas Kristof wrote that a column he had written changed the mission of the Bill and Melinda Gates Foundation from distributing computers to world health. Kristof later found out that it wasn’t his heartfelt prose but rather a simple graphic, designed by Jim Perry. We tracked it down. It was indeed a very simple graphic, mostly words, no bars or lines, just a table with the header Death by Water. A column below and to the left listed four kinds of waterborne diseases and the number of deaths per year each caused. To the right, another column lined up with each cause described the painful course of each disease. The total count of deaths for 1997 was 3,530,000. I doubt that there has been a more influential graphic (or paragraph of prose) than that one.

  CREATING EFFECTIVE DIAGRAMS, GRAPHS, CHARTS, TABLES, INFOGRAPHICS

  The fundamentals: the message and the audience. What do you want to say and to whom? We’ve revealed the tools, many design principles, guidelines, and rules of thumb. The forms: dots, lines, boxes, bars, networks, trees, tables (arrays of boxes) each of which has meaning, in context. A line on a map doesn’t have the same meaning as a line on a graph. Just like line in a clothesline, line of work, or a ticket line. Use the forms separately or combine them. Consider their meanings and the inferences they are likely to promote. Think about place in space. Add appropriate icons, words, sentences, and symbols. You can use other elements of meaning: color, texture, font style and size, and more. If you want your graphic to be remembered, make it distinctive. Whatever you do, make it beautiful—or try to. Check your design intuitions with real people, ideally using the Three Ps. Bear in mind that there is no single best way. Just as there are many ways to be beautiful, many ways to sing, many ways to be a good athlete or business person or actor, there are many designs that work. That’s why they keep coming! Now to the messages.

  FORMS OF DISCOURSE: DESCRIPTION, EXPLANATION, AND STORY

  Recall that large survey we did of diagrams in college textbooks. We found a small number of discourse forms, essentially descriptions, explanations, and stories. Descriptions included labeling the parts of a leaf or a cell and examples of different kinds of leaves or cells. Explanations included photosynthesis and cell division. Stories included Mendel’s discovery of genetics or Watson and Crick’s double helix of DNA. Yes, they can be diagrammed.

  The big three, descriptions, explanations, and stories, are also the big three that characterize discourse purely in language. Each of these builds on and expands the previous kind, and any particular piece of discourse, descriptive or depictive, may include a mix of the three kinds. Descriptions present a state of affairs in space or time. A map or a time line. Explanations add causality. How a pulley system works or how to register to vote. Stories have all that and then some: crucially, a narrative voice, but also suspense, drama, emotion, protagonists and antagonists, and more. In case you hadn’t noticed, story is a current buzzword; everyone’s looking for stories. Everyone’s writing about stories (those are usually descriptions and explanations—like this one). For good reason. Stories have enormous impact on us. Stories typically have characters, good ones we root for and villains we hate. Like us, the characters have desires and goals and emotions that sometimes conflict, they get in and out of trouble, they try and fail or try and succeed. Stories have suspense and emotion to engage us; they have vivid detail for memorability; they have morals or lessons or takeaway messages to keep. That’s a lot of bang for the buck. Remember that First Law of Cognition, benefits come with costs, advantages come with disadvantages.

  The big, big trouble with stories i
s that they override facts in people’s minds. Stories are colorful, engaging, and memorable. Stories stand in sharp contrast to facts, to data. Data reduce individuals to dots or numbers. Stories have life and are about life. We can learn life lessons from stories. Data are dry, numbers are easily confused. One story of one terrorist attack puts millions in fear. One story of one lucky lottery winner gets millions to buy tickets.

  There are two more forms of discourse I should mention, conversation and argument. Conversation is interactive; it entails alternating contributions from each party. It’s not appropriate for one person to dominate. At the same time, there’s less control over the content and the direction; conversations can and do meander. One hallmark of “modern” media is claims of interactivity for diagrams, infographics, animations, literature, music, theater, art. The reader-viewer-listener is expected to participate in meaning making. And does. But this kind of interaction is often only one-way, so calling media interactive can seem puzzling. The idea seems to be that you look (or listen), get a thought, and look again, but you look differently the second time because of that thought. Another spiral, and one that underlies many of the creative enterprises described in Chapter Nine.

  Next, argument, familiar from politics and courtroom dramas. Academia, too, is not without polemics, arguing for or against a theory or position or prediction. In making a case, building an argument, people bring to bear the evidence or analyses that support the position they are promoting. They may anticipate counterclaims, but usually only in the service of their position, to counter the counterclaims.

  STORIES: COMICS!

  For more on storytelling, we turn to comics, the most inventive form of storytelling around. Because comics use all sorts of depictions and all sorts of verbals, what we say applies far more broadly, to stories in prose, to visualizations. Comics typically show bodies acting in space, the foundational theme of this book. Comics are also diagrams: they use boxes to contain and separate, they line up the boxes in rows and columns and group them on pages. They use language and symbols in many ways.

  Graphic storytelling is everywhere. The superheroes are still around; they’ve become myths that transfix their fans with new episodes. Serious works of fiction in comic format are inspiring excellent authors and reaching mainstream audiences. There are superb cartoon guides to history, psychology, philosophy, physics, chemistry, statistics—just about any topic you can think of. Comic journalism is proliferating. Comics for kids, toddlers and up, are increasingly purchased by parents, teachers, and librarians, and loved by kids.

  Comics are good for us and good for our children

  Like all stories, they offer pleasure, they can be poignant, exciting, or funny. They can successfully educate and inform. They can match the media to the message, using all kinds of depictions when depictions are likely to be effective, all kinds of language when language is likely to be effective, and both for a double-whammy. Unlike stories in prose, they teach us how to look and what to look for, so important for a world that is increasingly using visual forms of communication. Comics can draw in readers, especially young reluctant ones. By now, dozens of studies have shown that comics are effective teachers. I have to go out of my way to say this because for years they had a bad rap; even the US Congress joined the fray. Even if comics didn’t turn youth to violence or communism, they were too easy, they weren’t really and truly reading. Low culture, to be disdained. No way! They’re an art form.

  Many have explained what comics do and how they work, legends like Eisner and Spiegelman. McCloud, another master of the medium, penned a comic on comics that has become a classic. What follows draws on their insights as well as research on cognitive science. I will highlight reasons for their success, some of the many astute devices and practices they use, but keep in mind that the meta-rule of comics is: break the rules!

  Pictures are remembered better than words

  Depictions are an important part of the advantage of comics. Pictures are not only remembered better than words, they are more distinctive than words, and they communicate faster and more directly—recall Chapter Two. They show nuances of action and emotion and setting that defy words. Witness the explosion of emojis, which have far overtaken internet slang like LOL and OMG; see the popularity of GIFs: one billion per day in 2016; watch the enthusiasm for Instagram, ninety-five million posts a day in 2018. Set images loose and there’s an explosion.

  Depictions can show and words can speak

  Spiegelman calls it Co-Mix, emphasizing the confluence of media, of depiction and description, allowing each to do what it does best, but even more, allowing them to work together, to interact, complement, supplement, contrast, contradict, meld, and blend. In that sense, comics bear resemblance to film or theater or computer games, but unleashed. Comics get away with more than the other multimedia. The medium allows, indeed, encourages and celebrates, unconventional, wildly creative ways of using each and both.

  Comics encourage and reward scrutiny

  The richness of the medium demands close study of each and both together. Those habits of looking and seeing can transfer to understanding real life—people, scenes, situations—and life on the page—maps, charts, visualizations, diagrams. Like comics, face-to-face communication is rich and multimodal, a mix of sound stuff—sighs, laughter, grunts, words, phrases, and sentences varying in intonation—and visual stuff—smiles, frowns, shrugs, nods, pointing of fingers, and enactments of hands and body.

  STORIES HAVE BEGINNINGS, MIDDLES, AND ENDS

  Stories, as far back as Aristotle’s Poetics, are said to have a narrative arc, visualized as a triangle by Freytag, action rising to a climax where tension is resolved, and then falling to a denouement where the strands are sewn together. The narrative arc gives stories distinct beginnings, middles, and ends. Yet, as Jean-Luc Godard famously said, “A story needs to have a beginning, middle, and end, but not necessarily in that order.”

  Beginning

  First, engage. Comics often begin with splash pages, fragments of the story splashed onto the page, usually a two-page spread. Splash pages are like an overture to an opera or ballet, hints of what’s to come artfully arranged. Like the cast of characters and settings for a play or the ingredients of a recipe. These overviews of the whole tell you what to look for, what to expect; they entice you, whet your appetite for what is to come.

  Middle: Segmenting

  The mind segments, connects, and sometimes reconnects everything it perceives in the world. Bodies segmented by their parts connected by their joints. Same for objects, connected by glue and nails. Events by goals and subgoals. Sentences into clauses and phrases, marked by pauses in speech and punctuation on the page. Often the segmentation is hierarchical, parts and parts of parts. Years by months, months by weeks, weeks by days, days by hours. Understanding just about anything begins with taking apart and putting together. The separate parts have separate meanings, as do the wholes.

  Comics segment stories explicitly and hierarchically, with boxes and pages and chapters. Comics put pieces of time into boxes that march in rows across the space of the page. Prose does it with paragraphs (note that indentation is a visual device) and chapters. Comic artists often think in two-page spreads of boxes; that’s what readers encounter when they turn a page. The contents have to work as two-page spreads and also box by box. Boxes are used systematically, forming a spatial structure for the story.

  Then there are those gaps between the boxes, colorfully referred to as the gutter. The boxes lack joints or glue; the gutters between them are empty, waiting to be filled in by the reader. Or hold the reader in suspense.

  It’s not just gutters in comics that need to be filled in. What we get from the world is always incomplete; we are constantly completing, jumping to conclusions from partial information. We see part of the face of a friend or only hear their approaching footfalls. A child hears “homework” and knows what to do. A siren from an unseen source; you know there’s an emergency. A s
hadow from behind. We don’t just fill in objects, we fill in context. If we see a close-up of an object like a garbage can, we remember it as if from farther away, we fill in the scene. Remember the Seventh Law of Cognition: The mind fills in missing information.

  Boxes in comics can do more than segment and contain. Their very forms can carry meaning. They can vary in size and shape; they can disappear entirely; they can tilt; they can be circular or elliptical, smooth or jagged, narrow or wide—any shape for that matter; they can overlap or go inside another box. They can turn into arrows and point, or circles and roll. These variants can be used to great effect in telling a story.

  McCloud classified box-to-box transitions in comics: moment-to-moment, action-to-action, subject-to-subject, scene-to-scene, aspect-to-aspect. And non sequitur, but one wonders if there is such a thing—people can find connections between anything, and odd combinations invite that. Someone could try that classification to see if it works on paragraphs of prose. His classification jibes with the work on event cognition: people segment live events when there’s a new action, a new object, a new person, a new scene. Moment-to-moment entails breaking up a larger action into small pieces, prolonging the moment and creating suspense. A spider crawling ominously up a body. Aspect-to-aspect has a similar effect, by pausing the action to scan the space from different perspectives, pondering the entire scene rather than the action, per se. Different objects, different characters, and what the characters are seeing. Authors can use these transitions to set the pace, from slow and contemplative and mysterious to fast and chaotic… and mysterious. Creating mystery is mysterious.