Ultralearning

by Scott Young

  Tactic 1: Flash Cards

  Flash cards are an amazingly simple, yet effective, way to learn paired associations between questions and answers. The old way of creating paper flash cards to drill yourself is powerful, but it has largely been superseded by spaced-repetition systems, as I’ll discuss in Principle 7. These software algorithms can handle tens of thousands of “cards” and also organize a review schedule so you can manage them.

  The major drawback of flash cards is that they work really well for a specific type of retrieval—when there’s a pairing between a specific cue and a particular response. For some forms of knowledge, for example memorizing foreign-language vocabulary, this works perfectly. Similarly, maps, anatomical diagrams, definitions, and equations can often be memorized via flash cards. However, when the situation in which you need to remember the information is highly variable, this kind of practice can have drawbacks. Programmers can memorize syntax via flash cards, but concepts that need to be applied in real programs often don’t fit the cue-response framework that flash cards demand.

  Tactic 2: Free Recall

  A simple tactic for applying retrieval is, after reading a section from a book or sitting through a lecture, to try to write down everything you can remember on a blank piece of paper. Free recall like this is often very difficult, and there will be many things missed, even if you just finished reading the text in question. However, this difficulty is also a good reason why this practice is helpful. By forcing yourself to recall the main points and arguments, you’ll be able to remember them better later. While doing research for this book, for instance, I would often print out journal articles and put them in a binder with a few blank sheets of paper after each of them. After I had finished reading, I’d do a quick free recall exercise to make sure I would retain the important details when it came time for writing.

  Tactic 3: The Question-Book Method

  Most students take notes by copying the main points as they encounter them. However, another strategy for taking notes is to rephrase what you’ve recorded as questions to be answered later. Instead of writing that the Magna Carta was signed in 1215, you could instead write the question “When was the Magna Carta signed?” with a reference to where to find the answer in case you forget. By taking notes as questions instead of answers, you generate the material to practice retrieval on later.

  One mistake I’ve made in applying this technique is to focus on the wrong kinds of things to ask questions about. I tried applying this method to a book on computational neuroscience, and I ended up asking myself all sorts of detailed questions such as what was the firing rate of certain neuronal circuits or who proposed a specific theory. That wasn’t intentional but rather a by-product of lazily restating the factual content in the book as questions. What’s harder and more useful is to restate the big idea of a chapter or section as a question. Since this is often implicit, it requires some deeper thinking and not just adding a question mark to some notes you copied verbatim. One rule I’ve found helpful for this is to restrict myself to one question per section of a text, thus forcing myself to acknowledge and rephrase the main point rather than zoom in on a detail that will be largely irrelevant later.

  Tactic 4: Self-Generated Challenges

  The above tactics work best with retrieval of simple information, such as facts or summaries of broad ideas you might encounter in a book or lecture. However, if you’re trying to practice a skill, not merely remember information, they might not be enough. For a programmer, it’s not enough to know what an algorithm means, but be able to write it in code. In this case, as you go through your passive material, you can create challenges for yourself to solve later. You may encounter a new technique and then write a note to demonstrate that technique in an actual example. Creating a list of such challenges can serve as a prompt for mastering that information later in practice and can expand your library of tools that you are able to actually apply.

  Tactic 5: Closed-Book Learning

  Nearly any learning activity can become an opportunity for retrieval if you cut off the ability to search for hints. Concept mapping, the strategy that didn’t work particularly well for students in Karpicke and Blunt’s experiments, could be beefed up considerably by preventing yourself from looking at the book when generating your concept map. I suspect that had this been done in the original experiment, students using this form of closed-book concept mapping would likely have done better on the eventual test that relied on creating a concept map. Any practice, whether direct or a drill, can be cut off from the ability to look things up. By preventing yourself from consulting the source, the information becomes knowledge stored inside your head instead of inside a reference manual.

  Revisiting Ramanujan

  Ramanujan was smart, there’s no denying it. However, his genius was aided immeasurably by two hallmarks of the ultralearner’s tool kit: obsessive intensity and retrieval practice. As he worked on his slate from morning to night, trying to figure out Carr’s sparsely written list of theorems was incredibly hard work. But it also created the desirable difficulties that allowed him to build a huge mental library of tools and tricks that would assist him in his later mathematical efforts.

  Retrieval played an important role in Ramanujan’s mathematical upbringing, but he is hardly the only one to take advantage of the tactic. In nearly every biography of great geniuses and contemporary ultralearners I have encountered, some form of retrieval practice is mentioned. Benjamin Franklin practiced his writing by reconstructing essays from memory. Mary Somerville worked through problems mentally when no candle was available for night reading. Roger Craig practiced trivia questions without looking at the answers. Retrieval is not a sufficient tool to create genius, but it may be a necessary one.

  Trying to produce the answer rather than merely reviewing it is only half of a bigger cycle, however. To make retrieval really effective, it helps to know whether the answer you dredged up from your mind was correct. Just as we often avoid testing ourselves until we’re ready because struggling with a test is uncomfortable, we often avoid seeking information about our skill level until we think it will be favorable. Being able to process that information effectively, hearing the message it contains loud and clear, isn’t always easy. Yet this is also why it is so important. This brings us to the next principle of ultralearning: feedback.

  Chapter IX

  Principle 6

  Feedback

  Don’t Dodge the Punches

  Everybody has a plan until they get punched in the mouth.

  —Mike Tyson

  From a narrow staircase in the back, Chris Rock enters the stage just as his name is being announced. With sold-out shows and HBO specials, Rock is no neophyte to stand-up comedy. His performances feel like a rock concert. With an energetic and punctuated delivery, he’s known for repeating the key phrase of a joke like the chorus of a song, the rhythm of it so precise that you get the feeling he would be able to make anything funny. And that’s exactly the problem. When everything you do is funny, how do you know what really makes a joke good?

  Far from the packed concert halls and jubilant crowds, Rock walks to the mic on the modest brick-backed stage at the Comedy Cellar in Greenwich Village, New York City. In his hand are scraps of cards on which he has scribbled bits of phrases, a trick for working out new material he learned from his grandfather, a cab driver who preached on weekends. Instead of his signature aggressive style, he slumps against the back wall. This is his laboratory, and he’s going to perform comedy with the precision of an experiment.

  “It’s not going to be that good,” Rock warns the crowd, who are stunned at his unannounced arrival on the small comedy stage. “Not at these prices,” he adds, joking “At these prices, I could leave right now!” He envisions the reviews: “Chris came out and he left. It was good! He didn’t tell any jokes—but it was good!” Notes in hand, Rock warns the audience playfully that this isn’t going to be a typical Chris Rock performance. Instead, he wants to
work out new material under controlled conditions. “They’ll give you about six minutes because you’re famous,” he explains. “. . . then you’re back to square one.” He wants to know what’s funny, when he’s not trying to be funny.1

  Rock’s method is not unique. The Comedy Cellar is famous for big-name drop-ins: Dave Chappelle, Jon Stewart, and Amy Schumer are just a few comedians who have tested out their material in front of small crowds here before performing it on prime-time specials and in concert-scale gigs. Why perform at a small club when you can easily draw large crowds and thousands of dollars from a huge performance? Why show up unannounced and then deliberately undersell your own comedic abilities? What Rock and these other famous comedians recognize is the importance of the sixth principle of ultralearning: feedback.

  The Power of Information

  Feedback is one of the most consistent aspects of the strategy ultralearners use. From the simple feedback of Roger Craig testing himself on Jeopardy! clues without knowing the answer to the uncomfortable feedback of Benny Lewis’s approach of walking up to strangers to speak a language he only started learning the day prior, getting feedback was one of the most common tactics of the ultralearners I encountered. What often separated the ultralearning strategy from more conventional approaches was the immediacy, accuracy, and intensity of the feedback being provided. Tristan de Montebello could have taken the normal route of carefully preparing his script and then delivering a speech once every month or two, as is the case for most Toastmasters. Instead he dove straight in, speaking several times per week, jumping among different clubs to gather different perspectives on his performance. This deep dive into feedback was uncomfortable, but the rapid immersion also desensitized him to a lot of the anxiety that being onstage can create.

  Feedback features prominently in the research on deliberate practice, a scientific theory of the acquisition of expertise initiated by K. Anders Ericsson and other psychologists. In his studies, Ericsson has found that the ability to gain immediate feedback on one’s performance is an essential ingredient in reaching expert levels of performance. No feedback, and the result is often stagnation—long periods of time when you continue to use a skill but don’t get any better at it. Sometimes the lack of feedback can even result in declining abilities. Many medical practitioners get worse with more experience as their accumulated knowledge from medical school begins to fade and the accuracy of their diagnoses is not given the rapid feedback that would normally promote further learning.2

  Can Feedback Backfire?

  The importance of feedback probably isn’t too surprising; we all intuitively sense how getting information about what we’re doing right and wrong can accelerate learning. More interestingly, the research on feedback shows that more isn’t always better. Crucially, what matters is the type of feedback being given.

  In a large meta-analysis, Avraham Kluger and Angelo DeNisi looked at hundreds of studies on the impact of providing feedback for learning.3 Though the overall effect of feedback was positive, it’s important to note that in over 38 percent of cases, feedback actually had a negative impact. This leads to a confusing situation. On the one hand, feedback is essential for expert attainment, as demonstrated by the scientific studies of deliberate practice. Feedback also figures prominently in ultralearning projects, and it’s difficult to imagine their being successful if their sources of feedback had been turned off. At the same time, a review of the evidence doesn’t paint the picture of feedback being universally positive. What’s the explanation?

  Kluger and DeNisi argue that the discrepancy is in the type of feedback that is given. Feedback works well when it provides useful information that can guide future learning. If feedback tells you what you’re doing wrong or how to fix it, it can be a potent tool. But feedback often backfires when it is aimed at a person’s ego. Praise, a common type of feedback that teachers often use (and students enjoy), is usually harmful to further learning. When feedback steers into evaluations of you as an individual (e.g., “You’re so smart!” or “You’re lazy”), it usually has a negative impact on learning. Further, even feedback that includes useful information needs to be correctly processed as a motivator and tool for learning. Kluger and DeNisi noted that some of the studies that showed a negative impact of feedback occurred because the subjects themselves chose not to use the feedback constructively. They may have rejected the feedback, lowered the standards they expect from themselves, or given up on the learning task altogether. The researchers note that who is giving the feedback can matter, as feedback coming from a peer or teacher has important social dynamics beyond mere information on how to improve one’s abilities.

  I find two things interesting about this research. First, it is clear that although informative feedback is beneficial, it can backfire if it is processed inappropriately or if it fails to provide useful information. This means that when seeking feedback, the ultralearner needs to be on guard for two possibilities. The first is overreacting to feedback (both positive and negative) that doesn’t offer specific information that leads to improvement. Ultralearners need to be sensitive to what feedback is actually useful and tune out the rest. This is why, although all the ultralearners I met employed feedback, they didn’t act on every piece of possible feedback. Eric Barone, for instance, did not attend to every comment and critique on early drafts of his game. In many cases he ignored them, when the feedback conflicted with his vision. Second, when it is incorrectly applied, feedback can have a negative impact on motivation. Not only can overly negative feedback lower your motivation, but so can overly positive feedback. Ultralearners must balance both concerns, pushing for the right level of feedback for their current stage of learning. Though we all know (and instinctively avoid) harsh and unhelpful criticism, the research also supports Rock’s strategy of disregarding the positive feedback that his celebrity automatically generates.

  The second interesting point about this research is that it explains why feedback-seeking efforts are often underused and thus remain a potent source of comparative advantage for ultralearners. Feedback is uncomfortable. It can be harsh and discouraging, and it doesn’t always feel nice. Standing up on a stage in a comedy club to deliver jokes is probably one of the best ways to get better at stand-up comedy. But the act itself can be terrifying, as an awkward silence cuts deep. Similarly, speaking immediately in a new language can be painful, as the sense of your ability to communicate goes down precipitously from when you use your native tongue.

  Fear of feedback often feels more uncomfortable than experiencing the feedback itself. As a result, it is not so much negative feedback on its own that can impede progress but the fear of hearing criticism that causes us to shut down. Sometimes the best action is just to dive straight into the hardest environment, since even if the feedback is very negative initially, it can reduce your fears of getting started on a project and allow you to adjust later if it proves too harsh to be helpful.

  All of these acts require self-confidence, resolve, and persistence, which is why many self-directed learning efforts ignore seeking the aggressive feedback that could generate faster results. Instead of going to the source, taking feedback directly, and using that information to learn quickly, people often choose to dodge the punches and avoid a potentially huge source of learning. Ultralearners acquire skills quickly because they seek aggressive feedback when others opt for practice that includes weaker forms of feedback or no feedback at all.

  What Kind of Feedback Do You Need?

  Feedback shows up in many different forms for different types of learning projects. Getting good at stand-up comedy and learning to write computer programs involve very different kinds of feedback. Learning higher math and learning languages are going to use feedback in different ways. The opportunities for seeking better feedback will vary depending on what you’re trying to learn. Rather than try to spell out exactly what feedback you need for your learning project, I think it’s important to consider different types of feedback, along w
ith how each one can be used and cultivated. By knowing what kind of feedback you’re getting, you can make sure to use it best, while also recognizing its limitations. In particular, I want to consider three types of feedback: outcome feedback, informational feedback, and corrective feedback. Outcome feedback is the most common and in many situations the only type of feedback available. Informational feedback is also fairly common, and it’s important to recognize when you can split apart outcomes to get feedback on parts of what you’re learning and when feedback only on holistic outcomes is possible. Corrective feedback is the toughest to find but when employed well can accelerate learning the most.

  Outcome Feedback: Are You Doing It Wrong?

  The first type of feedback, and the least granular, is outcome feedback. This tells you something about how well you’re doing overall but offers no ideas as to what you’re doing better or worse. This kind of feedback can come in the form of a grade—pass/fail, A, B, or C—or it can come in the form of an aggregate feedback to many decisions you’re making simultaneously. The applause Tristan de Montebello received (or the crickets he heard) after a speech is an example of outcome feedback. It could tell him if he was getting better or worse, but it couldn’t really say why or how to fix it. Every entrepreneur experiences this kind of feedback when a new product hits the market. It may sell wildly well or abysmally, but that feedback comes in bulk, not directly decomposable into the various aspects of the product. Did the product cost too much? Was the marketing message not clear enough? Was the packaging unappealing? Customer reviews and comments can provide clues, but ultimately the success or failure of any new product is a complex bundle of factors.