by Scott Young
The difficulty and usefulness of drills repeat a pattern that will recur throughout the ultralearning principles: that something mentally strenuous provides a greater benefit to learning than something easy. Nowhere is this pattern more clear than in the next principle, retrieval, where difficulty itself may be the key to more effective learning.
Chapter VIII
Principle 5
Retrieval
Test to Learn
It pays better to wait and recollect by an effort from within, than to look at the book again.
—William James, psychologist
In the spring of 1913, the mathematician G. H. Hardy received a letter that would forever define the course of his life. Sent by an accounting clerk working for the Port Trust Office of Madras in India, the letter contained a humble note of introduction along with some startling assertions. The author claimed that he had found theorems for problems that the best mathematical minds of the time had yet to solve. What’s more, he claimed that he had “no University education” and had derived these results from his own solitary investigations.1
Receiving letters from amateur crackpots who claimed to have solutions to famous problems was a common occurrence for someone of Hardy’s stature in mathematics, so at first he simply dismissed the letter as being more of the same. Still, flipping through the several pages of notes attached to the letter, the equations wouldn’t leave his mind. When he found himself thinking about them hours later, he brought the letter to the attention of his colleague John Littlewood. As the two of them toyed at trying to prove the strange assertions, they found that some of them they were able to prove with great effort, while others remained, in Hardy’s words, “scarcely possible to believe.” Maybe, Hardy thought, this wasn’t a letter from a crackpot but something rather different.
The formulas written were so bizarre and alien that Hardy remarked, “They must be true because, if they were not true, no one would have had the imagination to invent them.” What he only vaguely understood that day was that he had just had his first introduction to one of the most brilliant and bizarre mathematicians of all time, Srinivasa Ramanujan.
Ramanujan’s Genius
Before writing his letter to Hardy, which changed the course of mathematical history, Ramanujan was a poor, pudgy south Indian boy with a special love of equations. More than anything else, he loved math. In fact, his love of math often got him into difficulties. His unwillingness to study other subjects flunked him out of university. Equations were all he cared about. In his spare time and during stretches of unemployment, he would sit for hours on the bench in front of his family home, slate in hand, playing with formulas. Sometimes he would stay up so late that his mother would need to put food into his hand so he would eat.
As he was thousands of miles away from the center of mathematics of his day, access to high-quality textbooks was quite a challenge for Ramanujan. One resource he did encounter and mined extensively was a volume by George Shoobridge Carr called A Synopsis of Elementary Results in Pure and Applied Mathematics. Carr himself was hardly a towering figure of mathematical genius. The book, intended as a guide for students, included large lists of various theorems from different fields of mathematics, usually without explanation or proof. However, even without having proofs or explanations available, Carr’s book became a powerful resource in the hands of someone smart and obsessed like Ramanujan. For instead of simply copying and memorizing how certain theorems were derived, he had to figure them out for himself.
Though many commentators of the time, including Hardy, argued that Ramanujan’s impoverished upbringing and late access to the cutting edge of mathematics likely did irreparable harm to his genius, modern psychological experiments may offer an alternative perspective, for when Ramanujan dealt only with Carr’s extensive list of theorems using his own quirky obsession with mathematical formulas, he was unwittingly practicing one of the most powerful methods known to build a deep understanding.
The Testing Effect
Imagine you’re a student preparing for an exam. You have three choices about how you can allocate your limited studying time. First, you can review the material. You can look over your notes and book and study everything until you’re sure you’ll remember it. Second, you can test yourself. You can keep the book shut and try to remember what was in it. Finally, you can create a concept map. You can write out the main concepts in a diagram, showing how they’re organized and related to other items you need to study. If you can pick only one, which one should you choose to do best on the final exam?
This is essentially the question posed by the psychologists Jeffrey Karpicke and Janell Blunt in one study examining students’ choice of learning strategy.2 In the study, students were divided into four groups, each given the same amount of time but told to use different study strategies: reviewing the text a single time, reviewing it repeatedly, free recall, and concept mapping. In each group, students were asked to predict their score on the upcoming test. Those who did repeated reviewing predicted that they’d score the best, followed by the single-study and concept-mapping groups. Those who practiced free recall (trying to remember as much as they could without looking in the book) predicted the worst for their final performance.
The actual results, however, weren’t even close. Testing yourself—trying to retrieve information without looking at the text—clearly outperformed all other conditions. On questions based directly on the content of the text, those who practiced free recall remembered almost 50 percent more than the other groups. How could students, who have spent years getting firsthand experience about what matters to learning, be so misguided about what actually produces results?
One might be tempted to argue that this benefit of self-testing is an artifact of the way success is measured. The principle of directness asserts that transfer is difficult. Since self-testing and actual testing are most similar, perhaps it is this similarity that allows this method to work better. Had the method of evaluation differed, it might be reasonable to suspect that review or concept mapping might come out on top. Interestingly, in another experiment, Karpicke and Blunt showed that this wasn’t the explanation, either. In this experiment the final test was to produce a concept map. Despite the overwhelming similarity to the evaluation task, free recall still did better than using concept mapping to study.
Another possible explanation for why self-testing works is feedback. When you review something passively, you don’t get any feedback about what you know and don’t know. Since tests usually come with feedback, that might explain why students who practiced self-testing beat the concept mappers or passive reviewers. Though it is true that feedback is valuable, once again, retrieval doesn’t simply reduce down to getting more feedback. In the experiments mentioned, students were asked to do free recall but weren’t provided any feedback about items they missed or got wrong. The act of trying to summon up knowledge from memory is a powerful learning tool on its own, beyond its connection to direct practice or feedback.
This new perspective on learning shows how Carr’s book, with its lists of proofs without solutions, could have become, in the hands of someone sufficiently motivated to master them, an incredible tool for becoming brilliant at math. Without the answers at hand, Ramanujan was forced to invent his own solutions to the problems, retrieving information from his mind rather than reviewing it in a book.
The Paradox of Studying
If retrieval practice—trying to recall facts and concepts from memory—is so much better for learning, why don’t students realize it? Why do many prefer to stick to concept mapping or the even less effective passive review, when simply closing the book and trying to recall as much as possible would help them so much more?
Karpicke’s research points to a possible explanation: Human beings don’t have the ability to know with certainty how well they’ve learned something. Instead, we need to rely on clues from our experience of studying to give us a feeling about how well we’re doing. These so-c
alled judgments of learning (JOLs) are based, in part, on how fluently we can process something. If the learning task feels easy and smooth, we are more likely to believe we’ve learned it. If the task feels like a struggle, we’ll feel we haven’t learned it yet. Immediately after spending some time studying, these JOLs may even be accurate. Minutes after studying something using a strategy of passive review, students perform better than they would if they had practiced retrieval.3 The feeling that you’re learning more when you’re reading rather than trying to recall with a closed book isn’t inaccurate. The problem comes after. Test again days later, and retrieval practice beats passive review by a mile. What helped in the immediate time after studying turns out not to create the long-term memory needed for actual learning to take place.
Another explanation for why students opt for low-efficiency review instead of retrieval is that they don’t feel they know the material well enough to test themselves on it. In another experiment, Karpicke had students choose a strategy for learning. Inevitably, students who were performing more weakly elected to review the material first, waiting until they were “ready” to start practice testing.4 If through experimental intervention, however, they were forced to practice retrieval earlier, they learned more. Whether you are ready or not, retrieval practice works better. Especially if you combine retrieval with the ability to look up the answers, retrieval practice is a much better form of studying than the ones most students apply.
Is Difficulty Desirable?
What makes practicing retrieval so much better than review? One answer comes from the psychologist R. A. Bjork’s concept of desirable difficulty.5 More difficult retrieval leads to better learning, provided the act of retrieval is itself successful. Free recall tests, in which students need to recall as much as they can remember without prompting, tend to result in better retention than cued recall tests, in which students are given hints about what they need to remember. Cued recall tests, in turn, are better than recognition tests, such as multiple-choice answers, where the correct answer needs to be recognized but not generated. Giving someone a test immediately after they learn something improves retention less than giving them a slight delay, long enough so that answers aren’t in mind when they need them. Difficulty, far from being an obstacle to making retrieval work, may be part of the reason it does so.
The idea of desirable difficulties in retrieval makes a potent case for the ultralearning strategy. Low-intensity learning strategies typically involve either less or easier retrieval. Pushing difficulty higher and opting for testing oneself well before you are “ready” is more efficient. One can think back to Benny Lewis’s strategy of speaking a new language from the first day. Though this approach is high in difficulty, research suggests why it might be more useful than easier forms of classroom study. Placing himself in a more difficult context means that every time Lewis needs to recall a word or phrase, it will be remembered more strongly than when doing the same act of retrieval in a classroom setting and much better than when simply looking over a list of words and phrases.
Difficulty can become undesirable if it gets so hard that retrieval becomes impossible. Delaying the first test of a newly learned fact has some benefits over testing immediately.6 However, if you delay the test too long, the information may be forgotten entirely.7 The idea, therefore, is to find the right midpoint: far enough away to make whatever is retrieved remembered deeply, not so far away that you’ve forgotten everything. Although waiting too long before you test yourself may have disadvantages, increasing difficulty by giving yourself fewer clues and prompts are likely helpful, provided that you can get some feedback on them later.
Should You Take the Final Exam Before the Class Even Begins?
The standard way of viewing tests is that they work to evaluate the knowledge you have learned elsewhere—through reading or listening to lectures. The concept of retrieval flips this view on its head, suggesting that the act of taking a test not only is a source of learning but results in more learning than a similar amount of time spent in review. However, this still fits within the conventional idea of knowledge being first acquired, and then strengthened or tested later.
An interesting observation from retrieval research, known as the forward-testing effect, shows that retrieval not only helps enhance what you’ve learned previously but can even help prepare you to learn better.8 Regular testing of previously studied information can make it easier to learn new information. This means that retrieval works to enhance future learning, even when there is nothing to retrieve yet!
A variety of mechanisms has been proposed for explaining why this forward testing effect exists. Some researchers argue that it may be that trying to find knowledge that hasn’t been learned yet—say, by trying to solve a problem you haven’t learned the answer to yet—nonetheless helps reinforce search strategies that are put to use once the knowledge is encountered later. An analogy here is that trying to retrieve an answer that doesn’t yet exist in your mind is like laying down a road leading to a building that hasn’t been constructed yet. The destination doesn’t exist, but the path to get to where it will be, once constructed, is developed regardless. Other researchers argue that the mechanism might be one of attention. By confronting a problem you don’t yet know how to answer, your mind automatically adjusts its attentional resources to spot information that looks like a solution when you learn it later. Whatever the exact mechanism is, the reality of the forward-testing effect implies that practicing retrieval might not only benefit from starting earlier than one is “ready” but even before you have the possibility of answering correctly.
What Should Be Retrieved?
The research is clear: if you need to recall something later, you’re best off practicing retrieving it. However, this neglects an important question: What kinds of things should you invest the time in to remember in the first place? Retrieval may take less time than review to get the same learning impact, but not learning something at all is faster still. This is an important practical question. Nobody has time to master everything. During the MIT Challenge, I covered a lot of different ideas. Some were directly relevant to the kind of programming I wanted to do when I was done, so making sure I retained those ideas was a priority. Others were interesting, but since I had no plans to use them immediately, I put more effort into practicing retrieving the underlying concepts than doing technical calculations. One class I did, for instance, was Modal Logic. As I have no plans to be a logician, I can honestly say, eight years later, that I couldn’t prove theorems in modal logic today. However, I can tell you what modal logic is for and when it is used, so if a situation arises in which the techniques I learned in that class might be useful, I’d have a much better time spotting it.* There will always be some things you choose to master and others you satisfy yourself with knowing you can look up if you need to.
One way to answer this question is simply to do direct practice. Directness sidesteps this question by forcing you to retrieve the things that come up often in the course of using the skill. If you’re learning a language and need to recall a word, you’ll practice it. If you never need a word, you won’t memorize it. The advantage of this strategy is that it automatically leads you to learn the things with the highest frequency. Things that are rarely used or that are easier to look up than to memorize won’t be retrieved. These tend to be the things that don’t matter so much.
The problem with relying on direct practice exclusively is that knowledge that isn’t in your head can’t be used to help you solve problems. For instance, a programmer may realize a need to use a certain function to solve a problem but forgets how to write it out. Needing to look up the syntax might slow her down, but she will still be able to solve the problem. However, if you don’t have enough knowledge stored to recognize when you can use a function to solve your problem, no looking up can help you. Consider that over the last twenty years, the amount of knowledge easily accessible from a quick online search has exploded. Nearly any fact or co
ncept is now available on demand to anyone with a smartphone. Yet despite this incredible advance, it is not as if the average person is thousands as times as smart as people were was a generation ago. Being able to look things up is certainly an advantage, but without a certain amount of knowledge inside your head, it doesn’t help you solve hard problems.
Direct practice alone can fail to encourage enough retrieval by omitting knowledge that can help you solve a problem but isn’t strictly necessary to do so. Consider our programmer who has two different ways to solve her problem, A and B. Option A is much more effective, but B will also get the job done. Now suppose that she knows only about option B. She’ll continue to use the way she knows to solve the problem, even though it is less effective. Here, our fledgling programmer might read about option A on a blog somewhere. But since simply reading is much less effective than repeated retrieval practice, chances are that she’ll forget about it when it comes time to apply the technique. This may sound abstract, but I’d argue that this is quite common with programmers, and often the thing separating mediocre programmers from great ones isn’t the range of problems they can solve but that the latter often know dozens of ways to solve problems and can select the best one for each situation. This kind of breadth requires a certain amount of passive exposure, which in turn benefits from retrieval practice.
How to Practice Retrieval
Retrieval works, but it isn’t always easy. Not only is the effort itself an obstacle, but sometimes it’s not clear exactly how to do it. Passive review may not be very efficient, but at least it’s straightforward: you open your book and reread material until you retain it. Most books and resources don’t have a handy list of questions at the end to test you to see if you remember what they contain. To help with that, below are some useful methods that can be used to apply retrieval to almost any subject.
