she was going to win Gerald Hanks, “Poker Math and Probability,” Pokerology, http://www.pokerology.com/lessons/math-and-probability/.
win a Nobel Prize Daniel Kahneman and Amos Tversky, “Prospect Theory: An Analysis of Decision Under Risk,” Econometrica: Journal of the Econometric Society 47, no. 2 (1979): 263–91.
a million television viewers The tournament drew an estimated 1.5 million viewers.
She’s not sure Annie, in a phone call to check facts in this chapter, expanded upon her thinking: “If Greg had jacks or better, I was in a bad situation. I was very undecided about the hand he could be holding, and I was in a situation where I really did have to create more certainty for myself. I really needed to decide if he had aces or kings, and then fold. Also, Greg Raymer, at that point, was an unknown quantity, but my brother and I had been watching videotapes of him play, and we had seen what we thought was a “tell,” something he did physically when he had a good hand, and I saw him do this particular thing that suggested to me that he had a strong hand. That’s not a certain thing, you don’t know if a tell is 100 percent, but it helped tip me into thinking he had a strong hand.”
“intelligence forecasts” “Aggregative Contingent Estimation,” Office of the Director of National Intelligence (IARPA), 2014, Web.
some fresh ideas For my understanding of the Good Judgment Project, I am indebted to Barbara Mellers et al., “Psychological Strategies for Winning a Geopolitical Forecasting Tournament,” Psychological Science 25, no. 5 (2014): 1106–15; Daniel Kahneman, “How to Win at Forecasting: A Conversation with Philip Tetlock,” Edge, December 6, 2012, https://edge.org/conversation/how-to-win-at-forecasting; Michael D. Lee, Mark Steyvers, and Brent Miller, “A Cognitive Model for Aggregating People’s Rankings,” PloS One 9, no. 5 (2014); Lyle Ungar et al., “The Good Judgment Project: A Large Scale Test” (2012); Philip Tetlock, Expert Political Judgment: How Good Is It? How Can We Know? (Princeton, N.J.: Princeton University Press, 2005); Jonathan Baron et al., “Two Reasons to Make Aggregated Probability Forecasts More Extreme,” Decision Analysis 11, no. 2 (2014): 133–45; Philip E. Tetlock et al., “Forecasting Tournaments Tools for Increasing Transparency and Improving the Quality of Debate,” Current Directions in Psychological Science 23, no. 4 (2014): 290–95; David Ignatius, “More Chatter than Needed,” The Washington Post, November 1, 2013; Alex Madrigal, “How to Get Better at Predicting the Future,” The Atlantic, December 11, 2012; Warnaar et al., “Aggregative Contingent Estimation System”; Uriel Haran, Ilana Ritov, and Barbara A. Mellers, “The Role of Actively Open-Minded Thinking in Information Acquisition, Accuracy, and Calibration,” Judgment and Decision Making 8, no. 3 (2013): 188–201; David Brooks, “Forecasting Fox,” The New York Times, March 21, 2013; Philip Tetlock and Dan Gardner, Seeing Further (New York: Random House, 2015).
A group of At various points during the GJP, the precise number of researchers involved fluctuated.
questions as the experts In response to a fact-checking email, Barbara Mellers and Philip Tetlock, another of the GJP leaders, wrote: “We had two different types of training in the first year of the tournament. One was probabilistic reasoning and the other was scenario training. Probabilistic reasoning worked somewhat better, so in subsequent years, we implemented only the probabilistic training. Training was revised each year. As it evolved, there was a section on geopolitical reasoning and another on probabilistic reasoning….Here is a section that describes the training: We constructed educational modules on probabilistic-reasoning training and scenario training that drew on state-of-the-art recommendations. Scenario training taught forecasters to generate new futures, actively entertain more possibilities, use decision trees, and avoid biases such as over-predicting change, creating incoherent scenarios, or assigning probabilities to mutually exclusive and exhaustive outcomes that exceed 1.0. Probability training guided forecasters to consider reference classes, average multiple estimates from existing models, polls, and expert panels, extrapolate over time when variables were continuous, and avoid judgmental traps such as overconfidence, the confirmation bias, and base-rate neglect. Each training module was interactive with questions and answers to check participant understanding.”
abilities to forecast the future In response to a fact-checking email, Don Moore wrote: “On average, those with training did better. But not everyone who got trained did better than all the people who did not get it.”
“tremendously useful” Brooks, “Forecasting Fox.”
“things you aren’t sure about” In response to a fact-checking email, Don Moore wrote: “What makes our forecasters good is not just their high level of accuracy, but their well-calibrated humility. They are no more confident than they deserve to be. It’s ideal to know when you have forecast the future with accuracy and when you haven’t.”
or roughly 20 percent In an email, Howard Lederer, a two-time World Series of Poker champion, explained the further nuances required in analyzing this hand: “The hand you use as an example is MUCH more complicated than it appears.” Given what’s known, Lederer said, there is actually a better than 20 percent chance of winning. “Here’s why. If you KNOW your opponent has an A or a K, then you know seven cards. Your two [cards], your opponent’s one card, and the four [communal cards] on the board. This means there are 45 unknown cards (you have no information on your opponent’s other card). This would mean you have nine hearts to win, and 36 non-hearts to lose. The odds would be 4 to 1, or 1 in 5. The percentages are 20%. As long as you are not putting more than 20% of the money into the pot, it’s a good call. Here’s where you might ask: if I am only 20% to win against an A or K, then how can I be better than [20%] to win? Your opponent might not have an A or K! He could have a spade flush draw without an A or K, he could have a straight draw with a 5–6. He could have a lower heart draw. That would be great for you! There’s also a chance he just has garbage and is trying to bluff you with nothing. In general, I’d calculate the chances that your opponent has one of these drawing or bluffing hands at about 30% (given how many of these possibilities there are). So let’s do some probabilistic math: 70% of the time he has an A or K, and you win 20% of those times. 25% of the time he has a draw and you win about 82% of those hands (I’m combining various possible odds given his range of holdings when he is drawing). And 5% of the time he has a total bluff and you win 89% of the time when he has garbage. Your total chances of winning are: (.7 × .2) + (.25 × .82) + (.05 × .89) = 39%! This is a simple ‘expected value’ calculation. You can see that the .7, .25 and .05 part of the calculation adds up to 1. Meaning we have covered all the possible holdings and assigned them probabilities. And we are making our best guess as to our chances against each holding. At the table, you don’t have time to do all the math, but ‘in your gut’ you can feel the odds and make the easy call. One other note, if you miss your flush and your opponent bets, you should seriously consider calling anyway. You will be getting well over 10–1, and the chances he is bluffing are probably higher than that. This is just a simple taste of the complexity of poker.”
they’ll quit For more on calculating odds in poker, please see Pat Dittmar, Practical Poker Math: Basic Odds and Probabilities for Hold’em and Omaha (Toronto: ECW Press, 2008); “Poker Odds for Dummies,” CardsChat, https://www.cardschat.com/odds-for-dummies.php; Kyle Siler, “Social and Psychological Challenges of Poker,” Journal of Gambling Studies 26, no. 3 (2010): 401–20.
“odds work for you” In response to a fact-checking email, Howard Lederer wrote: “It’s more complex than that. Amateurs players make many different kinds of errors. Some play too loose. They crave the uncertainty and favor action over prudence. Some players are too conservative, favoring a small loss in a hand over taking the chance to win, but also the chance to take a large loss. Your job as a poker pro is to simply play your best each hand. In the long run, your superior decisions will defeat your opponent’s poor decisions, whatever they may be. T
he societal value of poker is that it is a great training ground for learning sound decision-making under conditions of uncertainty. Once you get the hang of playing poker, you develop the skills necessary to make probabilistic decisions in life.”
Annie’s brother, Howard Though it does not bear on the events described in this chapter, disclosure compels mentioning that Lederer was a founder and board member of Tiltware, LLC, the company behind Full Tilt Poker, a popular website that was accused of bank fraud and illegal gambling by the U.S. Department of Justice. In 2012, Lederer settled a civil lawsuit with the Department of Justice related to Full Tilt Poker. He admitted no wrongdoing, but did agree to forfeit more than $2.5 million.
winning this hand Technically, Howard has an 81.5 percent chance of winning—however, because it is hard to win half a hand of poker, this has been rounded up to 82 percent.
remaining cards on the table In response to a fact-checking email, Howard Lederer wrote: “I would say that in a 3 handed situation, [a pair of sevens] is close to 90% to be best before the flop. This is the hand where I agree anyone would have played her hand and my hand the same way; all in before the flop. After we had all the money in, I am not a slight favorite, but instead a large favorite. This [is] a unique feature of hold’em. If you have a slightly better hand than your opponent, you are often a big favorite. 7–7 is about 81% to beat 6–6.”
“they tell you might occur” In response to a fact-checking email, Howard Lederer wrote: “It’s not an easy thing to choose a profession where you lose more often than you win. One has to focus on the long run, and realize that if you get offered 10–1, on enough 5–1 shots, you will come out ahead, while also realizing that you will lose 5 out of 6 times.”
humans process information Tenenbaum, in an email responding to fact-checking questions, described his research this way: “Often we start with what looks like a gap between humans and computers, where humans are outperforming standard computers with intuitions that may not look like computations….But then we try to close that gap, by understanding how human intuitions actually have a subtle computational basis, which then can be engineered in a machine, to make the machine smarter in more human-like ways.”
“seeing just a few examples” Joshua B. Tenenbaum et al., “How to Grow a Mind: Statistics, Structure, and Abstraction,” Science 331, no. 6022 (2011): 1279–85.
“examples of each?” Ibid.
(which has no strong pattern) In an email responding to fact-checking questions, Tenenbaum said that many of the examples they used were fairly complex, and “the reasons for the prediction functions having these shapes are the combination of (1) the priors, plus (2) a certain assumption about when an event is likely to be sampled (the ‘likelihood’), (3) Bayesian updating from priors to posteriors, and (4) using the 50th percentile of the posterior as the basis for prediction. What’s correct about what you have is that in our simple model, only (1) varies across domains—between movies, representatives, life spans, etc.—while (2–4) are the same for all the tasks. But [it’s] because of these causal processes (which vary across domains) together with the rest of the statistical computations (which are the same across domains) that the prediction functions have the shape they do.” It is important to note that the graphs in this text do not represent accurate empirical results, but rather patterns of predictions—the estimations that represent the 50th percentile of being right or wrong.
You read about a movie These are summaries of the questions asked. The direct wording of each question was: “Imagine you hear about a movie that has taken in 60 million dollars at the box office, but don’t know how long it has been running. What would you predict for the total amount of box office intake for that movie?” “Insurance agencies employ actuaries to make predictions about people’s life spans—the age at which they will die—based upon demographic information. If you were assessing an insurance case for a 39-year-old man, what would you predict for his life span?” “Imagine you are in somebody’s kitchen and notice that a cake is in the oven. The timer shows that it has been baking for 14 minutes. What would you predict for the total amount of time the cake needs to bake?” “If you heard a member of the House of Representatives had served for 11 years, what would you predict his total term in the House would be?”
variation of Bayes’ rule In an email responding to fact-checking questions, Tenenbaum wrote that “the most natural way to make these kinds of predictions in computers is to run algorithms which effectively implement the logic of Bayes’ rule. The computers typically don’t explicitly ‘use’ Bayes’ rule, because the direct computations of Bayes’ rule are typically intractable to carry out except in simple cases. Rather the programmers give the computers prediction algorithms whose predictions are made to be approximately consistent with Bayes’ rule in a wide range of cases, including these.”
data and your assumptions Sheldon M. Ross, Introduction to Probability and Statistics for Engineers and Scientists (San Diego: Academic Press, 2004).
skewed, as well “Base rate” typically refers to a yes-or-no question. In the Tenenbaum experiment, participants were asked to make numerical predictions, rather than answer a binary question, and so it’s most accurate to refer to this assumption as a “prior distribution.”
failures we’ve overlooked In an email responding to fact-checking questions, Tenenbaum wrote that “It’s not clear from our work that predictions for events in a certain class improve progressively with more experience with events of that type. Sometimes they might, sometimes they don’t. And this is not the only way to acquire a prior. As the pharaohs example shows, and other projects by us and other researchers, people can acquire a prior in various ways beyond direct experience with a class of events, including being told things, making analogies to other classes of events, forming analogies, and so on.”
“the Poker Brat” Eugene Kim, “Why Silicon Valley’s Elites Are Obsessed with Poker,” Business Insider, November 22, 2014, http://www.businessinsider.com/best-poker-players-in-silicon-valley-2014-11.
“bluff when it matters” In response to a fact-checking email, Hellmuth wrote: “Annie is a great poker player, and she has stood the test of time. I respect her, and I respect her Hold’em game.”
He folds In response to a fact-checking email, Hellmuth wrote: “I think she was trying to tilt me (get me emotional and upset) by showing a nine in that situation. A lot of players would have gone broke with my hand there (top pair) w[ith] a ‘Safe’ turn card, but I’ve made a living deviating from the norm and trusting my instincts (my white magic, my reading ability). I trusted it and folded.”
middle of the table In response to a fact-checking email, Hellmuth wrote: “With the chips I had at that time I had to go all in w[ith] 10–8 on that flop (I had top pair and there were flush draws, and straight draws possible). Completely standard. If you’re trying to imply that I put the money because I was emotionally tilted, you’re wrong. Nothing I could do there.”
Phil is out In response to a fact-checking email, Hellmuth contends that he and Annie had struck a deal when the tournament came down to the two of them in which they pledged to guarantee each other $750,000 regardless of the winner, and play for the last $500,000. Annie Duke confirmed this deal.
CHAPTER SEVEN: INNOVATION
movie everyone is talking about For my understanding of Frozen’s development, I am particularly indebted to Ed Catmull, Jennifer Lee, Andrew Millstein, Peter Del Vecho, Kristen Anderson-Lopez, Bobby Lopez, Amy Wallace, and Amy Astley, as well as other Disney employees, some of whom wished to remain anonymous, who were generous with their time. Additionally, I relied upon Charles Solomon, The Art of Frozen (San Francisco: Chronicle Books, 2015); John August, “Frozen with Jennifer Lee,” Scriptnotes, January 28, 2014, http://johnaugust.com/2014/frozen-with-jennifer-lee; Nicole Laporte, “How Frozen Director Jennifer Lee Reinvented the Story of the Snow Queen,” Fast Company, February 28, 2014; Lucinda Evere
tt, “Frozen: Inside Disney’s Billion-Dollar Social Media Hit,” The Telegraph, March 31, 2014; Jennifer Lee, “Frozen, Final Shooting Draft,” Walt Disney Animation Studios, September 23, 2013, http://gointothestory.blcklst.com/wp-content/uploads/2014/11/Frozen.pdf; “Frozen: Songwriters Kristen Anderson-Lopez and Robert Lopez Official Movie Interview,” YouTube, October 31, 2013, https://www.youtube.com/watch?v=mzZ77n4Ab5E; Susan Wloszczyna, “With Frozen, Director Jennifer Lee Breaks Ice for Women Directors,” Indiewire, November 26, 2013, http://blogs.indiewire.com/womenandhollywood/with-frozen-director-jennifer-lee-breaks-the-ice-for-women-directors; Jim Hill, “Countdown to Disney Frozen: How One Simple Suggestion Broke the Ice on the Snow Queen’s Decades-Long Story Problems,” Jim Hill Media, October 18, 2013, http://jimhillmedia.com/editor_in_chief1/b/jim_hill/archive/2013/10/18/countdown-to-disney-quot-frozen-quot-how-one-simple-suggestion-broke-the-ice-on-the-quot-snow-queen-quot-s-decades-long-story-problems.aspx; Brendon Connelly, “Inside the Research, Design, and Animation of Walt Disney’s Frozen with Producer Peter Del Vecho,” Bleeding Cool, September 25, 2013, http://www.bleedingcool.com/2013/09/25/inside-the-research-design-and-animation-of-walt-disneys-frozen-with-producer-peter-del-vecho/; Ed Catmull and Amy Wallace, Creativity, Inc.: Overcoming the Unseen Forces That Stand in the Way of True Inspiration (New York: Random House, 2014); Mike P. Williams, “Chris Buck Reveals True Inspiration Behind Disney’s Frozen (Exclusive),” Yahoo! Movies, April 8, 2014; Williams College, “Exploring the Songs of Frozen with Kristen Anderson-Lopez ’94,” YouTube, June 30, 2014, https://www.youtube.com/watch?v=ftddAzabQMM; Dan Sarto, “Directors Chris Buck and Jennifer Lee Talk Frozen,” Animation World Network, November 7, 2013; Jennifer Lee, “Oscars 2014: Frozen’s Jennifer Lee on Being a Female Director,” Los Angeles Times, March 1, 2014; Rob Lowman, “Unfreezing Frozen: The Making of the Newest Fairy Tale in 3D by Disney,” Los Angeles Daily News, November 19, 2013; Jill Stewart, “Jennifer Lee: Disney’s New Animation Queen,” LA Weekly, May 15, 2013; Simon Brew, “A Spoiler-Y, Slightly Nerdy Interview About Disney’s Frozen,” Den of Geek!, December 12, 2013, http://www.denofgeek.com/movies/frozen/28567/a-spoiler-y-nerdy-interview-about-disneys-frozen; Sean Flynn, “Is It Her Time to Shine?” The Newport Daily News, February 17, 2014; Mark Harrison, “Chris Buck and Jennifer Lee Interview: On Making Frozen,” Den of Geek! December 6, 2013, http://www.denofgeek.com/movies/frozen/28495/chris-buck-and-jennifer-lee-interview-on-making-frozen; Mike Fleming, “Jennifer Lee to Co-Direct Disney Animated Film Frozen,” Deadline Hollywood, November 29, 2012; Rebecca Keegan, “Disney Is Reanimated with Frozen, Big Hero 6,” Los Angeles Times, May 9, 2013; Lindsay Miller, “On the Job with Jennifer Lee, Director of Frozen,” Popsugar, February 28, 2014, http://www.popsugar.com/celebrity/Frozen-Director-Jennifer-Lee-Interview-Women-Film-33515997; Trevor Hogg, “Snowed Under: Chris Buck Talks About Frozen,” Flickering Myth, March 26, 2014, http://www.flickeringmyth.com/2014/03/snowed-under-chris-buck-talks-about.html; Jim Hill, “Countdown to Disney Frozen: The Flaky Design Idea Behind the Look of Elsa’s Ice Palace,” Jim Hill Media, October 9, 2013, http://jimhillmedia.com/editor_in_chief1/b/jim_hill/archive/2013/10/09/countdown-to-disney-quot-frozen-quot-the-flaky-design-idea-behind-the-look-of-elsa-s-ice-palace.aspx; Rebecca Keegan, “Husband-Wife Songwriting Team’s Emotions Flow in Frozen,” Los Angeles Times, November 1, 2013; Heather Wood Rudulph, “Get That Life: How I Co-Wrote the Music and Lyrics for Frozen,” Cosmopolitan, April 27, 2015; Simon Brew, “Jennifer Lee and Chris Buck Interview: Frozen, Statham, Frozen 2,” Den of Geek!, April 4, 2014, http://www.denofgeek.com/movies/frozen/29346/jennifer-lee-chris-buck-interview-frozen-statham-frozen-2; Carolyn Giardina, “Oscar: With Frozen, Disney Invents a New Princess,” The Hollywood Reporter, November 27, 2013; Steve Persall, “Review: Disney’s Frozen Has a Few Cracks in the Ice,” Tampa Bay Times, November 26, 2013; Kate Muir, “Jennifer Lee on Her Disney Hit Frozen: We Wanted the Princess to Kick Ass,” The Times, December 12, 2013; “Out of the Cold,” The Mail on Sunday, December 29, 2013; Kathryn Shattuck, “Frozen Directors Take Divide-and-Conquer Approach,” The New York Times, January 16, 2014; Ma’ayan Rosenzweig and Greg Atria, “The Story of Frozen: Making a Disney Animated Classic,” ABC News Special Report, September 2, 2014, http://abcnews.go.com/Entertainment/fullpage/story-frozen-making-disney-animated-classic-movie-25150046; Amy Edmondson et al., “Case Study: Teaming at Disney Animation,” Harvard Business Review, August 27, 2014. 207 surprised by all the criticisms In an email sent in response to fact-checking questions, Andrew Millstein, president of Disney Animation Studios, wrote: “These are the kind of notes that fuel our creative process and help propel the forward progress of all of our films in production. The creative leadership on any film often gets too close to their films and loses objectivity. Our Story Trust functions like a highly critical and skilled audience that can point to flaws in the story-telling and, more important, provide potential solutions….You’re describing a process of experimentation, exploration and discovery that are key components of all our films. It’s not a question of if this will happen, but to what degree. This is a constant part of our process and the expectation [of] every filmmaking team. It is what contributes to the high standards that our films set.”
Smarter Faster Better: The Secrets of Being Productive in Life and Business Page 36
