by Matthew Syed
*This narrowing of the definition of a mistake has an echo in the “wrong” approach to science. In chapter 3 we looked at the example of a hypothesis: namely, that water boils at 100˚C. We now know that this breaks down when water is boiled at altitude. But we could salvage the initial hypothesis by simply narrowing its content, as the philosopher Bryan Magee has pointed out. We could reformulate the hypothesis as: “Water boils at 100˚C at sea-level atmospheric pressure.” And when we discover that water does not boil at 100˚C in sealed containers, we could narrow the hypothesis still further: “Water boils at 100˚C at sea-level atmospheric pressure in open containers.” But to go down this route, placing ever more caveats upon the hypothesis, thereby progressively narrowing its empirical application, would be to destroy its usefulness. It would also obscure the most important feature of the situation, namely, that the failure of the initial hypothesis was an opportunity not to salvage it, but to reform it. It was a chance to come up with a theory that explains both why water boils at 100˚C at sea level and why it does not boil at altitude and in sealed containers. Science is not just about detecting errors but about responding in a progressive way.
*Was it right or wrong? With some decisions, it is very difficult to reach definitive answers. The situation is complex, and you can’t rewind the clock to see if an alternative approach would have worked better. This is sometimes called the “counterfactual problem.” In the next section, we will look at how to learn in situations such as these.
*Some refused the interview request, others did not respond. One of the signatories had died during the intervening period.
*Interviews by the author with twelve economists, three academic and nine working for financial institutions.
*An element of Lamarckism has resurfaced in recent years due to advances in epigenetics, which refers to changes in organisms caused by modification of gene expression rather than alteration of the genetic code itself. But this should not be held up as evidence that Lysenko was, in some curious way, right. After all, the phenomenon is being debated via testing and data rather than threats and intimidation. And it certainly doesn’t imply that it is legitimate to base science on ideology rather than evidence.
*Recent research suggests that this feature of memory may have benefits in terms of our imagination. For example, we can all imagine going to a café with David Beckham and drinking cappuccino. We simply retrieve a memory of the last time we went to a café and splice it together with an image of David Beckham, and a time when we drank coffee.
*There is a famous newspaper cartoon with a lineup consisting of a refrigerator, a hen, and a man with an Afro.
*Some people argue that juries are important independently of how well they reach accurate verdicts; that having a lay component in the justice system is an important aspect of democracy and has a legitimizing function. But, even so, this should not prevent us from trying to improve the way that juries operate. After all, this is what justice means.
*There is something of an analogy with sport. A top soccer player can take a free kick from thirty yards and bend it into the top corner of the goal. In order to do this he must solve differential equations and various problems of aerodynamics. But he does not solve these equations mathematically. His knowledge is practical: he solves these problems implicitly. Where does this practical understanding come from? Again it comes through trial and error (i.e., practice). Over thousands of hours he kicks balls at a target and gradually reduces the gap between where the ball lands and the target by varying and improving his technique.
*Francis Bacon, the philosopher, identified this dynamic interplay as early as the seventeenth century. In his book Novum Organum he writes: “Let no man look for much progress in the sciences—especially in the practical part of them—unless natural philosophy be carried on and applied to particular sciences, and particular sciences be carried back again to natural philosophy.”
*Random allocation (effectively flipping a coin) is important because it means that, providing the sample size is big enough, the two groups are likely to be similar. The only systematic difference between the groups is that one gets the treatment and the other does not.
*“Observational statistics” is a phrase that encompasses all the statistics drawn from looking at what happened. Randomized control trials are different because they encompass not merely what happened, but also construct a counterfactual for comparison.
*The process of conducting an RCT was much more difficult than Finckenauer thought possible. Advocates of Scared Straight didn’t cooperate. Judge Nicola, a high-profile supporter, tried to halt the trial before it had even started. “He saw no need for an evaluation since he had already collected hundreds of letters attesting to the success of the project,” Finckenauer says.
*Eventually federal funding was withdrawn from schemes that used the Scared Straight methodology. But they still keep popping up, not just in the United States, but elsewhere in the world. Until data are taken more seriously than narrative, they always will.
*Doctors from Great Ormond Street Hospital for children visited a Formula 1 team to witness how a pit stop happens. They were seeking to learn how to improve the handover from operating room to the intensive care unit. The number of errors dropped significantly in the aftermath. See http://asq.org/healthcare-use/why-quality/great-ormond-street-hospital.html.
*We saw in the early part of this book how aviation learns from mistakes by studying accidents and near-miss events. These adverse events are used to generate hypotheses about what went wrong, and possible ways of amending the system. But these are not the final word. After all, the proposed changes, however intuitive, might cause unforeseen dangers. Instead, proposed changes are always trialed in simulators, under different conditions and with different pilots, before being incorporated into the real-world system. In other words, aviation uses learning from error at multiple levels to drive progress.
*In order to conduct RCTs effectively, it is important to create the right methodology, including a large enough sample size. See http://www.evanmiller.org/how-not-to-run-an-ab-test.html.
*This should be taken as an estimate rather than a definitive amount, since many variables will have affected revenues following the implementation of the new color.
*This is sometimes called the problem of “external validity”: it is about the extent to which the results of one RCT can be applied to new contexts. Pharmacogenetics is a field based on this realization: the efficacy of many drugs depends on the genotype (and hence, often the ethnic origin) of patients. Consequently most drugs currently prescribed work well for Europeans and white Americans, because they formed the majority of test groups.
*Blockbuster turned down a chance to purchase the then fledgling Netflix for $50 million in 2000.
*In many cases genetic evolution is also a strategy for local optimization. Many optimization algorithms—computer programs that broadly mimic the evolutionary process—have steps where large changes are made at regular intervals to explore distant parts of the parameter space, and thus move away from the local optimum toward a higher peak.
*Think, for example, of economists who reframe their predictions so that they never actually fail. This systematically undermines the creative process. For without the failure, without the flaw, without the frustration, they are deprived not just of the motivation but also the conceptual fuel to reimagine their models. Their considerable intellectual brilliance is directed at defending their ideas rather than revolutionizing them.
*There is a fascinating, related literature on how innovators have fought over the credit for particular breakthroughs. Some of these battles have been fierce, as between Newton and Leibniz, who argued over who was the first to think of mathematical calculus. Less often, these disputes are resolved amicably, as between Wallace and Darwin. As one author put it: Wallace, “admirably free from envy or jeal
ousy,” was content to remain in Darwin’s shadow (Tori Reeve, Down House: The Home of Charles Darwin).
*Getting the manufacturing process running seamlessly is often about ironing out unwanted deviations. It is about using process controls and the like to reduce variation. Creative change is often about experimentation; in other words, increasing variation. For more on this distinction, and how to reconcile it, see: http://www.forbes.com/sites/ricksmith/2014/06/11/is-six-sigma-killing -your-companys-future/.
*One issue that was never fully resolved with Libyan Arab Airlines Flight 114 is why, according to the pilot of one of the Israeli Phantoms, all the window shades were down. It seems almost certain that, with pressure high and time limited, the pilot did not notice that some of the shades were, in fact, up.
*As estimated by how often the nursing units were intercepting errors before they became consequential, and other key variables governing self-correction and learning.
*“Hindsight bias,” another well-studied psychological tendency, also plays a role here. Once we know the outcome of an event—a patient has died, a plane has crashed, an IT system has malfunctioned—it is notoriously difficult to free one’s mind from that concrete eventuality. It is tough to put oneself in the shoes of the operator, who is often acting in high-pressure circumstances, trying to reconcile different demands, and unaware of how a particular decision might pan out.
As Anthony Hidden QC, the man who investigated the Clapham Junction Rail Disaster, which killed thirty-five people in 1988, put it: “There is almost no human action or decision that cannot be made to look flawed and less sensible in the misleading light of hindsight.”
*This has a rather obvious analog with what is sometimes called “defensive medicine,” in which clinicians use a host of unnecessary tests that protect their backs, but massively increase health-care costs.
*Science is not without flaws, and an eye should always be kept on social and institutional obstacles to progress. Current concerns include publication bias (whereonly successful experiments are published in journals), the weakness of the peer review system, and the fact that many experiments do not appear to be replicable. For a good review of the issues, see: www.economist.com/news/briefing/21588057-scientists-think-self-correcting-alarming-degree-if-not-trouble.
*As the creativity researcher Charlan Nemeth has put it: “The presence of dissenting minority views appears to stimulate more originality.”
*For a look at how the method of learning from failure has altered in aviation over the years, and with interesting thoughts on how it will continue to evolve, see Sidney Dekker’s lecture: https://vimeo.com/102167635.
Looking for more?
Visit Penguin.com for more about this author and a complete list of their books.
Discover your next great read!
