Human Error

by James Reason

  Hannaman and his coworkers listed eight desirable features for HRA models:

  (a) They should be compatible with and complement current PRA techniques.

  (b) They should be scrutable, verifiable and repeatable.

  (c) Their application should result in quantification of crew success probability as a function of time.

  (d) They should take account of different kinds of cognitive processing (i.e., at the skill-based, rule-based and knowledge-based levels of performance).

  (e) They should identify the relationship to the model of various performance-shaping factors (e.g., design features affecting the human-machine interface, operator training and experience levels, stress factors, the time available for effective action, etc.).

  (f) They should be comparable to the highest degree possible with existing data from plant experience, simulator data or expert judgement.

  (g) They should be simple to implement and use.

  (h) They should help to generate insights and understanding about the potential for operators to cope with the situations identified in PRA studies.

  As we have seen, few techniques can satisfy the repeatability criterion; but many of the HRA models listed earlier have made effective progress towards meeting the more qualitative criteria.

  John Wreathall, one of the leading figures in the development of HRA techniques, posed the question: Is human reliability analysis a fact or is it fiction? His own attempt to answer this question represents as good a summary of the current state of the HRA art as any: “There is and always has been an aversion to modelling humans; reasons range from “it’s too difficult’ and ‘people are too varied’ to ‘it reduces people to mere cogs’. Each of these has an element of truth, but no more than an element. Human reliability models have been, and still are being developed—this is a fact. However, the existing methods are very simplistic—to claim that they represent reality is a fiction” (Wreathall, 1981).

  It is important to see these HRA models in the context of their development. They emerged to meet the demands upon PRA analysts to quantify the large and hitherto neglected human error contribution to systems accidents. PRAs, for their part, evolved to meet the demand for a priori assessments of the risks associated with potentially hazardous and often controversial technologies.

  Some HRA methods were created by engineers already professionally engaged in general reliability analyses, Bill Hannaman and John Wreathall are two notable instances of this breed. Others have been devised by human factors specialists, Alan Swain and David Embrey fall into this category. For both of these camps, there was, at the time of greatest industry demand, the mid-to late-1970s, little or nothing in the way of an agreed theoretical framework for human error mechanisms. Rasmussen’s skill-rule-knowledge classification has done an enormous service in filling this vacuum, both for the reliability engineers and for the psychologists. I hope that this book will carry this essential process some way further. Without an adequate and conceptually meaningful error classification and a workable theoretical infrastructure, there can be little or no principled basis to the business of human reliability quantification. Now that we have made some progress in that direction, we can reasonably hope for the emergence of more effective HRA techniques. Whether their predictions will ever do better than the proverbial orders of magnitude remains to be seen.

  One final point: It may seem on the face of it that these HRA methods have been more concerned with assessment than error reduction. However, it should be stressed that these two aspects of PRA/HRA are intimately related. Given the inevitability of human error, there are many who would argue that the best way of protecting hazardous systems against human failure is to make them more error tolerant (see Rouse & Morris, 1985, for a cogently argued expression of this view). One way of doing this is by identifying those human failures most likely to jeopardise the integrity of the plant and to defend against them by engineered safety devices or by procedures such as the ‘30-minute rule’ that buy operators thinking time in an emergency by demanding automatic systems capable of restoring the plant to a safe state without the need for human intervention for some fixed time after the initiating event And where these safety devices themselves might be under threat from possible human errors, it is necessary to build in independent back-up systems or ‘redundancies’. The need for such safety measures and guidance as to where they should be deployed are, in theory, the natural products of combined PRA/HRA studies. In an ideal world, good assessment should always drive effective error reduction.

  4. Risk management

  PRA does more than identify an acceptable level of risk. It also defines a set of conditions for the safe operation of the plant. Thus, probabilistic risk assessment constitutes a reference model to which risk management must aspire. Rasmussen and Pedersen (1984, p. 183) expressed this view as follows:

  The result of the PRA is a calculated risk figure which, if accepted, covers the ‘accepted risk’. If not accepted, the design has to be modified until acceptance has been achieved. Owing to incompleteness and errors during the PRA, an ‘additional risk’ may exist, which is not included in the accepted risk.

  Sources of this unaccounted risk include: (a) the use of components and materials which fall outside the populations providing the PRA failure data, or that are substandard, (b) the fact that the real plant does not conform to the model underlying the PRA, and (c) that the plant is not operated and maintained according to the assumptions made in the PRA. The function of risk management, therefore, is to limit these additional risks through such means as quality control, inspection and continual monitoring of failure data. The latter provide crucial feedback as to whether the design preconditions are being met by the operational reality.

  Rasmussen and Pedersen (1984, p. 183) also note that: “The major part of the human decision-making and administrative functions involved in operations management is not accessible to formal analyses with the present state of PRA. Errors of management may, however, be significant sources of common-mode errors and are therefore important candidates for risk management by feedback control.”

  5. Potential measures for error reduction

  So far, we have discussed relatively well-established assessment and reduction methods, or at least ones that have been or might soon be applied to high-risk technologies. In the remainder of this chapter, we will examine reduction possibilities that are still in the early stages of research and development, but that nevertheless offer some promise for improving the current state of affairs.

  Human error, as we have seen, appears in many guises and has a variety of causes. It is therefore not surprising that no single, universally applicable, error-reducing technique is either available or in prospect. Human reliability specialists will always need to rely upon a wide range of remedial tools in order to find methods best suited to their immediate needs. Considered below are some relatively new (or at least largely untried) approaches that may help to extend the utility of the future toolbag.

  5.1. Eliminating affordances for error

  Most human beings enjoy the experience of free will. They form intentions, make plans and carry out actions, guided by what appears to them to be internal processes. Yet these apparently volitional activities are subtly constrained by the character of the physical objects with which they interact. The term affordance refers to the basic properties of objects that shape the way in which people react to them.

  In his most recent book, The Psychology of Everyday Things (1988), Donald Norman has explored, among other things, how man-made objects and procedures offer affordances for error. Norman’s reasons for being drawn to this topic are worth quoting at some length (1988, pp 1-2):

  Over the years I have fumbled my way through life, walking into doors, failing to figure out water faucets, incompetent at working the simple things of everyday life. Just me, I mumbled, just my mechanical ineptitude. But as I studied psychology, as I watched the behavior of other people, I began to realize I
was not alone. My fumblings and difficulties were mirrored by the problems of others. And everyone seemed to blame themselves. Could it be that the whole world was mechanically incompetent?

  The truth emerged slowly. My research activities led me to the study of human error and industrial accidents. I began to realize that human error resulted from bad design. Humans did not always behave so clumsily. But they do so when the things they must do are badly conceived, badly designed. Does a commercial airliner crash? Pilot error, say the reports. Does a Soviet nuclear power plant have a serious problem? Human error, says the newspaper. Do two ships at sea collide? Human error, is the official cause. But careful analysis of the events that transpired during these kinds of incidents usually gives the lie to such a story. At the famous nuclear power plant disaster, Three-Mile Island, the blame was placed on the humans, on the plant operators who misdiagnosed the problems. But was it human error? Consider the phrase operators who misdiagnosed the problems. Aha, the phrase reveals that first there was a problem, in fact a series of mechanical failures. Then why wasn’t equipment failure the real cause? What about the misdiagnoses? Why didn’t the operators correctly determine the cause? Well, how about the fact that the proper instruments were not available, that the plant operators did the actions that had always been the reasonable and proper ones to do. How about the pressure relief valve that failed to close To me it sounds like equipment failure coupled with serious design error.

  In brief, people are not so much the possessors of ‘original fallibility’ as the victims of user-hostile objects. Norman’s analyses of error-affording situations are directed by the argument that people underestimate the extent to which the knowledge is located in the world, rather than in their heads. There is a trade-off between these two kinds of knowledge. Knowledge in the world (KIW) is accessible. It is always there and does not need to be prompted: but it is subject to the ‘out-of-sight-out-of-mind’ principle. While knowledge in the head (KIH) is efficient and independent of the immediate environment, it needs to be retrieved, and this often requires reminding. KIW does not have to be learned, merely interpreted. As such, it is easier to use. But such interpretations can lead to erroneous actions.

  These distinctions lead to a set of design principles for minimising error affordances:

  (a) Use both knowledge in the world and in the head in order to promote a good conceptual model of the system on the part of its users: this requires consistency of mapping between the designer’s model, the system model and the user’s model.

  (b) Simplify the structure of tasks so as to minimise the load upon vulnerable cognitive processes such as working memory, planning or problem solving.

  (c) Make both the execution and the evaluation sides of an action visible. Visibility in regard to the former allows users to know what is possible and how things should be done; visibility on the evaluation side enables people to gauge the effects of their actions.

  (d) Exploit natural mappings between intentions and possible actions, between actions and their effects on the system, between the actual system state and what is perceivable, between the system state and the needs, intentions and expectations of the user.

  (e) Exploit the power of constraints, both natural and artificial. Constraints guide the user to the next appropriate action or decision.

  (f) Design for errors. Assume their occurrence. Plan for error recovery. Make it easy to reverse operations and hard to carry out nonreversible ones. Exploit forcing functions (see Chapter 6).

  (g) When all else fails, standardise—actions, outcomes, layouts, displays, etc. The disadvantages of a less than perfect standardisation are often compensated for by the increased ease of use. But standardisation for its own sake is only a last resort. The earlier principles should always be applied first.

  This brief summary of Norman’s book does scant justice both to its richness and scope. Psychologists and practitioners are strongly urged to read it, the former because it shows where Norman’s 10-year error quest has currently brought him, and the latter because the book is highly readable and filled with practical illustrations of error-affording objects and situations.

  For those who might feel that the The Psychology of Everyday Things is strong on anecdotal evidence but weak on laboratory support, this section ends with a brief account of an empirical study that more than bears out Norman’s basic thesis about everyday objects. Hull, Wilkins and Baddeley (1988) asked 24 intelligent men and women to wire an electric plug. Only 5 succeeded in doing so safely, even though 23 of the subjects had wired at least one plug during the preceding 12 months. The errors were attributed to failure to read the instructions (subjects preferred “to act rather than reflect”); negative transfer —subjects treated new designs as though they were more familiar ones; the inability to formulate an adequate mental model of the task; and—most significantly for Norman’s thesis—the failure of plug designers to provide clear physical constraints on errant actions.

  5.2. Intelligent decision support systems

  The Three Mile Island accident in March 1979 was, as we have seen, a watershed in the history of nuclear power generation. Among other things, it brought home to the nuclear industry with painful clarity the marked error proneness of human decision making in emergency conditions. Similar conclusions have been drawn from retrospective analyses of the subsequent accidents at Prairie Island, North Anna, Oconee, Oyster Creek and Ginna (Pew, Miller & Feeher, 1981; Woods, 1982). Following TMI-2, the industry (specifically the Electric Power Research Institute in Palo Alto) explored various ways of aiding operator decision making in accident conditions. These included the following:

  (a) The addition of a shift technical adviser to the crew: A suitably qualified individual whose task was to monitor events and advise the shift supervisor regarding the interpretation of data and possible courses of action.

  (b) Training improvements: Specifically in the areas of skills, emergency procedures, specific plant characteristics, fundamental power plant knowledge and decision skills.

  (c) Computerised support systems: These ranged from safety parameter display systems (showing trends in important system state variables) to predictive on-line simulation capable of answering ‘What if?’ questions during the course of an emergency.

  Subsequent events have not inspired great confidence in the efficacy of some of these methods. In the Davis-Besse accident (see Chapter 7), for example, both independent safety parameter display systems were out of action before and during the event. According to a U.S. Nuclear Regulatory Commission report (NUREG, 1985), they were inoperable “due to separate but similar problems in the data transmission system between the control room terminals and their respective computer processors.” The shift technical adviser was on a 24-hour shift and was asleep in his apartment at the time of the reactor trip. He was summoned to the control room by the shift supervisor, and although he drove there immediately, he arrived some 15 minutes after the trip, by which time the event was essentially over. Thereafter he acted as an administrative assistant.

  In 1985 and 1986, two meetings were held in Italy under the auspices of NATO to discuss the issue of providing intelligent decision aids for process operators (Hollnagel, Mancini & Woods, 1986, 1988). A detailed description of the topics covered in these meetings can be found in the published sources. Of more general interest here was the emergence of a debate between two quite opposing schools of thought: those who regarded intelligent decision aids as tools and those who saw them as prostheses. These positions are outlined below.

  The optimists believe that the problem of operator error in high-risk installations will ultimately have a technical solution. The same exponential growth in computer technology that made centralised supervisory control possible in the first place can also provide the ‘cognitive tools’—felicitous extensions of normal brainpower—that will enable operators to deal successfully with its current problems of complexity and opacity. Such optimists, particularly if they are systems desi
gners, might also wish to claim that, in any case, human involvement in such systems (and hence the attendant risk of dangerous operator errors) will gradually decline as the possible scenarios of failure are better understood and further nonhuman means of coping with them are devised.

  A more pessimistic view, and the one espoused here, is that most operator errors arise from a mismatch between the properties of the system as a whole and the characteristics of human information processing. System designers have unwittingly created a work situation in which many of the normally adaptive characteristics of human cognition (its natural heuristics and biases) are transformed into dangerous liabilities. And, the argument continues, since the problem is fundamental to the design of such systems, and since (short of closure) these installations are likely to be with us for many years to come, the only immediate remedy is to provide the human participants with cognitive ‘prostheses’ (or, in plainer English, ‘mental crutches’) that will help compensate for some of these artificially-enhanced error tendencies and minimise the exaggerated danger of their consequences.

  Let us assume that progress in artificial intelligence will enable a new and powerful set of intelligent decision aids to be developed. For whom should these devices be tailored? In the post-TMI era the focus was upon control room operators; but the more recent developments discussed in Chapter 7 indicate that the operator’s mistakes can be traced back to latent decision errors made in the higher echelons of the system long before an accident sequence even began. How far back from the system ‘front line’ should these aids be deployed? By management? Among regulators? Designers? Governmental decision makers? Then there is the question of dependency. If the decision aid is worth its salt, its users will come to overrely upon it to the neglect of their own unaided diagnostic skills. The history of high-risk technology is littered with instances of accidents being caused by the failure of some safety-enhancing aid combined with operator overdependence, so that alternative sources of evidence are not consulted.