The Crash Detectives

by Christine Negroni

  It had become clear to Hersman that the lithium-ion cells in the Dreamliner battery box—the cells Boeing was supposed to coddle like a temperamental child, the cells the FAA warned had characteristics that “could affect their safety and reliability”—had just done the very thing they were not supposed to do under any circumstances: heat up uncontrollably. In Boston, which was the incident the NTSB would investigate, this had caused a very-high-temperature chain reaction that destroyed the entire eight-cell assembly.

  “These events should not happen as far as design of the aircraft is concerned,” Hersman told reporters. “There are multiple systems to protect against a battery event like this. Those systems did not work as intended. We need to understand why.”

  This was not as easy as it sounded. Every air accident may be different, but there are common investigative themes: mechanical, operational, organizational. The mystery on the Boeing 787 wasn’t about the physics of flight or the human operation of the plane. This was a puzzle about complex electrochemistry. When Hersman’s investigators in America and their counterparts in Japan set about to educate themselves on the topic, they quickly found that many of the experts had already been hired by Boeing. Though it had eschewed outside advice during the development phase, the company was now assembling advisory committees and review boards and scooping up spare batteries for its own testing.

  Dana Schulze, deputy director of aviation safety at the NTSB, said getting up to speed for her investigation was frustrating. Still, she showed remarkable understanding of Boeing’s point of view.

  “Our job is safety, but it’s hard to argue that the priority was getting this plane back in the air,” she told me. “At the same time, we had an investigation to conduct and wanted to be sure that the safety issues were addressed.”

  Judy Jeevarajan was one of those experts who found herself on the receiving end of a call from Boeing. From 2011 to 2015, she was battery group lead for safety and advanced technology at NASA. Part of her job was to study and publish articles on how to use lithium-ion batteries safely in a manned space environment. Because the batteries are so much larger and more powerful than those used on personal devices, when they go bad, they have the potential to trigger catastrophe. Jeevarajan’s job was to manage those risks on the International Space Station by critically assessing the battery at three levels: the individual cell, the cells as a unit (the battery), and the electrical system into which they were integrated.

  When the Dreamliner was grounded, Jeevarajan was recruited by Boeing to give an honest opinion about the battery design and function. On the day the experts got together in Seattle, the damaged battery from the JAL plane in Boston was wheeled into the room on a cart. Everyone crowded around for a look. Jeevarajan was surprised at what she saw. The cells were missing any bracketing substantial enough to keep them firmly in place, protected from vibration, and separated from one another. This observation was shared by Kazunori Ozawa, a battery engineer who was involved with the development of lithium-ion batteries many years before at Sony, but who was not part of the review. He had seen photos of the damaged battery.

  “By looking at the inside of the pack after the accident, it can be easily understood that those cells were not clamped well,” Ozawa told me. He was concerned that the cells would vibrate in an aviation environment, causing the jelly roll–like windings to move around inside the case, perhaps even triggering short circuits if the terminals touched.

  Also, Jeevarajan noted that hot spots could develop inside each cell and within the foil/chemical roll if “used outside of manufacturer’s specifications.” Substantial heat could melt the film separating the cathode from the anode, a big no-no because it can trigger a short circuit and immediate thermal runaway.

  Professor Dahn also thought that was a problem. Referring to tests by the NTSB during the investigation, he noted that “when the cells were discharged at their max-rated power, the terminals were hot,” he said. “They became one hundred eighty Celsius [three hundred sixty degrees Fahrenheit] above the melting point of the separator. That’s bad.” As far as he was concerned, part of the problem was the size of the cells. At two inches each, the individual cell pack was just too thick. One failure might propagate and “set off the neighbors,” as he put it.

  GS Yuasa designed the Dreamliner battery based on one it produced for all-terrain vehicles. When I met with Dana Schulze at the NTSB’s offices in Washington, I asked her if it had been a leap to suggest an ATV battery and a battery on an airplane were the same. She said GS Yuasa’s experience could be applicable as long as the differences between the two were considered, something Schulze, a mechanical engineer, called a similarity analysis. In the NTSB’s review of the battery design, “We couldn’t find evidence that they had done a lot of that.”

  Locked in competition with Boeing with its same-size and same-range Airbus A350, it might sound like bragging when the French airplane maker Airbus points out the various differences between the 787 battery and the lithium-ion battery now powering the A350. But, actually, I called Airbus, not the other way around.

  Airbus accounted for the possibility of an internal short circuit in one cell, according to Marc Cambet, a systems component architect for the A350. The design included precautions to prevent one failing cell from spreading to another. “On our side, the cells are insulated one from each other and insulated in terms of thermal insulation and in terms of vibration and in terms of separation between the cells,” Cambet said.

  Jeevarajan’s review of the Boeing battery covered many issues, including propagation, insulation, and vibration. She had one overriding problem, though, and that was Boeing’s safety philosophy when it came to the batteries. The company worried too much about overcharging them and ignored the potential for and response to internal and external short circuits. These could result from something such as the melting of the separator; the build-up of dendrites; or field failures, internal pressure, and deformations in the cell case.

  It was not that Boeing failed to consider them, Jeevarajan said. The company had, but it had also dismissed the hazards. When it tested for external shorts, “every single cell” vented and emitted smoke. When she asked the company, “Why didn’t you take that into consideration?” the answer was, “There was no fire. We didn’t think it was a big deal.”

  “It blew my mind that they had every single cell vent with smoke,” which was expressly prohibited under the FAA’s conditions, she reminded me, “and they just ignored it.”

  “They had good protection for overcharge, but none for short circuits,” Jeevarajan concluded, and overcharging wasn’t a factor on either of the Japanese airliners. In the pages of notes she gave to Boeing, Jeevarajan spelled out what she saw as the design’s many shortcomings, including the one that the Japanese investigators would seize on much later as the likely culprit for what happened on the ANA Dreamliner.

  “If you want to charge at cold temperatures, you’ve got to reduce the charge current,” she explained, or risk generating too much heat. It’s harder for the ions to move when it’s cold, so charging generates more heat in the battery.

  On the A350, the battery temperature is taken into account, Cambet explained. “The charge is slowed when the temperature is too low,” he said, because “if you try to charge full power at low temperatures you can build some risk of internal short circuit.”

  Moderating the rate was not possible on the Dreamliner battery system, and the fact that the ANA and JAL events happened in January was an interesting piece of the puzzle. Around this time, one of the government investigators (who asked not to be identified by name) summed it up, saying, “The cause was not found, but so many potential causes were found that [it] was pretty surprising.”

  In Tokyo, as in Washington, DC, the air safety folks were trying to solve a mystery while taking a crash course in electrochemistry. The Japan Aerospace eXploration Agency (JAXA) was advising, as were NASA and the U.S. Naval Sea Systems Command, which, like the s
pace programs, is intrigued by the potential of these batteries, yet cautious about the hazards.

  “It was quite difficult to take enough time to learn the new technology,” said Masaki Kametani, fifty-three, one of the seven investigators assigned to the case in Japan. Their little department was the subject of international attention. While the Boston battery fire had taken place on the ground in a nearly empty plane, the ANA flight had been in the air with passengers. “What if” was on everybody’s mind.

  The investigations’ official reports ran to hundreds of pages. Japan’s took twenty months to complete, and the American version nearly two years. While the flaws in the design of the battery were many, neither agency could say specifically what initiated the incidents they’d investigated. The JTSB concentrated on the cold weather charging phenomenon. The NTSB thought a manufacturing defect in the cell case might have created a hot spot leading to a short circuit. In both cases the entire battery fell to what the industry and physicist Lewis Larsen graphically call “thermal fratricide.”

  In its report, the NTSB went beyond what had happened on the plane in Boston to review how Boeing had convinced the FAA that the battery would meet the special condition. There were dozens of tests with a mind-numbing collection of titles and findings. Yet later, when I interviewed the NTSB’s Schulze, she explained it to me quite simply. It wasn’t the number of tests that was relevant; it was the assumptions made by Boeing as to what actually needed to be tested and how closely those checks reflected how the battery would operate in an actual airplane. “The test in and of itself really didn’t provide enough information to assume that an internal short circuit wasn’t going to result in propagation to the other cells,” Schulze said.

  The parallel to the Comet was unmistakable. The de Havilland engineers sought reassurance by referring to demonstrations of the cabin withstanding twice the amount of pressure the plane would experience. Yet during testing, the structure had been given supplemental support. It was therefore not representative of the plane in flight.

  By the time the investigators published their findings, the Dreamliners had been back in the air for nearly two years, released to fly again after a number of changes that recognized the battery’s newly exposed hazards, if not its suitability. Boeing also acted on the suggestions of experts such as Jeevarajan and put more insulation between the cells, placing them in a stronger frame and isolating them electrically with Kapton tape. These were precautionary measures. They didn’t change the battery’s volatile nature, so Boeing opted to cage the beast.

  Each battery went into a stainless steel housing with a titanium vent tube. The box would contain and smother a fire and protect against heat. The vent would release outside the airplane smoke or fumes generated by a failure. The system was an insurance policy against more unknowns, known or otherwise.

  Boeing opted not to talk to me for this book, rejecting repeated requests made in person and via e-mail. I didn’t like it, but I certainly understood. The company might be concerned that the plane it spent a decade developing will be permanently associated with one particular design failure. Had someone at Boeing given an interview, I’m sure that person would have reminded me that the 787 is carrying out the mission for which it was created. Airline customers love it, and passengers do, too. One thousand had been sold by 2015.

  What Boeing probably believes in its corporate heart, is akin to Wilbur Wright’s thoughts about “mounting a machine and becoming acquainted with its tricks.” Progress involves risk. Innovation will always have unanticipated consequences. The challenge for plane makers past and present has not been finding the guts to gamble, but balancing audacity with prudence before the plane moves from design to product.

  The price of getting it wrong can be too high, as Rolls-Royce and Lockheed learned in the early 1970s during the development of the L-1011 jumbo jet.

  Ask any airline pilot who has ever flown it to tell you about the Lockheed L-1011 and be prepared for a long soliloquy. “Best seat in the sky,” one captain told me. Passengers enjoyed its unprecedented space and the quiet that had L-1011 customer Eastern Airlines calling the plane the “Whisperliner.” Yes, everybody loved the three-engine Tristar, but unlike its competitors, the Boeing 747 and the McDonnell Douglas DC-10, the L-1011 is not seen flying anymore. Only two hundred fifty were made, and aviation analyst Richard Aboulafia, who writes the widely read industry newsletter Teal Monthly, cites an engine design gone terribly wrong with contributing to the ultimate failure of the L-1011.

  We hear about the use of composite material in aviation all the time these days, as it replaces heavier metal in airplane structures and components in planes such as the Dreamliner, the Airbus A350, and the A380. Back in the sixties, however, when Rolls-Royce was designing a more powerful, lighter-weight engine to power wide-body aircraft, its plan to use woven layers of compressed and hardened glass fibers to replace the metal on engine fan blades was new. Using a composite called Hyfil for the blades would save three hundred pounds and make the engine 2 percent more fuel efficient, but it didn’t work as planned.

  In a 2012 Royal Aeronautical Society magazine story called “Blades of Glory,” writer Tim Robinson called it “the blade that almost broke the company.” While the engine was still in development, Rolls-Royce and Lockheed discovered that the Hyfil composite was prone to delaminate. The layers would separate, just like what happens when a laminate board is left out in the rain. The fan blades would also not withstand the assault of frozen poultry. That’s not as bizarre as it sounds; to ensure that an engine will survive ingesting birds without coming apart, manufacturers shoot hard-as-ice frozen chickens into the core of a spinning jet engine. In this case, it was a battle the roasters won.

  While Boeing and McDonnell Douglas churned out their wide-bodies, sending them to airlines so they could start flying to far-flung places, Lockheed’s L-1011 was stuck on the ground waiting for Rolls-Royce to refit the engine with titanium blades. Unable to pay its bills, Rolls went into government receivership. Lockheed fared only slightly better; it required government loans to stay in business. It took years for Rolls-Royce to finally deliver the engine Lockheed had ordered for the L-1011.

  Aboulafia says the bad bet on Hyfil was orders of magnitude worse for the companies involved than what Boeing faced with its lithium-ion batteries, even though the engines should have been an easier fix. “You have to distinguish between two types of design. On the Dreamliner the issue is fundamental, and on the L-1011 the problem was an accessory; an expensive accessory, yes, but it was a discrete system,” he said. “Replace the engines, problem solved; it was not a fundamental design flaw.”

  The problem for Lockheed was that it was locked into the Rolls-Royce engines; it could not swap them out for the product of another engine maker. Lockheed had no choice but to wait out the delay. Once the composite blade was replaced with titanium, the RB221 became one of Rolls-Royce’s best-selling engines. Updated versions still power the Boeing 747 and Boeing 767.

  The Dreamliner battery saga does not have such an unequivocal finish or even a dignified one. Aboulafia calls the stainless steel containment box “inelegant,” the opposite of a Rube Goldberg invention, where a comically over-engineered response addresses a simple quandary. “It’s a simple solution to a complex problem,” he said. And it is a problem that just won’t go away.

  On January 14, 2014, nearly a year to the day after the ANA Dreamliner made its emergency landing at Takamatsu Airport, maintenance workers in the cockpit of a Japan Airlines Dreamliner preparing to depart Narita for Bangkok saw white smoke coming off the plane. Checking the battery, they discovered that one of the eight cells had vented, leaking fluid into the box. The problem did not spread to other cells.

  Ten months later, a Qatar Airways 787 had to make an emergency landing when one cell in the airplane’s battery vented. Boeing notified the NTSB and the FAA, but neither agency conducted an investigation.

  “The airplane performed as certified for
this failure,” FAA spokeswoman Laura Brown told me. When I asked by what means the air safety authority would know how the airplane performed since it had not investigated, Brown said that Boeing had taken a look and shared its findings.

  Not only is the FAA not investigating when cells vent, but it is not even keeping count of how often it happens. Since the box contains the smoke, fumes, and presumably fire, the American aviation regulator’s position is that a battery malfunction is no longer their concern. “We don’t consider something an ‘event’ if the containment box performs as designed,” she said.

  Several battery scientists, including some who do not want their names to be associated with their opinions because they work with Boeing, say it is lunacy to dismiss the seriousness of continuing cell failures.

  John Goodenough, a physicist and professor at the University of Texas who is considered the inventor of lithium-ion batteries, points out that by the time electrolyte vents, there is fire within the cell. “If you are hot enough to start boiling and needing to vent, the electrolyte will have caught fire by then.”

  Since it first approved Boeing’s use of lithium-ion energy storage on the 787, the FAA went from saying that fire in any situation was unacceptable to calling containing a fire no big deal.

  “Why play with fire when you don’t have to play with fire?” asked Dalhousie University’s Jeff Dahn. That cells vented on four or maybe five batteries in the plane’s first three years of service means the GS Yuasa–produced cells were failing at an “astronomically high rate,” Dahn said.

  Here’s how he figures this. Each plane has sixteen cells, and by the end of 2014 there were fewer than three hundred Dreamliners in service. If the cell events noted here are the only ones, and we can’t know, because no one aside from Boeing seems to be counting, and Boeing’s not saying, then the failure rate in the cells is one in every few hundred. By contrast, Dahn said, “for the cells used in laptops and phones, the failure rate is one in more than twenty million. It’s irresponsible to continue with such a product.”