Code Warriors
Page 1
ALSO BY STEPHEN BUDIANSKY
Biography
Mad Music: Charles Ives, the Nostalgic Rebel
Her Majesty’s Spymaster
History
Blackett’s War
Perilous Fight
The Bloody Shirt
Air Power
Battle of Wits
Natural History
The Character of Cats
The Truth About Dogs
The Nature of Horses
If a Lion Could Talk
Nature’s Keepers
The Covenant of the Wild
Fiction
Murder, by the Book
For Children
The World According to Horses
THIS IS A BORZOI BOOK PUBLISHED BY ALFRED A. KNOPF
Copyright © 2016 by Stephen Budiansky
All rights reserved. Published in the United States by Alfred A. Knopf, a division of Penguin Random House LLC, New York, and distributed in Canada by Random House of Canada, a division of Penguin Random House Canada Limited, Toronto.
www.aaknopf.com
Knopf, Borzoi Books, and the colophon are registered trademarks of Penguin Random House LLC.
Library of Congress Cataloging-in-Publication Data
Names: Budiansky, Stephen, author.
Title: Code warriors : NSA’s codebreakers and the secret intelligence war against the Soviet Union / by Stephen Budiansky.
Description: New York : Alfred A. Knopf, 2016. | Includes bibliographical references and index.
Identifiers: LCCN 2015045330
| ISBN 978-0-385-35266-6 (hardcover) | ISBN 978-0-38535267-3 (ebook)
Subjects: LCSH: United States. National Security Agency—History. |
Cryptography—United States—History | United States—Foreign relations—Soviet Union. | Soviet Union—Foreign relations—United States.
Classification: LCC UB256.U6 B83 2016 | DDC 327.7304709/045—dc23 LC record available at http://lccn.loc.gov/2015045330.
eBook ISBN 9780385352673
Cover image: Russian M-125 Fialka Cipher Machine (detail). Courtesy of Glenn Henry, Glenn’s Computer Museum.
Cover design by Chip Kidd
Maps by Dave Merrill
v4.1
a
TO DAVID KAHN,
who led the way
CONTENTS
Cover
Also by Stephen Budiansky
Title Page
Copyright
Dedication
Abbreviations
Author’s Note
Prologue: “A Catalogue of Disasters”
1 The Russian Problem
Arlington Hall, 1943—Assembly-line cryptology—“Getting everything”—The Soviet one-time-pad systems—Future of the special relationship—Spying in peacetime—“Like a deserted barn”
2 Unbreakable Codes
A defector in Ottawa—Reading depths—Russian teleprinters—TICOM and the Karrenberg Party—Caviar and Longfellow—“Low pay and too many military bosses”—Cable intercept and moral murkiness
3 Learning to Lie
NKGB messages and Soviet spycraft—A Russian genius for conspiracy—“Plausible deniability”—The perils of prosecution— Soviet rotor machines—Who interprets signals intelligence?
4 Digital Dawn
Computers for cryptanalysis—ERA’s Task 13—Special-purpose comparators—Abner, Goldberg, Demon, and Swish—“Black Friday,” October 1948—Russian plaintext on “the Plantation”
5 Shooting Wars
Ferret flights—Shootdown in the Baltic—Traffic analysis and ELINT—Looking the other way in Korea—MacArthur’s SIGINT blindness—Low-level intercept and the air war—The birth of NSA—Philby, Maclean, and Weisband
6 “An Old Mule Skinner”
“Reasonable dictator” Ralph J. Canine—Square spies, obtuse security—Inquisition by polygraph—A Dutch mole—Intercept overload—The move to Fort Meade
7 Brains Versus Bugs
Improving on Enigma—Theremin’s “Thing”—The Berlin Tunnel—TEMPEST—Hungary, Suez, and the chaos of 1956—Harvest, Lightning, and IBM hegemony—Information theory, and the changing of the cryptanalytic old guard
8 Days of Crisis
Martin and Mitchell—Glimmers of accountability—The “Boris deal” and cryptographic innocents abroad—SIGINT in space—Cuban Missile Crisis—Critics and Bullmeese
9 Reinventing the Wheel
Lyndon Johnson’s SIGINT fascination—Gulf of Tonkin, 1964—Learning to fight, again—Tet and the “Ultra Syndrome”— USS Pueblo—Growing spectrum, shrinking resources
10 Brute Force and Legerdemain
SIGINT sclerosis—“Disreputable if not outright illegal”—Battle of the bugs, continued—The Inman era and the last hurrah of the codebreakers, 1979—Pelton, Walker, and the “year of the spy,” 1985
Epilogue: The Collapse of the Wall, and a Verdict
Appendixes
A Enciphered Codes, Depths, and Book Breaking
B Russian Teleprinter Ciphers
C Cryptanalysis of the Hagelin Machine
D Bayesian Probability, Turing, and the Deciban
E The Index of Coincidence
Notes
Bibliography
Illustration Credits
A Note About the Author
Illustrations
ABBREVIATION
AEC Atomic Energy Commission
AFSA Armed Forces Security Agency
ASA Army Security Agency
ASAPAC Army Security Agency, Pacific
COMINT communications intelligence
CSAW Communications Supplementary Activities, Washington (U.S. Navy)
DF direction finding
ELINT electronic intelligence
FISC Foreign Intelligence Surveillance Court
GC&CS Government Code and Cypher School (UK)
GCHQ Government Communications Headquarters (UK)
GRU Main Intelligence Directorate (Soviet military intelligence)
HF high frequency
HUAC House Un-American Activities Committee
KGB Soviet Committee of State Security (1954–91)
MGB Soviet Ministry of State Security (1946–54; predecessor to KGB)
MI5 UK counterintelligence
MVD Soviet Ministry of Internal Affairs (1946–91)
NKGB People’s Commissariat for State Security (Soviet foreign security service, 1943–46; predecessor to MGB and KGB)
NKVD People’s Commissariat for Internal Affairs (Soviet internal security service, predecessor to MVD; incorporated State Security, 1934–43)
NSC National Security Council
NSG Naval Security Group
ONI Office of Naval Intelligence
Op-20-G Office of Naval Communications cryptanalytic section
OSS Office of Strategic Services
PFIAB President’s Foreign Intelligence Advisory Board
RAM rapid analytical machinery
SAC Strategic Air Command
SCAMP Special Cryptologic Advisory Math Panel
SIGINT signals intelligence
SIS Secret Intelligence Service (UK)
TICOM Target Intelligence Committee (Allied project to capture German cryptologists and cryptologic material at end of World War II)
USAFSS U.S. Air Force Security Service
USCIB U.S. Communications Intelligence Board
USIB U.S. Intelligence Board
AUTHOR’S NOTE
In May 2013, a twenty-nine-year-old computer security expert who had worked for three months as a $200,000-a-year contractor for the National Security Agency in Hawaii told his employer he needed to take a leave of absence for “a couple of weeks” to receive treatment f
or the epileptic condition he had recently been diagnosed with. On May 20, Edward J. Snowden boarded a flight to Hong Kong, carrying with him computer drives to which he had surreptitiously copied thousands of classified intelligence documents. Their contents, revealing copious details about NSA’s domestic surveillance of telephone and e-mail communications, would begin appearing two weeks later in a series of sensational articles in the Guardian and the Washington Post.
It was a move he had been secretly preparing for some time, having secured the job with the specific aim of gaining access to classified NSA material. (He was ultimately able to do so only by duping more than twenty coworkers into giving him their computer passwords, which he said he needed for his duties as a systems administrator; most of the colleagues whom he betrayed were subsequently fired.) Snowden would later explain that he chose Hong Kong as his place of intended sanctuary because “they have a spirited commitment to free speech and the right of political dissent”—an assertion that would have come as a surprise to members of the city’s pro-democracy movement, whose peaceful mass protests the following year would be efficiently crushed by the Hong Kong authorities at the behest of their Chinese Communist Party masters in Beijing.1
A month later, his U.S. passport canceled and under indictment for theft of government property and violations of the Espionage Act, Snowden fled to Moscow. There the government of President Vladimir Putin, a former lieutenant colonel of the Soviet KGB whose increasingly dictatorial control of the media, ruthless suppression of political opposition, and chest-thumping nationalist bellicosity was reviving the worst memories of the Cold War superpower confrontation, soon granted Snowden asylum, then temporary residency in Russia.
Snowden’s political naïveté was honestly come by: a self-taught computer whiz who never finished high school, a supporter of the quixotic campaign of the libertarian presidential candidate Ron Paul, he was given to sweeping, conspiratorial pronouncements about his duty to expose “the federation of secret law, unequal pardon, and irresistible executive power that rule the world.”2 Whatever his motives, there was no denying the impact of his revelations concerning NSA’s surveillance programs, particularly those involving the unauthorized monitoring of American citizens. No single incident in NSA’s sixty-one-year history came close to bringing so many of its most secret activities into the harsh glare of public scrutiny or so shook public confidence in the agency’s mission.
Three of the programs in particular seemed to epitomize a secret agency out of control, venturing well beyond the bounds of legitimate foreign intelligence gathering. The architects of the post–World War II permanent intelligence establishment, recognizing the fundamental incompatibility of deeply ingrained American beliefs in open government, liberty, and privacy with the tools of the shadowy intelligence trade—there had been much indignant talk about an “American Gestapo” when reports surfaced toward the end of the war that FDR was considering a plan to preserve the Office of Strategic Services (OSS), the forerunner of the Central Intelligence Agency, into the postwar period—sought to draw a sharp line that would resolve the dilemma. CIA and NSA would be strictly limited to foreign intelligence targets. Abroad, anything might go; it was after all a dangerous world, and the United States, having been wrenched from its long dream of isolationism, was determined never to be caught by another Pearl Harbor. But at home, the rule of law and American values would be maintained as always; a man’s home would still be his castle, and a warrant issued by a court would be required to search his personal effects or spy on his conversations.
Yet inevitably there were gray areas, and now it seemed that in the aftermath of the September 11, 2001, terrorist attacks on New York and Washington by the Islamic fundamentalist group al-Qaeda, the gray areas had spread across the entire horizon. One of the NSA programs Snowden revealed, the Bulk Telephony Metadata Program, employed secret orders issued to American telephone companies to obtain records of the duration and number dialed of every call made in the United States, and stored this information—billions of records, amassing five years’ worth of calling data—in vast data warehouses where they could be searched by NSA analysts.
The second, Prism, was an even more comprehensive dragnet; it collected from major Internet servers the contents of the billions of e-mails, Web videos, voice-over-Internet phone calls, and other data that passed through the United States to other countries. (A related “upstream collection” program directly tapped undersea fiber-optic cables to intercept the same kinds of Internet traffic.)3
A third program, the SIGINT Enabling Project—SIGINT stood for signals intelligence, NSA’s core mission—provoked outrage not only from civil libertarians but also from the high-tech computer and cybersecurity sector, which had enjoyed a close working relationship with NSA dating back to the very beginnings of the computer age in the postwar years. It was an industry upon which the agency depended more than ever as a source of expertise in the age of the Internet. Yet it turned out that NSA was at the same time undermining the industry’s commercial products by devising ways to insert hidden vulnerabilities that rendered otherwise unbreakable public encryption systems “exploitable.” According to a comprehensive description in one document leaked by Snowden, the methods NSA used included covert hacking of devices and networks as well as “investing in corporate partnerships” to ensure that cryptologic weaknesses, known only to NSA, were built into the products.
For years there had been conspiratorial whispers about the agency’s secret sabotaging of advanced digital encryption schemes via “back doors”; now it seemed that the conspiracy theorists had if anything underestimated the reality. Many in Silicon Valley took it as a personal betrayal: they felt they had been duped, and, more to the point, it was a terrible commercial blow to American businesses that offered computer security products now known to have been deliberately compromised.
Worse, it was a reckless and dangerous policy; as one computer expert told a White House review group, anything that makes it easier for NSA to spy on computer systems and the Internet “also inevitably makes it easier for criminals, terrorists, and foreign powers to infiltrate these systems for their own purposes.”4 (Even a system designed to allow law enforcement and intelligence agencies access to secure communications under a legally regulated framework would be the equivalent of leaving a key under the doormat, warned another high-level group of academic cryptologic experts; any such stored master key would itself become a target for hacking by Chinese government agencies, Russian organized crime syndicates, and others who had repeatedly demonstrated their skill in raiding U.S. government computer systems.)5
Six months later a federal district court found the bulk collection programs an unconstitutional violation of the Fourth Amendment’s privacy protections, calling particular attention to their “Orwellian” sweep. NSA was exploiting a vast loophole in Section 215 of the post–9/11 Patriot Act—which permitted the government to demand business records that were “relevant” to an investigation—to indiscriminately sweep up vast amounts of U.S. citizens’ data without a warrant, mining it for evidence of contact with foreign targets, then declaring it “relevant” after the fact when it did find such evidence. Such a procedure stood completely on its head a hallowed prohibition, enshrined in centuries of English common law and the U.S. Constitution, against exactly this sort of dragnet-like “general warrant.”
Earlier secret court orders, subsequently declassified by President Obama’s administration in an effort to respond to the Snowden revelations with a demonstration of commitment to “transparency,” revealed that the special federal court charged with overseeing NSA’s foreign-intelligence-gathering activities, the Foreign Intelligence Surveillance Court—itself the product of reforms enacted in the wake of revelations of NSA’s surveillance of American citizens during the 1960s and 1970s—had repeatedly chastised NSA for “substantial misrepresentations” of its bulk collection practices and for overstepping court-mandated rules to “minimize” the c
ollection of U.S. citizens’ data intercepted in the course of monitoring a legitimate foreign target.6
These initial disclosures undeniably raised important questions about domestic surveillance policy, legality, and morality. The FISC itself acknowledged that Snowden’s “unauthorized disclosure” of one of the court’s rulings had “engendered considerable public interest and debate,” and agreed that authorized declassification of additional rulings would similarly “contribute to an informed debate.” In May 2015, the U.S. House of Representatives voted 338–88 to end NSA’s bulk metadata collection program, and subsequently passed a Senate bill explicitly phasing out any such programs under Section 215, an outcome that would have been hard to imagine absent the public debate Snowden set off.7
But the classified information that continued to dribble out at well-timed intervals from Snowden soon crossed the line from defensible whistle-blowing to reckless exposure of ongoing foreign intelligence operations. Snowden—and even more so his chief journalistic collaborator, Glenn Greenwald of the British newspaper the Guardian—saw the world in simplistic terms: one was either a tool of the “establishment…elite,” sycophantically “venerating” and meekly obedient to “institutional authority,” or one daringly engaged in “radical dissent from it,” Greenwald asserted. Among the laundry list of disclosures Greenwald reported were NSA’s monitoring of radio transmissions of armed Taliban militants in northwest Pakistan, the bugging of twenty-four embassies (all identified by name), and technical specifics of listening devices used to intercept fax messages of foreign diplomats. Snowden and Greenwald presented these all as equally shocking evidence of the rise of a “menacing surveillance state” that was threatening to bring about the end of “privacy,” “internet freedom,” and “intellectual exploration and creativity” throughout the world.8
There was an astounding historical and moral blindness in lumping all these together. No one familiar with the diplomatic, military, and intelligence history of the preceding half century or more would have seen anything even worthy of surprise, much less anything illegal or immoral, in U.S. efforts to intercept and decipher the communications of foreign governments and military organizations using any means possible. Nor could they have so recklessly doubted the essential importance of foreign signals intelligence in safeguarding national security during that fraught chapter of world conflict.