Spycatcher
Page 24
- 13 -
As soon as I joined the Mitchell case, I was indoctrinated into the greatest counterintelligence secret in the Western world - the VENONA codebreak. To understand what VENONA was, and its true significance, you have to understand a little of the complex world of cryptography. In the 1930s, modern intelligence services like the Russian and the British adopted the one-time code pad system of communications. It is the safest form of encipherment known, since only sender and receiver have copies of the pad. As long as every sheet is used only once and destroyed, the code is unbreakable. To send a message using a onetime pad, the addresser translates each word of the message into a four-figure group of numbers, using a codebook. So if the first word of the message is "defense," this might become 3765. The figure 3765 is then added to the first group on the one-time pad, say 1196, using the Fibonacci system, which makes 4851. It is, in effect, a double encipherment. (The Fibonacci system is also known as Chinese arithmetic, where numbers greater than 9 are not carried forward. All cipher systems work on the Fibonacci system, because carrying numbers forward creates nonrandom distribution. )
The VENONA codebreak became possible because during the early years of the war the Russians ran short of cipher material. Such was the pressure on their communications system that they made duplicate sets of their one-time pads and issued them to different embassies in the West. In fact, the chances of compromising their communications were slim. The number of messages being transmitted worldwide was vast, and the Russians operated on five channels - one for Ambassadorial communications, one for the GRU, another for the Naval GRU, a fourth for the KGB, and lastly a channel for trade traffic connected with the vast program of military equipment passing from West to East during the war, which on its own comprised about 80 percent of total Russian messages. A set of pads might be issued to the KGB in Washington for their communications with Moscow, and its duplicate might be the trade traffic channel between Mexico and Moscow.
Shortly after the end of the war a brilliant American cryptanalyst named Meredith Gardner, from the U.S. Armed Forces Security Agency (the forerunner of the NSA), began work on the charred remains of a Russian codebook found on a battlefield in Finland. Although it was incomplete, the codebook did have the groups for some of the most common instructions in radio messages - those for "Spell" and "End-spell." These are common because any codebook has only a finite vocabulary, and where an addresser lacks the relevant group in the codebook - always the case, for instance, with names - he has to spell the word out letter by letter, prefixing with the word "Spell," and ending with the word "Endspell" to alert his addressee.
Using these common groups Gardner checked back on previous Russian radio traffic, and realized that there were duplications across some channels, indicating that the same one-time pads had been used. Slowly he "matched" the traffic which had been enciphered using the same pads, and began to try to break it. At first no one would believe him when he claimed to have broken into the Russian ciphers, and he was taken seriously only when he got a major breakthrough in the Washington-to-Moscow Ambassadorial channel. He decrypted the English phrase "Defense does not win wars!" which was a "Spell/Endspell" sequence. Gardner recognized it as a book on defense strategy published in the USA just before the date the message was sent. At this point, the Armed Forces Security Agency shared the secret with the British, who at that time were the world leaders in cryptanalysis, and together they began a joint effort to break the traffic, which lasted forty years.
Operation BRIDE (as it was first known) but later DRUG and VENONA, as it was known in Britain, made painfully slow progress. Finding matches among the mass of traffic available took time enough. But even then there was no certainty the messages on each side of the match could be broken. The codebook was incomplete, so the codebreakers used "collateral" intelligence. If, for instance, they found a match between the Washington-to-Moscow KGB channel and the New York-to-Moscow trade channel, it was possible to attack the trade channel by using "collateral," information gathered from shipping manifests, cargo records, departure and arrival times, tide tables, and so forth, for the date of the message. This information enabled the codebreakers to make estimates of what might be in the trade traffic. Once breaks were made in one side of a match, it provided more groups for the codebook, and helped make inroads on the other side.
The British and Americans developed a key device for expanding the VENONA breaks. It was called a "window index." Every time a word or phrase was broken out, it was indexed to everywhere else it appeared in the matched traffic. The British began to index these decrypts in a more advanced way. They placed two unsolved groups on each side of the decrypted word or phrase and after a period of time these window indexes led to repetitions, where different words which had been broken out were followed by the same unsolved group. The repetition often gave enough collateral to begin a successful attack on the group, thus widening the window indexes. Another technique was "dragging." Where a "Spell/Endspell" sequence or name came up, and the cryptanalysts did not know what the missing letters of the spelled sequence were, the groups were dragged, using a computer, across the rest of the channels, and out would come a list of all the repeats. Then the cryptanalysts would set to work on the reverse side of the repeat matches, and hope to attack the "Spell/Endspell" sequence that way.
It was an imperfect art, often moving forward only a word or two a month, and then suddenly spilling forward, like the time the Americans found the complete text of a recorded speech in the Washington Ambassadorial channel. Often terrible new difficulties were encountered: one-time pads were used in unorthodox ways, up and down, or folded, which made the process of finding matches infinitely more problematic. There were difficulties, too, with the codebooks. Sometimes they changed, and whereas the Ambassadorial, GRU, and trade channels used a straightforward alphabetically listed codebook, rather like a dictionary, so that the codebreakers could guess from the group where in the codebook it appeared, the KGB used a special multivolume random codebook which made decrypting matched KGB channels a mindbending task. The effort involved in VENONA was enormous. For years both GCHQ and NSA and MI5 employed teams of researchers scouring the world searching for "collateral"; but despite the effort less than 1 percent of the 200,000 messages we held were ever broken into, and many of those were broken only to the extent of a few words.
But the effect of the VENONA material on British and American intelligence was immense, not just in terms of the counterintelligence received, but in terms of the effect it had on shaping attitudes in the secret world. By the late 1940s enough progress was made in the New York/Moscow and Washington/Moscow KGB channels to reveal the extent of massive Russian espionage activity in the USA throughout and immediately after the war. More than 1200 cryptonyms littered the traffic, which, because they were frequently part of "Spell/Endspell" sequences, were often the easiest things to isolate in the traffic, even if they could not be broken. Of those 1200, more than 800 were assessed as recruited Soviet agents. It is probable that the majority of these were the low-level contacts which are the staple currency of all intelligence networks. But some were of major importance. Fourteen agents appeared to be operating in or close to the OSS (the wartime forerunner of the CIA), five agents had access, to one degree or another, to the White House, including one who, according to the traffic, traveled in Ambassador Averill Harriman's private airplane back from Moscow to the USA. Most damaging of all, the Russians had a chain of agents inside the American atomic weapons development program, and another with access to almost every document of importance which passed between the British and U.S. governments in 1945, including private telegrams sent by Churchill to Presidents Roosevelt and Truman.
Using leads in the decrypted traffic, some of these cases were solved. Maclean was identified as one of the sources of the Churchill telegrams, and many others besides; Klaus Fuchs and the Rosenbergs were unmasked as some of the nuclear spies; while comparison of geographical clues in the decryp
ts with the movements of Alger Hiss, a senior U.S. State Department official, over a lengthy period made him the best suspect as the agent on Harriman's plane. But despite frenzied counterintelligence and cryptanalytical effort, most of the cryptonyms remain today unidentified.
In Britain the situation was equally grim, but with one major difference. Whereas the Americans had all the Soviet radio traffic passing to and from the USA during and after the war, in Britain Churchill ordered all anti-Soviet intelligence work to cease during the wartime alliance, and GCHQ did not begin taking the traffic again until the very end of the war. Consequently there was far less traffic, and only one break was made into it, for the week September 15 to September 22, 1945, in the Moscow-to-London KGB channel.
There was a series of messages sent to a KGB officer in the London Embassy, Boris Krotov, who specialized in running high-grade agents. The messages came at a time of some crisis for the Russian intelligence services in the West. A young GRU cipher clerk in the Russian Embassy in Canada, Igor Gouzenko, had just defected, taking a mass of material incriminating spies in Canada and the USA, and in Britain a nuclear spy, Alan Nunn May. Most of the messages to Krotov from Moscow Center concerned instructions on how to handle the various agents under his care. Eight cryptonyms were mentioned in all, three of which were referred to as the "valuable ARGENTURA [spy ring] of Stanley, Hicks, and Johnson," two who were routinely referred to together as David and Rosa, and three others. By the end of the week's traffic all contact with the eight spies had been put on ice, and reduced to meetings, except in special circumstances, of once a month.
When I was indoctrinated into VENONA, I remember my first sight of the GCHQ copies of the Moscow-to-London KGB channel. Every time GCHQ broke a few more words in a message, they circulated to the very few users drop copies of the new decrypt. The copies were stamped TOP SECRET UMBRA VENONA, and listed the addresser and addressee, the date and time of the message, the channel and direction (for instance, KGB Moscow/London), and the message priority (whether it was routine or urgent). Underneath would be something like this:
TEXT OF MESSAGE
YOUR COMMUNICATION OF 74689 AND 02985 47199 67789 88005 61971 CONCERNING SPELL H I C K S ENDSPELL 55557 81045 10835 68971 71129 EXTREME CAUTION AT PRESENT TIME 56690 12748 92640 00471 SPELL S T A N L E Y ENDSPELL 37106 72885 MONTHLY UNTIL FURTHER NOTICE. SIGNATURE OF MESSAGE
(This is not a verbatim decrypt; merely a very close approximation to the kind of challenge we were faced with.)
VENONA was the most terrible secret of all, it was incomplete. It was obvious from the decrypts that each of the eight cryptonyms was an important spy, both from the care the Russians were taking to protect them all in September 1945, and because we knew that Krotov specialized in that type of agent. But there was precious little evidence from the traffic which could help us identify them. GCHQ circulated only translations which they had verified, and included the verbatim unsolved groups where they occurred, but they often attached to the copy a separate page of notes giving possible translations of the odd group, which had not yet been verified. Often a message would be repeated several times, as more groups were got out, and it was re-circulated.
Stanley, we were sure, must be Philby. Golitsin had heard the code name Stanley, and associated it with KGB operations in the Middle East, but there was no proof of this in the traffic. Hicks, therefore, was almost certainly Burgess because of the reference to the ARGENTURA, and because of a veiled reference to Hicks' temperament. Johnson was probably Blunt, although again there was no proof of it in the traffic. But the identity of the five other spies remained a mystery. Maclean was obviously not one of these, since he was in Washington in September 1945. The consequences for the Mitchell investigation were obvious. Any one of the five unidentified cryptonyms could be the spy inside MI5. I remember wondering, as I read the tantalizing decrypts, how on earth anyone at the top of MI5 had slept at night in the dozen years since they were first decrypted.
Perhaps the most extraordinary thing in the whole VENONA story was the fact that it was closed down on both sides of the Atlantic in 1954. After the initial surge of activity in the late 1940s and early 1950s, and the rash of prosecutions which followed, cryptanalytical progress slowed to a virtual halt. Hand matching had reached the limitations of the human brain, and computers were not then powerful enough to take the program much further. There was another reason too; in 1948 the Russians began to alter their code procedures worldwide, removing all duplicated pads. The last casualty of this was the Australian VENONA operation, which had been making so much progress that the British and Americans were virtually reading the Russian ciphers continuously as the messages were produced. The Australians were never told at the time but were brought into it some years later, although when the extent of Soviet espionage penetration, especially of the Department of External Affairs, became apparent, they were provided with the intelligence in bowdlerized form, and it led to the establishment of ASIO (Australian Security Intelligence Organization) with MI5's help.
The reason for the change in Soviet codes became apparent in the early 1950s. The secret of the break had been leaked to the Russians by a young Armed Services Security Agency clerk, William Weisband. In fact, Weisband did not know the extent of the Russian mistake and it was only when Philby was indoctrinated in 1949 that they knew the breadth of their disaster, although other people, such as Roger Hollis, were indoctrinated in 1948, when the match suddenly ceased in Australia after he returned from organizing the setting up of ASIO. Although the duplicate one-time pads were withdrawn, the Russians could do nothing to prevent the continuing work on the traffic they had already sent up until 1948. But thanks to Philby's posting to Washington in 1949, they were able to monitor the precise progress that was being made. Once the Russians knew the extent of the VENONA leak, and the technical difficulties of finding more matches multiplied, it was only a matter of time before priorities moved on. In 1954 most of the work was closed down.
Years later, I arranged for Meredith Gardner to visit Britain to help us on the British VENONA. He was a quiet, scholarly man, entirely unaware of the awe in which he was held by other cryptanalysts. He used to tell me how he worked on the matches in his office, and of how a young pipe-smoking Englishman named Philby used to regularly visit him and peer over his shoulder and admire the progress he was making. Gardner was rather a sad figure by the late 1960s. He felt very keenly that the cryptanalytical break he had made possible was a thing of mathematical beauty, and he was depressed at the use to which it had been put.
"I never wanted it to get anyone into trouble," he used to say. He was appalled at the fact that his discovery had led, almost inevitably, to the electric chair, and felt (as I did) that the Rosenbergs, while guilty, ought to have been given clemency. In Gardner's mind, VENONA was almost an art form, and he did not want it sullied by crude McCarthyism. But the codebreak had a fundamental effect on Cold War attitudes among those few indoctrinated officers inside the British and American intelligence services. It became the wellspring for the new emphasis on counterespionage investigation which increasingly permeated Western intelligence in the decades after the first break was made. More directly, it showed the worldwide scale of the Soviet espionage attack, at a time when the Western political leadership was apparently pursuing a policy of alliance and extending the hand of friendship. In the British traffic, for instance, most of the KGB channel during that September week was taken up with messages from Moscow detailing arrangements for the return of Allied prisoners to the Soviet authorities, groups like the Cossacks and others who had fought against the Soviet Union. Many of the messages were just long lists of names and instructions that they should be apprehended as soon as possible. By the time I read the messages they were all long since dead, but at the time many intelligence officers must have been struck by the sense that peace had not come in 1945; a German concentration camp had merely been exchanged for a Soviet Gulag.
In 1959, a new discovery was
made which resuscitated VENONA again. GCHQ discovered that the Swedish Signals Intelligence Service had taken and stored a considerable amount of new wartime traffic, including some GRU radio messages sent to and from London during the early years of the war. GCHQ persuaded the Swedes to relinquish their neutrality, and pass the material over for analysis. The discovery of the Swedish HASP material was one of the main reasons for Arthur's return to D1. He was one of the few officers inside MI5 with direct experience of VENONA, having worked intimately with it during the Fuchs and Maclean investigations.
There were high hopes that HASP would transform VENONA by providing more intelligence about unknown cryptonyms and, just as important, by providing more groups for the codebook, which would, in turn, lead to further breaks in VENONA material already held. Moreover, since powerful new computers were becoming available, it made sense to reopen the whole program (I was never convinced that the effort should have been dropped in the 1950s), and the pace gradually increased, with vigorous encouragement by Arthur, through the early 1960s.
In fact, there were no great immediate discoveries in the HASP material which related to Britain. Most of the material consisted of routine reports from GRU officers of bomb damage in various parts of Britain, and estimates of British military capability. There were dozens of cryptonyms, some of whom were interesting, but long since dead. J.B.S. Haldane, for instance, who was working in the Admiralty's submarine experimental station at Haslar, researching into deep diving techniques, was supplying details of the programs to the CPGB, who were passing it on to the GRU in London. Another spy identified in the traffic was the Honorable Owen Montagu, the son of Lord Swaythling (not to be confused with Euan Montagu, who organized the celebrated "Man Who Never Was" deception operation during the war). He was a freelance journalist, and from the traffic it was clear that he was used by the Russians to collect political intelligence in the Labor Party, and to a lesser degree the CPGB.