The Secret War

by Max Hastings

  Tokyo’s anodyne reply was inevitable: ‘The foreign policy of Japan is determined upon after an unbiased examination of all the facts and a very careful weighing of all the elements of the situation she confronts.’ Yet such a process never took place in Tokyo. Japan’s rejection of strategic intelligence assessment, and of rational decision-making, was responsible for its commitment to a path to catastrophe on 7 December 1941.

  The only species of intelligence the Japanese high command treated seriously was that which concerned immediate objectives. Thus, in the months before the army and navy went to war, both energetically probed the defences of the European South-East Asian empires, the Philippines and Pearl Harbor. Tokyo’s appetite for an advance into Indochina was sharpened by the July 1940 decryption of a telegram to Washington from the American consul in Saigon, saying that the British would make no military response to such a Japanese initiative without a promise of US support, which would not be forthcoming. Further decrypts of diplomatic messages in August, notably including those of the Vichy French authorities in Indochina, confirmed the inability or unwillingness of the Western Powers to resist a Japanese takeover. Thus, on 22 September, the Japanese forced a French signature on an agreement which admitted their troops next day. Here was a case where intelligence played a significant role, albeit in confirming Japan’s commitment to a course its rulers favoured anyway.

  The army’s South-East Asia Group, established in 1939, was responsible for exploring the defences of the European empires. Because Thailand was the only independent country in the region, the Japanese made it the hub of their intelligence-gathering, directed by military attaché Col. Hiroshi Tamura, who focused especially on identifying Malayan invasion routes. His soldiers measured every road and bridge from Indochina and Thailand into Malaya. Agents explored the huge Dutch oil refineries at Palembang so diligently that when Japanese paratroops later descended upon them, each man knew every detail of the target. The Japanese acknowledged that the British had some formidable codebreaking talent, focused in their Far East Combined Bureau in Singapore, which worked closely with Bletchley Park. A naval codebreaker who monitored its traffic, Commander Monotono Samejima, decrypted material which showed that the British had been reading some low-grade Japanese signals within twenty-four hours of transmission. Samejima recalled later: ‘I became aware of the tremendous capability of UK intelligence.’ His superiors, however, cared only about counting their enemies’ soldiers. By the end of 1940 a thousand Japanese ‘tourists’, all graduates of the army intelligence school, were working out of Thailand. British troops exercising in Malaya found themselves followed everywhere by bicycling Japanese equipped with pencils and notebooks. Their findings were circulated throughout the army as an ‘Intelligence record of British Malaya’, which included maps of the Singapore garrison’s installations. The general staff concluded from such agents’ reports that the ethnic diversity of British imperial forces was a weakness. It was dismissive of the Australians: ‘Their quality is bad. The troops are composed chiefly of jobless men and rough individuals. They are not a well-disciplined army. Their valour in battle is famous, but their training and equipment are not adequate.’ Tokyo asserted that many Indian soldiers were both ill-trained and anti-British; they could fight bravely in a head-on positional clash, but were vulnerable to rapid flanking movements – a shrewd assessment, vindicated by events in the subsequent campaign.

  In the course of 1941 Japanese agents made contacts with Indian, Malayan and Burmese nationalist groups, offering covert support for their ambitions for independence, which many found an attractive proposition. A January 1941 telegram from Tokyo to Japan’s consul-general in Singapore ordered him to accelerate ‘agitation, political plots, propaganda and intelligence’. In May, Japan’s Foreign Service cabled all its missions in the region, urging them to hasten the expansion of clandestine networks on the periphery of the Indian Ocean, because war was obviously looming. The Indians, especially, were urged to promote disaffection among soldiers of the Raj, and began to do so in 1939; when the assault on Malaya was unleashed, the poor performance of several Indian regiments suggested that the Japanese propaganda offensive had achieved some success: the 1st Hyderabads, for instance, are believed to have shot their own British colonel and adjutant at an early stage of the battalion’s disintegration on the battlefield.

  Tokyo thought British aircrew green, and it was true that most of the pilots deployed in Malaya and Burma were less experienced than their enemy counterparts. The Japanese probably received some information from a traitor in the British ranks: for several months during 1941 Captain Patrick Heenan of 300 Air Intelligence Liaison Section appears to have wirelessed information about RAF dispositions from a secret transmitter. Thirty-one, born in New Zealand, Heenan had inherited from his father a warm sympathy for the Irish Republican Army. He was recruited by Tokyo during a 1938 leave spent in Japan, and ended his espionage career by being shot against a harbour wall in Singapore just before its fall.

  As for Pearl Harbor, in August 1941 twenty-eight-year-old Ensign Takeo Yoshikawa of the US & British section of Japanese naval intelligence arrived in Hawaii with diplomatic cover, and spent the months that followed exploring every accessible area of interest to the planners back at home, while conducting an uncommonly energetic off-duty love life. He reported three times a week by Purple cipher, and though his messages were routinely intercepted, the US Signals Intelligence Service in Washington was often three weeks in arrears decrypting them, not least because it had only two fluent and accurate Japanese linguists. In October, Yoshikawa’s boss Lt. Cmdr Minato Nakajima made a personal visit on a Japanese liner, and received the ensign’s written report on the US Navy’s local strengths, deployments and defences. Though Yoshikawa was interned after Pearl Harbor, he was subsequently allowed to return home under the agreement on exchange of diplomatic personnel.

  Further information was acquired by the local consular staff, assisted by the large Japanese expatriate colony on Hawaii – 41,346 of them. Their data confirmed the navy in its determination to strike at Pearl Harbor rather than against the alternative American anchorage at Lahaina Port. Neither the army nor the Ministry of Foreign Affairs was informed about the agreed objectives, and Pearl was never explicitly mentioned in naval radio traffic. The critical mistake in the planning of the onslaught was not one of intelligence collection, but of analysis: the admirals back in Japan failed to recognise the importance of Pearl’s huge oil-tank farms and repair facilities, and never included these in their target programme. As for Japanese perceptions of the US Army, they assessed American troops – especially the Philippines garrison – as individualists who lacked staying power and fighting spirit for a protracted struggle. Japan’s generals planned for a victorious termination of the Pacific war in the spring of 1942, whereupon they would launch a major assault on the Soviet Union.

  Thus Japan went to war knowing much about its immediate objectives, but wilfully ignorant and naïve about what would follow. Yamamoto’s air squadrons took off for Pearl Harbor on 7 December 1941 at just the moment when German failure before Moscow was becoming apparent in Berlin. The only success of the advocates of caution in Tokyo was to persuade Japan’s rulers to confine themselves to assaulting the US and Western European empires, and to avoid joining hostilities with the Soviet Union until German victory seemed imminent. The judgements on which the Japanese based their decision to fight – to shackle themselves to a tottering giant – were fantastically ill-informed. They overvalued German might, underrated that of the United States. Moreover, they persuaded themselves that they could conduct a limited war, which they could terminate by negotiation at a moment of their own choice. Instead, of course, they found themselves engaged in an existential struggle in which they must either achieve total victory or face almost annihilatory defeat.

  3 THE MAN WHO WON MIDWAY

  The surprise suffered by the United States at Pearl Harbor was as great as that which the Soviet Union incu
rred at the launch of ‘Barbarossa’, and equally inexcusable. Its army codebreakers led by Frank Rowlett had achieved an extraordinary feat by cracking Japan’s ‘Purple’ diplomatic cipher in August 1940. During the weeks and days before the Japanese attack on Hawaii, almost as much information became available to the US government to indicate the imminence of war as Stalin received before Hitler attacked him. But the US administration’s response was as supine as had been that of the Kremlin. Just as the British required many months under the stimulus of war before they developed effective machinery for managing and exploiting intelligence, so the American armed forces began to do so only amid the wreckage of the battleships of its Pacific Fleet.

  Admiral John Godfrey’s July 1941 British report on US intelligence concluded that ‘cooperation between the various organisations is inadequate and sources are not coordinated to the mutual benefit of the departments concerned. There is little contact between the intelligence officers of the different departments and the desire to obtain a “scoop” is fairly general … The value of the material obtained by the US intelligence organisations from Europe is not considerable, though information on the Pacific area and South America is … on the whole, high-grade … The Office of Naval Intelligence is in danger of degenerating into a graveyard for statistics because it is inclined to regard intelligence as an end in itself … Many of the faults which have been enumerated will be recognised as similar to those from which British Intelligence suffered before the war.’ Godfrey concluded that once William Donovan’s new intelligence-gathering organisation got going, many good things might be possible, ‘but it would be prudent to conclude that US Intelligence is unlikely to be of much assistance to the joint war effort for many months to come’.

  This proved to be true. The US Army and US Navy contributed nothing significant to Allied knowledge of Axis motions for months after Pearl Harbor. But then, in June 1942, from out of a dank basement in the Navy Yard on Oahu came a single piercing shaft of light that illuminated the entire Pacific theatre. It made possible the US Navy’s victory at Midway, which inside forty-eight hours transformed the course of the war against Japan. It was arguably the most influential single intelligence achievement of the global conflict.

  While the British even before the war deployed brilliant civilians to spearhead their codebreaking, the US Navy chose to rely instead upon an almost randomly selected group of career officers, among the least valued of their service. With pitifully slender resources, their achievement could not match in scale that of Bletchley Park, nor did it save the US from humiliation on the December 1941 ‘Day of Infamy’. But the officer who made the greatest single contribution to subsequent triumph at Midway retired from his service with little honour, and went to his grave known only to historians.

  Joseph Rochefort cut an awkward figure: he was a poor seaman, with no talent for making important friends. Without his gifts, however, it is unlikely that a decisive battle would have been fought in the Pacific between 4 and 7 June 1942, and even less plausible that it would have been won by the United States. What happened that day was the outcome not of a sudden flash of inspiration, but of two decades of weary, thankless labour.

  Rochefort was born in 1900, youngest son of Irish parents; his father was an Ohio rug salesman. An untidy child, in high school he excelled only at maths. At seventeen he enlisted in the US Navy as an electrician 3rd class, then scraped a commission as a reservist and became an engineer. At twenty-one he married Elma Fay, his childhood sweetheart, though he was Catholic and she a Baptist. In 1921 he managed to transfer to the regular navy, but his career languished: he narrowly escaped court-martial when a tanker on which he was duty officer dragged its anchor in San Francisco Bay amid six destroyers. In 1925 he was detached from service on the battleship Arizona to study cryptanalysis, for which skill at bridge and crosswords seemed to fit him. He worked in the Navy Department on Washington’s Constitution Avenue, but the assignment did not represent promotion: intelligence ranked low on the service’s totem pole. The US Navy had thus far achieved nothing to match the achievement of the army’s codebreaking department, the ‘Black Chamber’ established in 1917 under Herbert Yardley, which broke a Japanese diplomatic cipher as early as 1921.

  The navy was learning, however. Rochefort started by reading the book Elements of Cryptanalysis, written by the War Department’s William Friedman. He worked under the brilliant Lt. Laurance Safford, a former chief yeoman in the US Naval Reserve, who became his tutor and mentor. They were assisted by a civilian, Agnes Meyer Driscoll, who also made a notable contribution. Rochefort found himself enjoying the work. Breaking a code, he said later, ‘makes you feel pretty good, because you have defied these people who have attempted to use a system they thought was secure … It was always somewhat of a pleasure to defeat them.’ In February 1925 Safford departed for an almost mandatory spell of sea duty, leaving Rochefort in charge of the research desk. This consisted of only three full-timers: himself; an ex-actor named Claus Bogel, who did little to justify his rations; and Driscoll, dubbed ‘Madame X’, who cursed fluently, despised make-up, but rubbed along pretty well with Rochefort. Japan’s secrets were always the principal targets. In 1920 the Office of Naval Intelligence had run a ‘black bag job’, photographing a copy of the Japanese Red Code from the New York consulate. It became obvious that higher language skills were indispensable if the codebreakers were to make serious headway: Lt. Cmdr Ellis Zacharias, a career intelligence officer and fluent Japanese speaker, was drafted into the section to work alongside Rochefort.

  Zacharias wrote later: ‘The few persons who were assigned to this section were taciturn, secretive people who refused to discuss their jobs … Hours went by without any of us saying a word, just sitting in front of piles of indexed sheets on which a mumbo-jumbo of figures or letters was displayed in chaotic disorder.’ Given that the United States was at peace and determined to stay that way, the intensity with which a handful of naval officers laboured at their arcane craft seems extraordinary, and far removed from the lazy tempo that prevailed elsewhere in the US Navy. Those men and that one remarkable woman were obsessives, who worked all hours and ignored Sundays. The office was permanently shrouded in tobacco smoke: Rochefort chain-smoked cigarettes, a pipe, the odd cigar. They sometimes went home too tired to eat until they had unwound for several hours. All lost weight.

  Safford had established a chain of intercept stations in Shanghai, Hawaii and elsewhere, which lifted Japan’s signals from the ether. Once the messages reached Rochefort’s office, the section worked together to crack them, with Agnes Driscoll probably the ablest analyst. Occasionally, new officers were sent to them, to test their suitability as codebreakers. Most were washed out: they lacked the peculiar, indispensable sense for the rhythm of puzzles.

  After two years, Rochefort had had enough – not of codebreaking, but of Navy Department politics. His section’s activities came under Communications, but Intelligence waged constant war to take it over. Still a lieutenant, Rochefort became executive officer of a destroyer. At sea his tactlessness, carried to the point of boorishness, exasperated superiors. In his spare time he checked out the US Navy’s codes, and told the commander-in-chief that the communications system was clogged with trivial messages that should never have been encrypted. This was poorly received.

  In 1929 his old colleague Ellis Zacharias arranged for him to be posted to Japan, to learn the language. While serving there for three years, Rochefort became a close friend of another American naval officer, Edwin Layton; it is a measure of his almost morbid sense of discretion that in all their hours together, he told Layton nothing about his background in cryptanalysis. Thereafter, Rochefort spent most of the 1930s at sea, successively as a gunnery, intelligence and navigation officer. In October 1939 he was posted to Pearl Harbor, where he was appalled by the casual routines, but shared the delusions of his superiors that no enemy would dare to attack Hawaii or the Philippines.

  Laurance Safford now headed Op-2
0-G, the navy’s codebreaking operation. With half the world at war this was modestly expanding, while Japanese relations with the US deteriorated. Between 1934 and 1939 the US government and armed forces had faithfully respected domestic law – explicitly Section 605 of the 1934 Federal Communications Act, which barred interception of messages between US and foreign countries, radio or cable. Thereafter, George Marshall granted some latitude to the various codebreaking agencies and their eavesdroppers, which made possible the small miracle of Purple. Safford’s activities, like Friedman’s, nonetheless represented lawbreaking. He asked for Rochefort to head up the Pearl station, known as COM 14. Rochefort accepted the posting reluctantly, because of his memories of the departmental struggles. But where else was he to go, at forty-one, with nobody clamouring for his services? In June 1941 he took up his new responsibilities, reporting direct to Admiral Husband Kimmel, Pacific Fleet C-in-C. Only belatedly, four months later, did he receive a long-delayed promotion to commander.

  COM 14 – ‘Station Hypo’ – where Rochefort and his team were to make history, was quartered in the echoing, unlovely basement of the Navy Yard administration building, entered through a time-locked unmarked door guarded by marines. ‘The Dungeon’, as it was dubbed by inmates, looked like a small-town pool hall, even unto the chronic smoke haze, because everybody worked with a cigarette stuck in his mouth. A chief petty officer, Tex Rorie, sat at a desk by the door, screening visitors. The floor was undressed concrete, the walls were painted with mud-coloured sealant, and the primitive ventilation system recycled stale air. When Rochefort first joined, much of the basement’s hundred feet by fifty was emptiness, but through the months that followed, it filled rapidly. A battery of IBM Hollerith punch-card tabulating machines clattered relentlessly: their contribution to Hypo’s successes, as indeed to all US codebreaking, deserves emphasis. In September, five Japanese-language officers joined the section. Soon there were twenty-three men, working in four sectors: the language team, traffic analysts, ship-plotters, cryptanalysts. Lt. Cmdr Thomas Dyer, a short, dark officer with heavy spectacles who looked more like an eccentric professor than a naval officer, was considered the best of the latter, and became a hero of America’s cryptographic war. Others included Lt. Ham Wright, who bore a passing resemblance to the actor Wallace Beery; and Lt. Cmdr Jack Holtwick. Rochefort no longer thought of himself as a cryptanalyst; instead, he said, ‘[I] fancied myself a translator.’