Impeached: The Trial of President Andrew Johnson and the Fight for Lincoln's Legacy

by David O. Stewart

  This track record should have given pause to would-be impeachers in 1867. The two successful impeachments involved a mental incompetent and a turncoat; neither had contested the charges against him. Those two convictions did not resolve whether impeachment was reserved for true crimes, or was available for abuses of office that might be called “political” offenses. Judge Pickering was convicted of conduct that was certainly no crime (mishandling his cases and being drunk on the bench), but Judge Humphreys’s actions on behalf of the Confederacy likely were criminal acts, even treason. The acquittals of Chase and Peck involved charges that also did not seem particularly like crimes.

  For the case of Andrew Johnson, the Constitution’s failure to define “high crimes and misdemeanors” would be pivotal. The early calls for his impeachment foundered because they expressed general outrage over his “usurpation” of congressional powers; they were never framed in terms of crimes. As Senator Sumner of Massachusetts put it in January of 1867, “The President has usurped the power of Congress on a colossal scale, and he has employed these usurped powers in facilitating a rebel spirit and awakening anew the dying fires of the rebellion.”

  The investigation of the House Judiciary Committee got under way slowly. Democrats predicted that the effort would never begin, while one Radical grumbled that Thad Stevens was “too old and feeble to fight the great battle.” Some Republican newspapers cautioned against impeachment, predicting that the effort would end up injuring those who undertook it. Through his impeachment ordeal, the president would work hard to present a calm face to the nation, but his anger over the House’s inquiry was clear to intimates. One called him “cross as a cinnamon bear” over it. The rage of his navy secretary singed the pages of his diary: “There is nothing judicial or fair in this proceeding. It is sheer partisanism with most of them, a deliberate conspiracy with the few…. A committee is sitting in secret,—a foul conspiracy,—trying to hunt up charges and evidence against as pure, as honest, as patriotic a chief magistrate as we have ever had.”

  The House committee began formal (though secret) hearings on February 6. The famous detective Lafayette Baker, who had tracked down John Wilkes Booth after the Lincoln assassination, testified first. Baker described a wartime letter he supposedly carried, but no longer possessed, on a date he could not recall, from Johnson to Confederate President Jefferson Davis. The detective did not know the contents of the letter but thought it implied that Johnson would “go with them.” Baker declined to state from whom he received the letter. He then described his actions the previous year to prevent Mrs. Lucy Cobb—“a disreputable woman, or, in other words, a woman of the town”—from visiting the White House. Supposedly, Mrs. Cobb disclosed to him both President Johnson’s secret methods for communicating with “his friends in the South” and her lively trade in selling presidential pardons to former Confederates. Having implicated Johnson in treason, bribery, and prostitution without producing evidence of any offense whatever, Baker retired.

  The hearings never recovered, careening from unsupported allegations to nefarious innuendos, all at numbing length. The committee heard testimony that missing portions of John Wilkes Booth’s personal diary might have implicated Johnson in the Lincoln assassination. It explored whether the president used intimidation or patronage inducements to block the admission of Colorado as a state. Had Johnson really asked his attorney general for a legal opinion on the legitimacy of a Congress that excluded Southern congressmen? It turned out that a newspaper correspondent fabricated that report because he “supposed” the president would want such a legal opinion and—if Johnson did want it—the attorney general was the logical person to ask for it. Had the president, the committee asked, issued pardons to West Virginia soldiers who deserted from duty? It also inquired into whether the commission for the minister to Sweden had been issued correctly.

  The committee examined the sale of Southern railroads seized by the Union Army during the war, exploring the president’s connection to Southerners who recovered those assets for nominal or no payment. Was this corrupt favoritism? Not according to Secretary of War Stanton, darling of the Radicals. Stanton took personal responsibility for the railroad transfers. The government did not know how to operate railroads, he told the committee; the nation’s economic recovery depended on delivering them into the hands of those who did. Other testimony bounced from the New Orleans massacre to allegations of import tax frauds in New York City.

  When the Thirty-Ninth Congress expired on March 3, the Judiciary Committee declined to report any findings on impeachment but recommended that the inquiry continue. When the Fortieth Congress convened the next day, a new impeachment champion entered the arena, Ben Butler of Massachusetts. As a Democrat who early supported the Northern war effort, Butler had used influence to win high military commands during the war. As a general, he showed political flair, not military skill. A smart lawyer with a jaded view of human nature and the instincts of a demagogue, Butler was a dangerous adversary. Ashley might be portrayed as a fanatic, but Butler had to be taken seriously. He would leave his mark on any enterprise he took up.

  Butler immediately pushed the Republican caucus to create a special impeachment panel, but Ashley managed to keep the matter before the Judiciary Committee. Ashley darkly told the House of Johnson’s “complicity in the assassination plot,” of the “mysterious connection between death and treachery which this case presents,” and of the need to remove Johnson, “the loathing incubus which has blotted our country’s history.”

  Johnson and his allies grew increasingly short-tempered with an investigation that was out of control. By May, Navy Secretary Welles dismissed the impeachment inquiry in a diary entry: “No facts, no charges, no malconduct are known or preferred, for the slip-slop of Ashley was long since discarded. A more scandalous villainy never disgraced the country.” When the committee subpoenaed his bank records, the president vented his wrath to an aide: “I have had a son killed, a son-in-law die during the last battle at Nashville, another son has thrown himself away, a second son-in-law is in no better condition. I think I have had sorrow enough without having my bank account examined by a Committee of Congress.” Nevertheless, he produced the records. A Republican senator pronounced the investigation “a complete failure.”

  By early June, the Judiciary Committee had heard enough. On a 5-to-4 vote, it declined to approve impeachment articles against Johnson. Johnson had dodged the impeachment bullet.

  Then he stepped back into the line of fire.

  The Reconstruction Acts that passed over Johnson’s vetoes that spring created five military districts across the South. The commanders of those districts had broad powers to enforce the law and oversee elections. The commanders could supersede orders of state and local governments and replace state and local officials. Southerners and Democrats denounced the system as despotic; it surely was authoritarian.

  The new laws instructed ten Southern states (all the seceding states but Tennessee) to ratify the Fourteenth Amendment and to grant the vote to blacks (who in 1867 could vote in only five of the Northern states). The goal of the legislation was to establish state governments in the South that were controlled by the few Southern Republicans and Unionists and the many freed slaves. This new leadership, unconnected with secession and rebellion, would replace the traditional Southern political structure. The reaction in the South was immediate. For the first time ever, blacks and whites mingled at political meetings as both groups tried to understand the new system.

  The president abhorred the Reconstruction statutes. Their result, he warned in a newspaper interview, would be “a war of races.” Because the executive branch had to implement the new system, he could obstruct it. For well over a year, he had actively undermined the Freedmen’s Bureau. He reassigned Bureau personnel who displeased him. He authorized investigations that paralyzed the agency’s initiatives. He reclaimed land that the Bureau had distributed to the freedmen, then returned it to former Confederates.

/>   To drain the vitality from the Reconstruction Acts, Johnson turned to his attorney general, Henry Stanberry (who had been kept off the Supreme Court when Congress shrank that court a year before). Johnson asked for legal opinions interpreting the powers of the military commanders in the South. In late May and mid-June, Stanberry issued opinions that reached the conclusions sought by the president, over War Secretary Stanton’s objections. According to the attorney general, the commanders had no powers to remove local officials who thwarted congressional policies. Stanberry also reasoned away the restrictions on voting by former Confederates. The Republican congressmen who wrote the statutes were stunned by the narrowness of their legislation when Stanberry was finished.

  Johnson’s use of the attorney general’s opinions was daring and provocative, undermining the plain intent of Congress. Republicans in Congress, and the military commanders, grew irate. As one of General Grant’s aides wrote, “The whole force of the Reconstruction measure lay in the power of the District Commanders to remove [Southern] civil officers who opposed or obstructed the new law.” The attorney general had eliminated that power.

  Navy Secretary Gideon Welles recognized the risks of the president’s maneuver. He predicted that it would ensure Johnson’s impeachment. Congress responded swiftly. The House Judiciary Committee decided to resume taking evidence on impeachment, then Congress adopted a third Reconstruction Act, over Johnson’s veto, to reverse Stanberry’s opinions. In yet another veto message, Johnson cried out that it was “impossible to conceive any state of society more intolerable than this.” He called for a time when “the rod of despotism will be broken, [and] the armed heel of power lifted from the necks of the people.” Congress immediately overrode that veto. Its leaders began to think hard about impeachment.

  In the eyes of Thad Stevens, the impeachment effort had been botched. Annoyed by the committee’s endless poking into corners of Johnson’s financial and official conduct, he complained that “the committee are making but a mere pretense of prosecuting the impeachment…. I do not believe they ever intended it.” He called the investigation “fussy, unnecessary, and absurd.”

  Stevens saw the case in simple terms. It was the president’s policies, he insisted, that warranted his removal. Asked in late June what grounds existed for impeachment, Stevens instantly answered, “his unlawful usurpation of the conquered territory and his attempt to raise up states therein,” which made him “as rank a usurper as was Caesar or Cromwell.” Those charges could be proved without any testimony at all. Stevens predicted that impeachment would fail, complaining that too many congressmen had “no bone in their backs and no blood in their veins.”

  Though Congress’s first exploration of impeachment had been a mess, the effort drew new life from Johnson’s determined effort to thwart the Reconstruction laws. Indeed, it seemed that so long as Andrew Johnson remained in the White House, he was bound to outrage his adversaries. When Congress left Washington in late July of 1867, the president got right to work on that.

  In early 1867, several leading Republican congressmen leveled the threat of impeachment at a different federal official. Henry Smythe was collector of the New York Custom House, the most lucrative public job in the nation. A New York banker with conservative credentials, Smythe supervised the 1,200 political appointees who administered the import taxes for New York Harbor, a golden river of revenue. Smythe’s employees kicked back to the party in power 2 percent of their salaries, while some estimated in 1860 that the collector received more than three times his salary in “pickings and fees.”

  Smythe gained the plum position in 1866 by promising Johnson that he would “serve and sustain the President throughout.” Smythe also agreed to pay $5,000 to Senator David Patterson of Tennessee, who was Johnson’s son-in-law, and another $5,000 to a Wisconsin senator, James Doolittle, who was a loyal Johnson supporter. Though Smythe denied actually making the payments, the new collector hired Doolittle’s son when he replaced almost 400 Republican employees from Lincoln’s time.

  In the winter of 1867, a congressional committee investigated the graft at Smythe’s Custom House. In mid-March, the committee chairman declared that “corruption reigns there,” calling Smythe’s management “the most shameless, flagitious, disreputable, utterly disgraceful, yet deliberate system of abuse, extortion, wrong, that has been developed in scores of years.” Much attention focused on Smythe’s practice of leasing to private parties the government warehouses that held imported goods before they were taxed. Smythe customarily received, personally, one-fourth of the lease revenue from each warehouse; one of the lessees paid him $40,000 a year, at a time when the president earned only $25,000. (Although there are several ways to calculate the current value of a dollar from 1868, a conservative estimate would peg its worth at fourteen current dollars; by that measure, Smythe’s $40,000 payoff was equivalent to $560,000 in today’s dollars.)

  A movement formed to impeach Collector Smythe. At the instigation of the House committee and the Radical press, a resolution denouncing him was debated in March 1867. Smythe responded with a mixture of excuses and defiance. He succeeded in riding out the storm, in large part because President Johnson stood by his crooked collector. Smythe would repay the president’s loyalty during a different impeachment season.

  THE DANGEROUS SPHINX

  AUGUST–NOVEMBER 1867

  The President don’t comprehend Grant.

  WILLIAM TECUMSEH SHERMAN, OCTOBER 7, 1867

  WALKING INTO ANDREW Johnson’s White House on August 1, 1867, General Ulysses Grant had to have conflicting feelings. Johnson, who would soon offer to appoint him interim secretary of war, was his superior officer. The Constitution made the president the commander-in-chief of all military forces. But Grant’s loyalties did not follow organization charts, even the one established by the Constitution, and definitely not with this president.

  Despite his mild demeanor, Grant was an unusually poor subordinate. For some years, he had paid little attention to the views of his nominal superiors. Through the Civil War and since, Grant had grown used to relying on his own judgment, even to the point of disobeying direct orders. Grant’s approach works only for generals who win, and not always for them, but the quiet man pulled it off with a unique combination of battlefield success and disarming modesty. When reports circulated during the war that Grant and Stanton had argued, Lincoln’s secretary dismissed them, insisting, “Grant quarrels with no one.” The general might not quarrel, but he also did not do as he was told.

  With Johnson, Grant’s personal views supported his habit of nonsubordination. The general disliked Johnson’s policies toward the Southern states. During the Swing Around the Circle eleven months before, compelled to attend rally after painful rally, Grant feared being identified with the president. “I am disgusted with this trip,” he wrote to his brother. “I am disgusted at hearing a man make speeches on the way to his own funeral.”

  That Grant was in the White House in the middle of 1867, sparring with the president on more or less equal terms, marked an astonishing personal turnaround. Eight years before, Grant despaired of being able to feed his family. Now, forty-five years old, he commanded the nation’s military, was adored by the northern two-thirds of the nation, and was the presumptive Republican candidate for president the following year. Facing Johnson that day, Grant could wonder what it would be like to sit on the other side of the desk.

  To Johnson, the bearded man before him had never seemed impressive. Small and quiet, puffing on cigars from breakfast until bedtime, Grant failed to impress people most of his life. As one of his senior generals observed, because Grant was “very reticent and somewhat ill at ease among strangers,…a first impression is never favorable.” Abraham Lincoln described him as “the quietest little fellow you ever saw,” then gave him command of the nation’s armed forces.

  In a military world filled with blustering popinjays and vainglorious peacocks, Grant stood out for his plainness and his ability (as one c
ontemporary put it) to remain silent in several languages. If conversation was absolutely necessary, Grant chatted about horses, his passion. Ben Wade of Ohio, the president pro tem of the Senate, once complained: “As quick as I’d talk politics, he’d talk horses, and he could talk for hours on that without getting tired.”

  Army comrades struggled to describe how such an unassuming character rose to command the largest fighting force that ever marched in the Western Hemisphere. One called him “the most modest, the most disinterested, and the most honest man I ever knew, with a temper that nothing could disturb,” adding with apparent puzzlement that Grant was “[n]ot a great man, except morally; not an original or brilliant man, but sincere, thoughtful, deep, and gifted with courage that never faltered.” One of his officers during the war thought he looked “as if he had determined to drive his head through a brick wall, and was about to do it.”

  General-in-Chief Ulysses S. Grant.

  As a West Point cadet, Grant got an early start on not impressing his peers, graduating in the bottom half of his class. “He couldn’t, or wouldn’t, dance,” one classmate remembered. “He had no facility in conversation with the ladies, a total absence of elegance.” Grant did not much care for his courses at the military academy, preferring to read novels. Evidence of Grant’s unpretentious manner begins with the transformation of his name. Before West Point, he was Hiram Ulysses Grant. The congressman who nominated him to the academy mistakenly submitted his name as “Ulysses S. Grant.” When another cadet saw a list featuring “U. S. Grant,” he said the new student must be “Uncle Sam Grant.” Thus, Grant lost his real first name (Hiram) and gained both a middle initial (S.) and his army nickname (Sam). Grant accepted all three changes without objection.