Impeached: The Trial of President Andrew Johnson and the Fight for Lincoln's Legacy

by David O. Stewart

  In the South, the human losses—260,000 soldiers dead—were a greater proportion of the population, while the physical wreckage was far worse. Nine million Southern whites and four million freed slaves faced fearsome challenges. Travelers were staggered by the destruction and poverty.

  In Virginia’s Shenandoah Valley, a visitor saw neither fences nor crops. Northern Virginia showed few signs of human industry, only an occasional wan, half-cultivated field. In Richmond and in Columbia, South Carolina, chimneys stood forlornly, surrounded by ash, blackened walls, and piles of bricks. Some shops had goods, but no one had money to buy them. Atlanta had burned also, leaving both races homeless. Blacks and whites lived in makeshift shelters on the edge of town, in clusters that looked like “fantastic encampment[s] of gypsies or Indians.” Young men were scarce in Charleston, birthplace of secession, which had become

  A city of ruins, of desolation, of vacant houses, of widowed women, of rotting wharves, of deserted warehouses, of weed-wild gardens, of miles of grass-grown streets, of acres of pitiful and voiceful barrenness.

  The Southern economy stood still. For slave owners, emancipation represented a massive loss of capital. According to a calculation prepared for Johnson, the South lost roughly 30 percent of its wealth when the slaves walked away from its plantations. As economic activity dwindled, whites and blacks faced idleness. In one town, “People on Main Street sat out on the sidewalks gossiping and smoking, some with tables playing chess, backgammon and cards. As the sun moved they moved from one side of the street to the other to get the shade.”

  For all the travail of daily life, a single sea change became the focus of political and social energy in the South. Four million blacks—almost one-third of the people in the region—were now free. Whites and blacks would have to work out what that freedom was, a process that was bound to be painful. By far the greater share of pain would be felt by the impoverished, unschooled, and weaponless freedmen. Most owned only the clothes on their backs. Somehow they had to provide for themselves in a hostile land. But for whites, too, emancipation brought upheaval.

  Most Southern whites rejected any suggestion that the freed people could be equal to them. A visiting writer concluded that Southerners viewed the former slave as “an animal; a higher sort of animal, to be sure, than the dog or the horse, but, after all, an animal.” Political rights for the former slaves, such as voting, seemed “the most revolting of all possibilities…[a] degradation upon a gallant people.”

  The changes brought by emancipation reached into every human interaction. How should a free black person address a white person? How could he or she walk on the street? Should a black step aside for whites? What if a white person acted improperly? Could a freedman object? The freed slaves wanted to act as equals. Whites were shocked. As one visitor observed, they “perceive[d] insolence in a tone, a glance, a gesture, a failure to yield enough by two or three inches in meeting on the sidewalk.”

  First came the simple act of traveling through the countryside. Blacks, in theory at least, now could come and go as they pleased, could walk the roads at any time of the day or night. In the early postwar months, and to the dismay of Southern whites, many freed slaves did just that. A Northern traveler reported that “the highroads and by-ways were alive with footloose colored people.” Some searched for family members who had been sold or carried into other states. Some fled angry masters or sought better work. Some longed to see what was on the other side of the hill. A black waiter explained the phenomenon: “You know how a bird that has been long in a cage will act when the door is opened; he make a curious fluttering for a little while. It was just so with the colored people. They didn’t know at first what to do with themselves. But they got sobered pretty soon.”

  The overthrow of slavery forced a shift to new labor arrangements. Often the results were harsh. Some freedmen were “turned off” farms where they had lived and worked for years. The owners did not care to pay them, or had no money to do so, so they sent them away. Some masters did not tell their slaves of emancipation, keeping them in bondage for months beyond the war’s end. The transition to paying for labor was tortuous. The former slaves had little bargaining power. Most knew only farm work yet had no land of their own. Without experience in making contracts for wages, many entered into yearlong contracts at rock-bottom wages, or for a share of the crop. Mississippi required proof every January that each freedman had a labor contract for the coming year. Some worked an entire year only then to be driven off the land, unpaid. Blacks who violated their contracts faced imprisonment and involuntary, unpaid labor. As one Northern correspondent summarized it, the “black codes” adopted by Johnson’s state governments allowed the states to “call them freedmen, but indirectly make them slaves again.”

  Conflicts flared when the former masters used physical force against their workers, as they were accustomed to do. Most whites thought it impossible that blacks would work without coercion. A Virginian acknowledged that “[a] good many of the masters forget pretty often that their niggers are free, and take a stick to them, or give them a cuff with the fist.”

  Southern whites responded to emancipation in complicated, sometimes contradictory ways. Many, believing that the freed people could never provide for themselves, predicted that they would simply die out. One South Carolinian insisted that “being left to stand or fall alone in a competitive struggle for life with a superior race, [the freedmen] would be sure to perish.” Abolitionists, he gloated, “would find they had exterminated the species.” But the freed slaves did not disappear. Indeed, many government records show that more whites than blacks took advantage of postwar food handouts.

  Some whites embraced the fanciful notion of sending the blacks someplace else—Africa, Texas, Peru, the Sandwich Islands, any place but where they were. Then, the whites argued, Southern fields could be worked by industrious German immigrants. Problems quickly emerged with this solution: there was neither transportation for four million freed slaves, nor a place to send them, nor any significant interest among the freedmen in leaving.

  The whites’ refrain turned to the need to “keep the nigger in his place.” A North Carolinian explained in the fall of 1865 that he would vote for political candidates who promised to do just that: “If we let a nigger git equal with us, the next thing we know he’ll be ahead of us.”

  Racial violence spread. White assaults on blacks erupted during the commonplaces of daily life. If freed slaves did not touch their hats to their former masters, they were saucy, unendurable. Whites railed when Negroes resisted beatings, seeing resistance as insubordination. One Southerner swore that “nothing would make me cut a nigger’s throat from ear to ear so quick as having him set up his impudent face to tell that a thing wasn’t so when I said it was so.” A New York correspondent wrote that “half-a-dozen times, in the course of a single day, I observed quarrels going on between negroes and white men.”

  The violence overwhelmed the peacemaking efforts of officials of the Freedmen’s Bureau, established by Congress in March 1865 to aid both whites displaced by the war and former slaves making the transition to freedom. Thinly sprinkled through the region, Bureau agents were supposed to secure fair treatment for the freedmen and prevent the reintroduction of slavery under another name. Many Bureau agents were startled by racial attacks they could not restrain. In his first ten days in Greensboro, North Carolina, a Bureau agent received two cases of whites shooting blacks. “The fact is,” he explained, “it’s the first notion with a great many of these people, if a Negro says anything or does anything that they don’t like, to take a gun and put a bullet into him.” In the interior of South Carolina, a monthly Bureau report recorded the whipping of a freedman and his children while tied to stakes, the whipping of another who also received a knife wound to the face, and the beating and unexplained disappearance of a freedwoman. A group of North Carolina whites went on a “spree,” whipping pro-Union whites, castrating and murdering a black man, then shooting other
blacks, including two young boys. In Columbia, South Carolina, a correspondent reported the shooting of a Negro “as if he had been only a dog,—shot at from the door of a store, and at midday!” When a black mistakenly cut down a tree on a white man’s farm in Virginia, the white man “deliberately shot him as he would shoot a bird.”

  In the summer of 1865, President Johnson sent Carl Schurz, an ambitious German-American politician, to examine conditions in the South. Schurz’s report was chilling:

  I saw in various hospitals negroes, women as well as men, whose ears had been cut off or whose bodies were slashed with knives or bruised with whips, or bludgeons, or punctured with shot wounds. Dead negroes were found in considerable number in the country roads or on the fields, shot to death, or strung on the limbs of trees. In many districts the colored people were in a panic of fright, and the whites in an almost insane state of irritation against them.

  Another presidential agent placed the blame on the Southern state governments nurtured by Johnson. In South Carolina, he reported, Johnson’s appointed governor had “put on their legs a set of men who…like the Bourbons have learned nothing and forgotten nothing.”

  By 1866, the violence would become organized. Southern whites formed terrorist groups with chivalric names—the Knights of the Golden Circle, the Knights of the White Camellia, the Teutonic Knights, the Sons of Washington, the Knights of the Rising Sun, and the Ku Klux Klan. As many as a hundred masked and hooded riders could descend on a homestead or town, killing and maiming at will.

  The Southern violence turned Northern observers into cynics. “[T]o knock [a Negro] down with a club,” wrote a New Englander, “or tie one of them up and horsewhip him, seems to be regarded as only a pleasant pastime.” A stableman in South Carolina, he added, “cut off a negro boy’s ear last week with one blow of a whip, and tells of the act as though it were a good joke.”

  The assailants rarely faced punishment from state or local authorities. In South Carolina, a white killed a black man who was stealing corn. He was tried and exonerated. An ex-Confederate official told of a white man in Texas who whipped his former slave woman for insolence; when the woman’s husband protested, the former master shot him. The wounded husband was jailed for assault; the former master went free. A Freedmen’s Bureau report for South Carolina explained that “it is difficult to reach the murderers of colored people, as they hide themselves and are screened by their neighbors.” A Bureau agent in North Carolina told Congress that he knew of many murders of blacks by whites; in none were charges brought. A Union general posted to Mississippi in 1865 and 1866 testified, “Murder was quite a frequent affair against freedmen everywhere in that community, and the commission of crimes of a lesser grade was still more frequent.” He knew of no occasion when a white was punished for an offense against a freedman.

  At the end of 1865, General Grant asked army commanders in the South to report on the violence. Commanders in only five states responded (the Carolinas, Mississippi, Georgia, and Tennessee). They described more than 200 assaults on blacks and 44 murders. Much of the violence, however, went unreported. No estimate of the murders and violence in the South after the war is entirely reliable, but the numbers were consistently dreadful.

  The carnage was worst in Texas, with its huge distances and wild frontiers. By 1868, an army commander reported that “the murder of negroes is so common as to render it impossible to keep accurate count of them.” The records of the Freedmen’s Bureau yield a dispiriting catalog of reasons for the murders of black Texans in the postwar years:

  [The] freedman did not remove his hat when he passed [a white man], negro would not allow himself to be whipped; freedman would not allow his wife to be whipped by a white man; he [the victim] was carrying a letter to a Freedmen’s Bureau official; kill negroes to see them kick; wanted to thin out niggers a little; didn’t hand over his money quick enough; wouldn’t give up his whiskey flask.

  In 1865 and 1866, more than 500 whites were indicted in Texas for murdering blacks; none was convicted. During the 1868 election campaign, an estimated 2,000 Negroes were murdered in Texas.

  A bureau agent in Mississippi captured the lawlessness of that time and place:

  Men, who are honorable in their dealings with their white neighbors, will cheat a negro without feeling a single twinge of their honor; to kill a negro they do not deem murder; to debauch a negro woman they do not think fornication; to take property away from a negro they do not deem robbery….

  While the president worked to return Southern state governments to the former rebels, the army drew the impossible assignment of keeping a lid on the region’s cauldron of race hatred. At the end of the war, no institution enjoyed as much stature in the North as the army. After Lincoln’s death, no individual could rival the prestige of General-in-Chief Ulysses Grant. Still, both the army and Grant were ill suited to the task of governing more than one-third of the country. The army shrank quickly after the Confederate surrender in April 1865, from 1 million men to fewer than 50,000. Thousands of federal troops remained in the South to keep the peace—enough soldiers to enrage white Southerners, yet not enough to protect the freedmen. Even more galling to Southern whites, many of those soldiers were blacks, former slaves. Without homes or employment to return to, colored troops were willing to remain in uniform. In September 1865, the army commander in Mississippi had thirteen infantry regiments; twelve were black.

  Southern whites blanched at “the degradation of being guarded by these runaway slaves of theirs. To be conquered by the Yankees was humiliating, but to have their own negroes armed and set over them they felt to be cruel and wanton insult.” Forty years later, a Southerner remembered with horror the experience of having “an alien race, an ignorant race, half-human, half-savage, above them.” Freedom for the slaves, she concluded, “was inversion, revolution.”

  The man in the White House shared those feelings. In September 1865, President Johnson sent a telegram to the army commander in East Tennessee, his home region. The president insisted that Negro troops be withdrawn from the area, which the Negro soldiers had “converted into a sink of pollution.” His own house in Greeneville, Johnson protested, was being used as a “rendezvous for male and female negroes who have been congregated there, in fact making it a common negro brothel.” The commander replied that he had no alternative to Negro troops. He added, without elaboration, that the president’s son-in-law had lodged a white family in the president’s home.

  The duties of occupation were far different from those of combat. As one general wrote, “you have not only to be a soldier, but must play the politician,” a part which many soldiers found “not only difficult but disagreeable.” General William Tecumseh Sherman complained that the Army was “left in the breach to catch all the kicks and cuffs of a war of races, without the privilege of advising or being consulted beforehand.” A third general, detailed to the Freedmen’s Bureau, stressed how difficult it was to keep peace with white Southerners who say “they are overpowered, not conquered,” and who “regard their treason as a virtue, and loyalty as dishonorable.”

  The assignment was a minefield, the conflicts unceasing. Black soldiers felt resentment toward the region that had practiced slavery so long, while many Southerners nursed hatreds from the war. As Chief Justice Salmon P. Chase wrote during a Southern tour in May 1865, “As yet the rebels are disarmed only, not reconciled, hardly acquiescent.” Southern whites brought lawsuits against army officers for false arrest, damage to property, murder, and assault. Local officials arrested federal soldiers for small infractions of local ordinances. Far more alarming were the outright murders of soldiers. In South Carolina in 1865, three soldiers from Maine were “shot from behind,…killed because they were Yankees.” Five more were killed in the western part of the state. Two Freedmen’s Bureau agents were murdered in Mississippi, another in Texas. Texas grand juries refused to indict those accused of shooting down Union soldiers. State militias and “home guards” posed an armed th
reat to federal troops and complicated many situations.

  The Army’s task would grow more difficult after political warfare broke out between Congress and the president over how reconstruction should proceed. From month to month, the Army’s mandate, and precise instructions, would change dramatically as Congress enacted a new statute, or Johnson’s attorney general issued a new interpretation of the last one, or the president replaced a regional military commander to soften the occupation, which he did repeatedly.

  In the coming conflict between Johnson and the Congress, control of the Army would become the great prize. Countervailing pressures drew its senior officers—particularly Grant and his second, William T. Sherman—into highly political situations. During a critical point in the contest for control of the Army, Sherman wrote to Grant, “We ought not to be involved in politics, but for the sake of the army we are justified in trying at least to cut this Gordian knot, which they do not appear to have any practicable plan to do.”

  There would never be enough troops and Bureau agents to pacify the angry South. At its peak, the Freedmen’s Bureau had only 900 agents, some of whom sympathized with Southern whites, not with the freedmen. To patrol the Rio Grande border with Mexico and to occupy all of Texas, the Army had only 5,000 men.

  The situation between 1865 and 1867 satisfied no one. The president had returned the state governments to the control of former Confederates, which infuriated Stevens and many Northerners. Southern black codes stoked their rage higher, as did the president’s amnesty program, while the freed slaves remained at the mercy of their former masters, enjoying only fitful protection from the Army and the Freedmen’s Bureau. In November of 1865, as the new Southern senators and congressmen began to arrive in Washington City, the fear spread among Republicans that Johnson was allowing the South to control the national government it had spurned only four years before. Effective reconstruction required a political strategy, not a military one, and that would require agreement between Congress and the president. Instead, the two branches of the government were spoiling for a showdown.