By this time, 1830, the Madisons were at their low point in their relationship with their son. Dolley, ever the enabler, was upset that he could not mature as she wanted him to and was forever spending money, money she did not have. In 1809, Dolley had sold the Madisons’ properties in Philadelphia (at much lower prices than she expected) to pay Payne's debts that, by that time, were running about $3,000 per year (about $51,000 today). His bills were even higher in the 1830s.29
Their son was also a victim of the debtors’ prison phenomena of the late 1830s. The recession caused by the Panic of 1837 had caused such financial strife that debtors were thrown into prison in order to get their friends or families to pay off what they owed in order to free them. Payne wound up in debtors’ jails several times.
The president, who had put up with Payne all of his life, wrote Coles a melancholy note that reflected his feelings about his wayward child. “His career must soon be fatal to everything dear to him in life…with all the concealments and alleviations I have been able to effect, his mother has known enough to make her wretched the whole time of his strange absence and mysterious silence.”30
By the mid-1830s, Payne was settled in his spendthrift and irresponsible ways, and the Madisons knew it. The president continually wrung his hands over his son, but Dolley was resolved to keep sending him money. Now, she rationed, the funds were not to help him live his profligate life but to keep him out of prison in whatever city he wandered into. His mother had simply given up on him. Finally.
She wrote him in 1835, “I was glad to hear [my letter] with the enclosure [money] reached you. You did not tell me whether you had been successful in your collections. If not, you will want supplies proportioned to your detention; I am anxious that you should have them, and you know the little I have in my power is at your command, though but a ‘drop in the bucket.’”31
And she always warned him to keep out of whatever towns where he had found trouble. “You will finish your business ’ere you leave a place which you may not find it convenient to return to directly,” she said once of his stay in Washington, where he always borrowed money that he could not pay back.32
The more the Madisons helped their son, the more they hurt themselves. The farm business was failing in the 1820s and 1830s. When it did improve, the shortage of laborers at Montpelier, due to sales of slaves to raise money to pay Payne's debts, left a small workforce that could not accomplish much, anyway.
Everybody sympathized with the Madisons and their struggles with their son. Edward Coles, who knew the Madisons and Payne, probably summed it up best in describing Payne's life at Montpelier: “[He is] the serpent in the Garden of Eden.”33
But the worst was yet to come as Payne's irresponsibility grew and he turned his claws on his parents.
When James Madison was a little boy, he played on his Virginia farms with young slave children whom his father owned. He went to the College of New Jersey for higher education and brought along a slave, Sawney, to act as his valet. When he traveled to the Continental Congress during the American Revolution, he was accompanied by a slave. When he married Dolley, their house was full of Madison's slaves. More slaves worked for him on his plantation and in Washington when he was secretary of state and when he was president. He retired to Montpelier, which was worked by nearly one hundred slaves, and spent much of his time each day working with and talking to his slaves. All of his meals were cooked and served by his slaves. When he looked out the window of his bedroom on the second story of the Montpelier mansion in the morning, he stared at his slaves’ dozens of wooden cabins one hundred yards down the slope in front of the home or thirty yards east of it; when he went to sleep at night, he peered out at the slave cabins again. Madison lived in a world of unending slavery.
Madison's life was like that of many slave-owning planters in Virginia. Slaves worked large farms that produced profits for their owners. Some planters, such as George Washington and Robert Carter, had more than three hundred. Most slave owners had only a handful, often just two or three. And these slave owners did not even represent the majority of Virginians; three quarters of Virginians did not own any slaves at all. The large plantation owners, wealthy men like Madison, controlled the state and county governments and helped to run the Federalist and Republican Parties. They used their influence, political clout, and extensive network of friends to perpetuate slavery. It had thrived in Virginia since shortly after the establishment of the disease-ridden Jamestown plantation in the early 1600s. By 1817, when Madison retired, slaves made up nearly half of the population of Virginia and, thanks to slavery, the plantation-ridden state was one of the most prosperous in America.
Madison hated slavery. He had spoken out and written against it all of his life. He was blunt about it, too. When the Constitution was being argued in 1787, southerners insisted that their slaves be counted as three-fifths of a white man to help them in population counts that would earn them congressional seats and give them power in the new government. Madison said the issue was strictly over “having and not having slaves.” He had argued at the convention, too, that pushing back the stoppage of the slave trade from Africa from the proposed year of 1800 to 1808 would not make any difference in what he said was an evil practice. He also sneered that the clever insertion of the word migration into official government documents to describe the movement of people from Europe to America also included the movement of slaves from Africa to the United States and from one slave state to another, which he opposed.1
In 1789, as the new government was taking office, Madison wrote a long memo calling for the creation of a colony in West Africa where freed blacks and freed slaves could be sent from the United States to end slavery here and return tens of thousands of blacks back to their homeland.
He also urged establishing some western colony in America, in the new and unsettled frontier on the Great Plains, where transported slaves could establish new lives for themselves as free men and free women. Madison told everybody he knew that he was opposed to slavery. Visitors to Montpelier recalled his recoiling from the practice in conversations and horseback-riding trips around the plantation. One woman wrote that Dolley always blurted out in conversations that no slaves were whipped at Montpelier and that everything that could be done to make their lives more tolerable was done by the Madisons. The couple had to talk about it continually, though, because as the years went by, the issue became more and more controversial in Virginia and in the nation. The Madisons rarely brought up the subject of slavery; slaves appear in very few of their letters and public speeches, and when they are mentioned they are usually referred to as “the hands” and not “the slaves.”
Back in the mid-1780s, just after the end of the Revolutionary War, Madison wrote Edmund Randolph that he wanted “to depend as little as possible on the labor of slaves.” In 1785, he spoke up in the state assembly to support a bill introduced by Thomas Jefferson to bring about the gradual emancipation of slaves in Virginia (it was defeated). He and Jefferson were applauded for their efforts by the heads of antislavery organizations in both the North and the South.2
And yet later, when he had power, whenever he had the chance as a congressman or as president to actually introduce or support legislation to end slavery, Madison always backed off. During Madison's first term in the Continental Congress, Benjamin Franklin tried to get him to champion a bill to end slavery and he refused. Madison said the reason was that it would annoy slaveholding congressmen. “The number of vessels employed in the trade to Africa is much greater than I should have conjectured. It hope it will daily diminish and soon cease altogether…should the evil still go on, it continues to be my opinion that the interposition of the general government ought to be applied as far as may be constitutional…. At present I not only flatter myself that the necessity may not exist but apprehend that a revival of the subject in Congress would be equally unseasonable and unsuccessful.”3
No matter how loudly or how often the Madisons wailed against sl
avery, though, they would not divest themselves of their slaves. Neither did Jefferson or the others. The South's prosperous planters complained about slavery, excoriated it, pitied their slaves, and vowed to get rid of them, but when they all died, they still had their slaves. Patrick Henry, another slaveholder in that era and a longtime political foe of Madison, explained their feelings best when he said of slaves that “I am drawn along by the general inconvenience of living without them; I will not, I cannot, justify it.” Then Henry added that he had to keep them. “If we cannot reduce this wished for reformation to practice, let us treat the unhappy victims with lenity, it is the furthest advance we can make towards justice. It is a debt we owe to the purity of our religion to show that it is at variance with that law which warrants slavery.” Many of them believed future calamities would occur because of slavery. “Indeed I tremble for my country when I reflect that God is just; that his justice cannot sleep forever,” said Jefferson.4
The Madisons and their friends were not only lenient toward their slaves, but, gingerly, brought them into their activities, always there against the walls at Christmas, out in the yard in summer, with them, smiling, at barbecues, taking care of the kids and enjoying the family travels. They saw the slaves as friends without any official acknowledgment and believed that the slaves saw them that same way. Older slaves were lovingly referred to as “uncles.” The Madisons would refer to their help as part of the extended family, such as Dolley's note to her niece in which she wrote lovingly that “even the old negroes and young ones want to see you.”5
Dolley's cousin, Sarah “Sally” Coles Stevenson, joked about bumpy roads and slaves that “unless indeed Uncle Sam was on the box, a very respectable old gentleman everywhere, but with the reins in his hands I shall never forget the fright he gave me.”6
Madison's best friend, Jefferson, was one of the most strident voices against slavery in the country. He had even tried to write a clause into the Declaration of Independence that denounced it. He told people all of his life how much he hated slavery. Yet Jefferson did not free his slaves, despite severe criticism of his behavior by many, north and south. In his lifetime, Jefferson owned approximately six hundred slaves; he freed a total of just seven. He even allegedly carried on a long-term love affair with one, Sally Hemings, and had six children with her. His own solution was taking good care of his slaves. He believed that in providing them with shelter, clothing, and food, and extensive freedom within the plantation, he was behaving in a Christian way. He told others that slave owners had a moral obligation to care for their slaves, their “people.” To free them, they said, would be to turn them loose in a country where they would be unable to function. Slave owners would care for them; others would not. Slaves were better off in bondage, Jefferson contended, provided for by loving owners such as himself. Nobody followed that mantra for a longer time, and more fervently, than James Madison.
Thomas Jefferson was challenged, and harshly, by antislavery advocates. One of the fiercest was Edward Coles, James Madison's secretary, who also badgered Madison on the issue of slavery. Coles wrote Jefferson in 1814 that he should lead the movement to free the slaves “from your known philosophical and enlarged view of subjects and from the principles you have professed and practiced through a long and useful life.” He added that “I hope the fear of failing, at this time, will have no influence in preventing you from employing your pen to eradicate this degrading feature of British Colonial policy, which is still permitted to exist, notwithstanding its repugnance” as well to “the principles to our evolution as to our free institutions.” Coles added that Jefferson's leadership of the antislavery cause, even more so than his presidency, would make him a great man.7
In his reply, Jefferson shrugged off his criticism and told him that most Virginians were not against slavery. He told all that he was too old to lead an antislavery movement anyway and that leaders from the next generation had to do that. Madison averted Coles's attack, too, telling him again and again that he agreed with him but could do little to end slavery.
Like all southern slave owners, Madison believed that without slaves his farms would go bankrupt. None of the slaveholders in Virginia embraced the idea that paid workers toiled harder for salary and bonuses and, in the end, made farms more profitable than slave-run farms. Farms in the northern states that did not have slaves in 1817 succeeded on that basis, and their owners suggested the concept to southerners for two hundred years. They were sneered at.
By the time Madison retired, there were numerous antislavery societies in Virginia, begun by the Abolition Society in Richmond, founded by Robert Pleasants in 1790, and the Society of Friends, the Quakers, which had a Virginia Meeting organization thriving in the state at the same time. They even hired agents from Virginia to travel through other southern states looking for runaway slaves to rescue. In 1785, the General Committee of the Virginia Baptists criticized slavery as “contrary to the word of God.” In 1790 they condemned it as “a violent deprivation of the rights of man and inconsistent with a republican government” and urged Baptists to do everything legally possible to eliminate it.8
Madison, Jefferson, and others saw the emancipation of slaves by sympathetic and well-intentioned owners as misguided idealism that had to be cut off before it could spread throughout Virginia and the South. As long as owners took care of their slaves, they assured everyone, all would be well.
Leading planters in the state became officials of some of the antislavery societies. The organizations gained many followers but could never get legislation outlawing slavery passed. Leading political figures always talked them out of it. George Washington tried to get a bill introduced in the House of Burgesses to eliminate slavery in the late 1760s and was persuaded by other legislators to withdraw it. In 1774, he and George Mason published the Fairfax Resolves in numerous newspapers that called for the elimination of slavery.9
James Madison was deeply conflicted by slavery. He opposed it everywhere he could and in any public gathering that would listen to him, but, in fact, he was one of the slaveholding public officials who, although championing the prohibition of slavery, worked hard to make sure no laws were passed to eliminate it. In 1791, Pleasants wanted to introduce a bill to end slavery in the Virginia legislature. Madison dissuaded him, rather bluntly, telling him that “those from whom I derive my public station are known by me to be greatly interested in that special of property, and to view the latter in that light.”10
Madison's views of slavery had been further upended by stern criticism of the institution by George Washington. During the American Revolution, Washington warned Americans that they should stop arguing that they needed to be free because they had become slaves to England. He fumed that none could make that charge because many Americans owned slaves themselves. New Jersey minister Samuel Allinson wrote that it would be the “lasting disgrace” of Congress to argue that America was enslaved and not to free the real slaves here “if they should spend so much time to secure their own liberties and leave no vestige of their regard to those of their fellow men in bondage to themselves.” Foreigners agreed. A British philosopher, Thomas Day, wrote in 1776 that “if there be an object truly ridiculous in nature, it is an American patriot, signing resolutions of independence with the one hand and with the other brandishing a whip over his affrighted slaves.” Writers were in agreement with Day; Thomas Paine, whose inspirational pamphlets helped to fuel the early days of revolution, added, “If they could carry off and enslave some thousands of us, would we think it just?”11
Madison knew what the arguments in favor of slavery were, and they were repeated to him all of his life by friends and relatives. They were succinctly stated by a man who adopted the pseudonym Eliobo in a continuing series of letters depicting the pros and cons of slavery in the New Jersey Journal newspaper in the winter of 1780–1781. Madison, in Congress in Philadelphia at that time, read every newspaper he could find and must have perused this one.
Eliobo's arguments fo
r slavery were (1) owners provide for slaves’ basic needs in life, such as food and shelter; (2) blacks were lazy and could not function in society; (3) slaves were happy; (4) freed blacks will rape white women; (5) freed slaves will obtain firearms and get what they want by force; and (6) freed slaves would join forces with American Indians and attack white villages. In some form, all slaveholders agreed with the arguments of the letter. Those against the arguments, such as Madison, knew that many of their neighbors embraced them.
Charles Cotesworth Pinckney of South Carolina had told friends that many opposed the Bill of Rights to Madison's Constitution because such legislation usually said that “all men are by nature born free” yet “a large part of our property consists in men who are actually born slaves.”12
Many of Madison's friends were members of antislavery societies. Tench Coxe, one of his friends from the revolution, belonged to the Pennsylvania Abolition Society and kept him abreast of their work. He even told Madison at one point that the group had been trying to get Benjamin Franklin to introduce an antislavery measure at the Constitutional Convention, but that did not happen.
Many Virginians considered slavery a curse on their state. When Kentucky separated from Virginia and its officials met to discuss their new constitution, David Rice, who had moved there from Virginia, told them that “holding men in slavery is the national vice of Virginia.”13
When Madison was president, several leaders of large abolitionist organizations met with him in an effort to engage his official support of their antislavery crusade. In the winter of 1813, as he was consumed in work over the war with England, he met with a representative of the Pennsylvania Abolitionist Society. Afterward, they released a statement that blasted slave owners. “That notwithstanding the laws which Congress have enacted inhibiting the slave trade, as well as for the punishment of citizens who may be concerned in the infraction of them, the inhuman, and unjust commerce in African subjects, continues in defiance of those laws and in violation of every honorable and benevolent principal to be pursued by some American merchants.” The document went on to state that its members had proof that seventy thousand men and women were kidnapped from Africa in 1810 and illegally taken to the United States as slaves. They were transported and delivered, the abolitionists charged, because foreign governments and the US government did nothing to stop the slave trade. Officials just looked the other way. They begged Madison to step in and curb it. He did not.14
James and Dolley Madison Page 43
