The Intimate Lives of the Founding Fathers

by Thomas Fleming

  In this first venture into public life, Madison said next to nothing throughout the proceedings as Virginia adopted a constitution and a bill of rights. As one of the youngest delegates, his silence was understandable, but it foreshadowed a style that flowed from his diminutive size and unprepossessing appearance. Madison was the polar opposite of the flamboyant leader of Virginia’s revolution, Patrick Henry, whose defiant shout, “Give me Liberty or Give me Death!” had become one of the mottos of the Revolution.

  Henry was a man’s man in every sense of the word, ready to flay an opponent with words or a whip and quick to defend his honor with a gentleman’s ultimate recourse, a pistol. Henry must have filled James Madison with a rueful envy—and not a little despair. Behind his rhetorical facade, the great man was an ignoramus. Madison’s brainpower exceeded Henry’s by at least ten to one. But the younger man’s thin, reedy voice, his parchment-like skin, and his diffident manner virtually guaranteed that no one would ever listen to him.

  In the fall of 1776, Madison participated in a minor way in a debate over religious freedom in the Virginia legislature. His Princeton education made him strongly sympathetic to the Baptists and Presbyterians in Virginia, who were often persecuted and sometimes jailed by zealots supporting the established Anglican Church. The leader of the assault on the established church was Thomas Jefferson, but not even his prestige as the author of the Declaration of Independence did him much good against the angry majority who sided with the religious status quo. Madison was more a spectator than a participant in the debates, and he later recalled that Jefferson paid little or no attention to him, because of “the disparities between us.”

  That phrase reveals a great deal about Madison’s self-image at this time. The tall, lanky Jefferson, with his genial, outgoing manner and gift for the smashing phrase, was another icon that James Madison could never become. The comparison was as painful to Madison as the contrast to Henry. The younger man admired Jefferson’s wide-ranging knowledge of the law, philosophy, and literature. Here was someone Madison would do almost anything to have as a friend. But there seemed to be no hope of such a relationship ever developing.

  V

  Back home in Orange County, Madison received a rude shock. He ran for reelection to the Virginia Convention and lost. Still a college idealist, he had disdained to offer the voters what they had become accustomed to getting from political candidates in Virginia—unlimited access to a liquor barrel. He had decided booze was “inconsistent with the purity of moral and republican principles.” The voters thought he was a cheapskate or had gotten too big for his breeches.

  Once more Madison retreated to his father’s library and continued to read deeply on the art and science of government. He might have stayed there for decades were it not for the good offices of his father’s friends in high places. In 1778, the legislature elected Madison to Governor Patrick Henry’s council, an eight-man body that was supposed to advise the chief executive on matters of politics and policy.

  Madison moved to Williamsburg once more and took a room with his second cousin, yet another James Madison, who was president of William & Mary College. He was soon embroiled full time in the problems of taxation and finance, army recruitment, Indian affairs, and the myriad other matters that fighting a war and running a government dumped on Governor Henry’s desk. The Virginia constitution, fearful of creating a tyrant, gave the governor and his councilors virtually equal power, creating what Madison later sarcastically called “eight governors and a councilor.” Ninety percent of the power remained with the legislature, prompting Madison to describe the job as “a grave of useful talents.”

  In 1779, Thomas Jefferson was elected governor, and Madison’s attitude toward the councilor’s job changed dramatically. Virginia faced a looming challenge as the British shifted the focus of the war south, and everyone realized that Henry had spent his three years as governor talking big and doing nothing to prepare the state for serious warfare. There were woeful shortages of everything from guns to supplies. We have already watched a dismayed Governor Jefferson discover the weakness of the state’s militia law, which all but made cowardice a virtue.

  In this crisis atmosphere, Jefferson found himself listening far more often to the twenty-eight-year-old Madison than to any of his other councilors. The soft-voiced little man had an uncanny ability to cut through details and fasten on the heart of a problem—and suggest a realistic solution. The beginning of a friendship that would powerfully influence the history of the United States took shape during these hectic days in Richmond. It grew in depth and intensity during Madison’s years in Congress. Without Jefferson’s sympathy and support, Madison’s humiliating debacle with Kitty Floyd might have sent him back to Virginia a recluse for the rest of his life.

  VI

  Somewhere deep in his unconscious or perhaps in his conscious mind, James Madison seems to have resolved not to seek romance again until he was a man of importance. For the next decade, he devoted himself exclusively to the future of the United States of America. He took Jefferson’s advice about unremitting occupations, altering it in only a single respect—he narrowed his occupation to one: his role as a public man.

  Americans from 1784 down to our prosperous and powerful present generation have been the beneficiaries of Kitty Floyd’s decision to jilt James Madison. One can only speculate what the ex-congressman might have done if he had carried her off to Virginia and began enjoying the pleasures of married life. Instead, he focused his powerful intellect on solving the fundamental problem confronting the nation—how to persuade the thirteen quarrelsome, semi-independent states to cede sufficient power to a central government to preserve the federal union.

  When Thomas Jefferson went off to France to replace Benjamin Franklin as ambassador, Madison turned to the ultimate American hero, George Washington. Instinctively, Madison seemed to reach out to more commanding figures to help him achieve his goals. He had already won Washington’s respect and attention by constantly supporting the army and other national interests in Congress. At the end of the war, Washington had won Madison’s respect by peaceably resigning his commission as commander in chief of the army, even though Congress had sent his soldiers home unpaid and embittered.

  The two men began a momentous correspondence, in which the ex-general made his vision of America’s future plain: an “indissoluble” union of states under a single head was vital to the nation’s survival. How to achieve this goal was beyond Washington’s capacity—but not Madison’s. While he began politicking for a convention to overhaul the ramshackle Articles of Confederation, he went to work on the masterly rearrangement of powers that became the United States Constitution.

  Even here, when the document that was his brainchild was presented to the historic convention in Philadelphia in 1787, Madison remained in the background. Edmund Randolph, the splendidly handsome, fulsomely oratorical governor of Virginia, introduced the “Virginia Plan” to the convention. But Madison played a leading role in the debates that followed. Delegate William Pierce of Georgia said that thanks to his “spirit of industry and application…he always comes forward as the best informed man of any point in debate.” He was a unique combination of “the profound politician and the scholar.”7

  Historically speaking, the convention was Madison’s finest hour, his rendezvous with fame and greatness. But from a personal point of view, the most satisfying moment came the following year, when he took on Patrick Henry at the Virginia ratifying convention and persuaded the delegates to accept the constitution by a whisker-thin majority. Speaking in his low soft voice, with reams of notes concealed in his hat, Madison bested Henry’s anti-federal fireworks with calm, unwavering logic. It did not hurt to have an invisible backer, George Washington, whose enormous prestige reinforced this reasoned persuasion. But the triumph, as far as the political world of Virginia was concerned, belonged to Madison.

  VII

  Elected a congressman from Virginia—the resentful Henry block
ed his appointment to the Senate—Madison won passage of the Bill of Rights and became the most powerful voice in the new House of Representatives. By this time, at least among his fellow politicians, his fame almost equaled the eminence of his two heroes, Jefferson and Washington. Moving in this aura, James Madison felt ready to resume his search for a wife.

  While attending Congress in New York, the nation’s first capital, he met an attractive widow, Henrietta Maria Colden, who had married into the family of Cadwallader Colden, a prominent prewar New York politician. Although he had remained neutral, many members of his family chose the king’s side. Mrs. Colden’s husband had become a British officer and died not long after they retreated to London with the rest of the British army. Mrs. Colden had returned to New York with two sons to try to regain some of the family’s property, which had been either confiscated or neglected during the war.

  Henrietta Colden was one of the few women who had a membership in her own name in the New York Society Library—one of the first semipublic libraries in the nation. The books she took out, according to the records of this venerable institution, were impressive and undoubtedly explain one reason why Madison pursued her. She read the Roman historians, Tacitus, Suetonius, and Julius Caesar, as well as the historian of Rome, Edward Gibbon, the French savant Jean-Jacques Rousseau, and similar weighty authors. According to one admirer, she combined her brainpower with “feminine graces,” which prompted some to refer to her as “the celebrated Mrs. Colden.”8

  Nothing came of Madison’s interest in her. No one knows whether she rebuffed the diminutive congressman or he had second thoughts about marrying into a family that was strongly tinged with Toryism. It is more than a little likely that Mrs. Colden, who was Scottish, shared her late husband’s sympathies. But the progression from sixteen-year-old Kitty Floyd to this elegant cosmopolitan lady was unquestionably a sign of Madison’s new sense of himself as a man who had achieved fame.9

  VIII

  Politics now transferred Madison to Philadelphia, the next national capital. He had numerous friends there from his years as a confederation congressman and he was soon enjoying the lively social life of the City of Brotherly Love. Politically he sided with his friend and fellow Virginian, Thomas Jefferson, and acquired more fame by opposing Hamilton’s Bank of the United States and President Washington’s policy of neutrality in the mounting conflict between England and revolutionary France. But marriage remained very much on his mind.

  At dinners and receptions, he often met Dolley Payne Todd, wife of a young Quaker lawyer and one of the most lively, attractive women in the capital. He probably bowed to her as often in the street; they lived only three blocks apart. In 1793, her husband and one of their sons died in a yellow fever epidemic. The twenty-five-year-old Dolley became one of the most sought-after widows in the city. According to one somewhat legendary story, a veritable corps of would-be husbands used to station themselves at the head of her street and wait for her to appear.

  One of her suitors was suave Senator Aaron Burr of New York, whose wife was dying of cancer back home. Burr already had a reputation as an irresistible lady-killer. He lived in the nearby boardinghouse run by Dolley’s mother and in 1794 helped Dolley draw up a will leaving all her property to her surviving son, Payne Todd, and making Burr the executor—and Payne’s guardian. Rumors swirled that Burr was going to propose the moment he heard the hourly expected news of his wife’s death.

  Madison decided on a preemptive strike with a neat political twist. He offered to back Burr as the next American ambassador to France, and he persuaded his friend and former ambassador James Monroe to support him. The ambitious Burr was grateful. Paris was a diplomatic post that was virtually guaranteed to get a man’s name in the newspapers. Madison took the proposal to President Washington, who thunderously rejected the idea. For reasons never completely understood, he loathed Aaron Burr.

  Having tried to do this large favor for his chief rival, Madison asked Burr if he would introduce him to Dolley Payne Todd. How could the senator say no? Dolley’s reaction makes plain how high Madison had risen on fame’s ladder. She rushed a note to her best friend, Eliza Collins: “Thou must come to me. Aaron Burr says the great little Madison has asked to be brought to see me this evening.” Dolley was not alone in using this phrase to describe Madison—but for her it seems to have had a romantic ring.10

  Madison came and was enthralled. Buxom, dark-haired, and bubbling with high spirits, Dolley was used to being the center of attention. Like Madison, she was the oldest child of a large family. Philadelphia friends were struck by her beauty and cheerful disposition from the moment she arrived in their city at the age of fifteen. Her Quaker father had freed his slaves and moved north to launch a business career. One man recalled how her “soft blue eyes” and “engaging smile” had raised the mercury in numerous “thermometers of the heart to fever heat.”

  Further cementing Madison’s attraction, Dolley had been born in North Carolina and raised in Virginia, and had numerous relatives there. A fellow Virginian, Congressman Richard Bland Lee, was engaged to Eliza Collins, and Dolley’s younger sister Lucy was about to marry George Steptoe Washington, nephew of the president.

  Madison launched a whirlwind courtship. He enlisted Dolley’s cousin Catherine Coles, the wife of Congressman Isaac Coles of Virginia, to write her a teasing letter, telling her how much she had mesmerized the great little man: “To begin, he thinks so much of you in the day that he has lost his tongue, at night he dreams of you and starts in his sleep calling on you to relieve his flame for he burns to such an excess that he will be shortly consumed and he hopes that your heart will be callous to every other swain but himself.”11

  For good measure, Catherine Coles added, “He has consented to every thing that I have wrote about him with sparkling eyes.” For even better measure, Madison enlisted none other than Martha Washington, who reportedly wrote Dolley a warm letter, urging her to marry her ardent forty-three-year-old suitor.12

  Dolley played the reluctant game for a while. But in the summer of 1794, while visiting her sister Lucy in Virginia, she wrote Madison a letter, saying yes. Madison replied that he had received her “precious favor” and hoped she could “conceive the joy it gave me.” The once jilted suitor anxiously added, “I hope you will never have another deliberation on that subject. If the sentiments of my heart can guarantee those of yours, they assure me they can never be cause for it.”13

  The couple were married on September 15, 1794, at Harewood, the Virginia plantation of George Steptoe Washington. Before the afternoon ceremony began, Dolley found time to write a letter to Eliza Collins, who had just married Congressman Lee. The bride’s remarks about James Madison were notably unromantic. She mentioned her “respect” for him and thought their marriage would give her “everything that is soothing and grateful.” What she meant was clear in the next sentence: “My little Payne will have a generous & tender protector.” She was thinking of her two-year-old son, Payne Todd, who needed a father.

  The word “love” went unmentioned in this intimate semi-confession. She signed the letter “Dolley Payne Todd.” That evening, after the ceremony and the wedding dinner, she added beneath the previous signature: “Dolley Madison! Alas!”14

  PARTNERS IN FAME

  Dolley Payne Todd wrote another letter on her wedding day that suggests it was a good thing James Madison campaigned vigorously for her hand. The letter was to her lawyer, William Wilkins, who had helped her settle her late husband’s complicated estate. He was not pleased by the news of her marriage. He gave his very reluctant approval and confessed that he was “not insensible to your charms.” Another hint of his feelings was the way he called her “Julia,” apparently a private name. This was a device lovers used—as we have seen in the correspondence of John and Abigail Adams.

  Wilkins added words that may further explain Dolley’s “alas.” He warned her that “the eyes of the world” were on her and “your enemies have already opened the
ir mouths.” Wilkins probably meant Dolley’s Todd in-laws, who had already quarreled with her over her late husband’s estate. They were likely to be unhappy about her new husband obtaining control of her modest inheritance and would probably inform their fellow Quakers that she had married an Anglican. That meant Dolley would be “read out” of their congregation. Aside from the romantic feelings she had aroused in Wilkins, Dolley’s marriage to Madison represented a radical break with her Quaker past.

  Dolley may not have married for love. But as the newlyweds visited friends on other plantations in Virginia, she soon realized that for her husband, there was no other reason. Beneath the rational logical persona James Madison presented to his fellow politicians, he was a romantic, a man whose heart spoke to him as often as his head. For him, marriage was a step that could be authorized only by his heart. The intensity of Madison’s feelings swiftly awoke a similar response in Dolley. Whatever ambivalence she felt about their marriage vanished forever.

  Madison’s heart also explained his loyalty to Thomas Jefferson and his hostility to Alexander Hamilton’s attempt to transform America into an industrial mirror image of Great Britain.1 Madison unhesitatingly shared his inner political self with Dolley. A warm letter of congratulations from Jefferson no doubt helped unite politics and personal affection. Jefferson himself was adept in that department. His letter included a plea not to retire from politics. “This must not be,” he wrote. He hoped Mrs. Madison would “keep you where you are for your own satisfaction and the public good.” How could any woman resist such a challenge?2

  II

  During the next three years in Philadelphia, Dolley experienced the excitement of being a political insider. She saw first hand the bruising partisan warfare of the 1790s, and participated in it as James Madison’s wife. She observed the toll that the insults and accusations of his opponents sometimes took on her husband’s fragile health—and also realized that he and his fellow politicians enjoyed such risks as well as the other less than wonderful effects of the pursuit and use of power in the name of a cause. As a fellow Virginian, Dolley had no difficulty identifying with the Jeffersonian Republicans’ hostility to Hamilton and his commercial ways.