Now, as a landowner, Payne could vote, hold office, become a respectable citizen, and, Dolley daydreamed, carry on the Madison name in national politics. Toward the end of 1818, Dolley paid $540 for the farm and handed it over to her son. Payne started to remove leaves from mulberry trees at his father's home as food for his silkworms. He hired a silkworm expert from France and brought him over to help with production. At first, he set up his shed, with trays for eggs, near Madison's house, but he later moved it to Toddsberth. Now he needed to keep larvae clean and heat the shed properly, year round, to induce the growth of silk and spin it into sheets. Payne was intrigued by the silkworm industry, and Madison was delighted that his stepson finally seemed to have something to do that made sense. They both waited for the silkworm business to grow.8
In April 1820, Payne said that he'd like to visit Dolley's old friend, the very single Phoebe Morris, at her home, Bolton Farms, in Pennsylvania. Dolley's heart leaped. He needed a wife. Then he could have children and she could have grandchildren. She had tried to match Payne up with eligible young women of every persuasion throughout Virginia, and Washington, for years, with no success. She immediately wrote Phoebe and asked if Payne could visit her to renew their friendship. Phoebe, as interested as Dolley in a marriage to Payne, and in a renewal of her own relationship with Dolley, whom she loved, agreed and invited Payne to her farm. She and her family cleaned up the farm, tidied up the interior of the house, ordered special foods and drinks, and awaited his arrival with great anticipation.
Payne left Montpelier in early April and promptly disappeared. No one ever found out where he went. New York? Philadelphia? Baltimore? Europe? Nearly four months later, he finally turned up at the Morris farm, startling Phoebe, who had given up all hope of seeing him again. “[He] has made the most favorable impression in all of our hearts,” Phoebe's uncle immediately wrote Dolley. “Indeed, my excellent friend, I can't convey to you the pleasure his company afforded to us all.”9
And then, as always, Payne tired of Phoebe and left Bolton Farms after just four days. She and her uncle tried to talk him into remaining for at least a few more days, but he refused. Phoebe told Dolley that Payne was probably bored by the slow life at Bolton, far from the exciting nightlife of Philadelphia. She might have put him off, too. “I dare say he has been sufficiently wearied of my questions, for I was so glad to see him and know everything about you, how you looked, what you did and what you put on, etc. all the minute details which I thought my long absence would make reasonable,” Phoebe wrote Dolley.10
And, too, although it was never expressed, it's possible that Payne was tired of talking to people who talked about nothing but how much they loved his parents.
There was never a good explanation for Payne's departure from the Morris home. He just could not remain there for long. His impulses drove him to move on, and he never considered how his sudden departure would hurt Phoebe. He never understood how his odd activities hurt any of his friends or his parents, and he really did not care.
Phoebe's letter concerning Payne's lack of interest in her was one of the many letters Dolley had received from prospective daughters-in-law and friends who had hosted Payne, whether he was fourteen or twenty-four. Everyone loved him very much and had a very good time with him and wished him all the best of luck. No one ever told Dolley what a problem Payne had been, and would always be, because they did not want to break her heart. These letters and conversations only built up the delusional world in which Dolley Madison resided. “[A friend] wrote me yesterday that you were popular in the city. I should like to be with you to witness it, as the respect and love shown my son would be the highest gratification the world could bestow upon me,” she wrote to Payne (this from a woman whose husband was the president of the United States).11
Back at Montpelier, an undaunted Dolley forgot about Phoebe Morris and began new schemes with new young women to lure her son into marriage, writing letter after letter to their mothers or friends in an effort to set a meeting and start a romance. After all, she thought, her son was a good-looking young man and was quite enticing. Added on to that, the woman who married him would be a close relation to the president of the United States and would live at Montpelier, which they could expand to an even larger home if need be for Payne and his family. What on earth was wrong?
Their son never abandoned them. He would spend long periods of time, eighteen months to two years, at Montpelier and then go off to another city for months at a time. He would always return to Virginia. Payne did not simply drift aimlessly at Montpelier when he was there. He worked on his silkworm business and spent much time improving and expanding Toddsberth, his new home, eventually finishing the house at five comfortable rooms. He ran errands for his parents, helped to refurbish their house, bought things for them when he traveled, and was of considerable help in emergencies. Although he remained behind in one typhus epidemic, he helped to get both Madisons packed and sent them off to safety from the disease in Baltimore.
Dolley originally told Payne that the epidemic was ebbing. “I had a wish to travel a distance from home on account of the typhus fever, but that fear has been dissipated for the present by the children in the house getting well and the Negroes also. I trust therefore that you will not leave your business unfinished on my account, though I cannot express my anxiety to see you,” she wrote.12
Paying no attention to her instructions to remain far away, Payne rushed to Montpelier and arranged transportation for his parents to Baltimore, where residents were safe from the fever. An enormously thankful Dolley, telling everyone how her son risked his life to save hers, wrote Payne from there that “I cannot express my desire to see you.” She had good reason to thank him, too. The fever and other diseases were the scourge of the country. Edward Coles wrote from New York later that “the influenza, scarlet fever & measles have been very prevalent and by afflicting many persons, and placing many families in mourning, have kept out of society many of its greatest ornaments. Many of the families here that I know, and like best…had their houses closed either by sickness or its effects.”13
But when they returned to Montpelier from Baltimore, the old Payne was waiting for them. He left his silkworm business in the hands of his French worker (the climate in Virginia turned out to be not suitable for silkworms and the business failed) and departed from Toddsberth and Montpelier for various eastern cities. Dolley, in denial, as always, reasoned that travel was good for a young man. And perhaps, somewhere, he would find a wife and she would reshape his life. That was Dolley's never-ending fantasy. Her husband loved Payne but saw right through him. He kept bad news about Payne away from her, told her to stop worrying about him, always expressed that he saw a bright future for him in conversations with her, and extolled his virtues to her. In private, he fumed about his irresponsible, reckless stepson. He also continually paid Payne's bills, each one larger than the next. Altogether, Madison estimated he paid over $60,000 (about $1 million in today's money) to cover Payne's unpaid tabs over the years. He often did not tell his wife about the bill payments because he worried that it might hurt her.
There were times when an irate Madison refused to pay Payne's debts and they mounted. He did not communicate with his son for months one year and refused to send money that Payne requested in his letters. The result was that Payne wound up in debtors’ prison, yet again, and Madison, once more, had to bail him out.
If he could not get funds from his father, he borrowed money from his uncle, Richard Cutts, always explaining that there was simply no other way to obtain money.
The Madisons had so little faith in their son's ability to take care of himself that they asked their friends in different cities where he wandered to keep an eye on him, and they did. For example, Sophie Bache and her husband looked out for Payne in Philadelphia and told Dolley that he had been sick but recovered from “the fever.” She told Dolley that not only were she and her husband looking after Payne but also that she had friend George Dallas loo
k in on him, too.14
The Madisons had to do that because Payne rarely wrote to them or responded to their letters. Dolley often wrote him just to beg him to write back, often reminding him that her letters to him were waiting at the local post office for him to read them. He rarely did.15
Dolley received numerous letters from friends in different cities who sent some news of her son. He had been rumored to be engaged to Anne Coles of Williamsburg, but nothing every came of it. There was a woman he met in the Virginia mountains in the summer of 1820, but that relationship went nowhere, too. In 1816, he was said to be ready to marry a young woman who was smitten with him. “She said she had never seen anybody like Mr. Todd,” wrote the friend. That relationship, for which Dolley had so much hope, and yet such suspicion died, too. Sarah “Sally” Coles Stevenson, always on the lookout for prospective wives for Payne, wrote Dolley at Christmas 1820, “Tell cousin Payne he will find his old flame disengaged and quite as pretty as ever.”16
By 1817, when her husband's second term was coming to a close and Payne was twenty-six years old, Dolley confided to a cousin that she was convinced Payne would never marry and that he would “become a rover.”17
There were other young women in other cities, but those relationships went nowhere. Then there was a surprise letter from Payne in Philadelphia. He was enjoying the city and was involved with a woman, and had been for some time, but it did not seem to be leading anywhere. At about the same time she had received his letter, Henry Clay had asked about Payne. So had many others. They all told the Madisons that perhaps the best way for Payne to mature was for him to live at home with them, so they could keep an eye on him. On the loose, he would always get into trouble.
His mother wrote Payne a sharp letter. “I am ashamed to tell [people] when asked, how long my only child has been absent from the home of his mother! Your Papa and myself entreat you to come to us; to arrange your business with those concerned to return to them when necessary and let us see you here as soon as possible. Your Papa thinks as I do that it would be best for your reputation and happiness, as well as ours, that you should have the appearance of consulting your parents on subjects of deep account to you and that you should find it so on returning to Philadelphia,” she said.18
Payne told her he was coming home, but he needed $20 to get him there. Dolley, naturally, sent him $30. At other times, she mailed Payne expensive pieces of her jewelry, without telling her husband, and told him to sell the jewelry and use the money to pay his debts or to sustain his high-flying social life. She would casually mention in her notes that she had recently paid off local debts for him, such as $200 he had owed a Mr. Holloway for over two years.19
His mother literally pined for him. She told him in 1827 when he was sick in Philadelphia that “I felt so full of fear that you might relapse that I hastened to pack a few clothes & give orders for the carriage to be ready [to see you].”20
They did not hear from Payne again and then, just before Christmas, he arrived. There he was, handsome and delightful as ever, as well dressed as any of the dandies in Paris or London. He resumed work in the silkworm shed, joined them for many dinners in the mansion, and greeted all of their old friends during the Christmas holidays. He renewed friendships with cousins he had spent years with as a child and he had hosted on trips to Washington and Richmond. Everyone met him and everyone was enamored by him.
Payne had no real feeling for his mother. He never repaid her loans or money forwarded to him by the president, and he often took advantage of her. The most blatant example of that took place in 1838. Payne talked his mother into giving him fifty acres of Montpelier land, on which stood a gristmill. It was a gift that made Dolley feel good, naturally. What she did not know was that Payne planned to sell the land and pocket the money. He did that four months later. Payne sold forty-one acres to James Newman for $571. Dolley then had no land and no money. It was typical of Payne's relationship to his parents.21
In 1824 and 1825, Payne plunged into debt, yet again, and his mother sent him money through the mail to cover his bills. He could not pay his rent, either, and the president paid for that as well. Bills arrived from all over, even a $500 bill from a lottery parlor in Washington, which Madison paid. Madison wrote his son, then thirty-three, “What shall I say to you? Weeks have passed without even a line…soothing the anxieties of the tenderest of mothers, wound up to the highest pitch by your long and mysterious absence…as ample remittances were furnished for all known purposes, your continuance where you are under such strange appearances necessarily produces distressing apprehension. Whatever the causes of [your long absence and debts] you owe it to yourself as well as to us to withhold them no longer. Let the worst be known, that the best may be made of it…. You cannot be too quick in affording relief to [your other] present feelings.”22
Payne disappeared again in 1826, and this time events took a dark turn. The Madisons had no word from him but several letters from his creditors. He had piled up bills in Philadelphia and still owed his rent money. Friends paid some of his bills, and Madison thanked them and told them to never do that again. He wrote Payne's landlord that he would cover his bills, but that he would need time to get the money. The landlord threatened to have Payne tossed into debtors’ prison, again, if the bill was not paid. Madison then heard from his old political crony in Philadelphia, Postmaster Richard Bache, who said he had cashed Payne's check for $300, but the boy then vanished. He told the president that if Payne did not give him the $300, he would have gone to jail. The president immediately mortgaged 361 acres of land for $2,000 to cover his son's latest debts.
The president's emotions were in a tailspin over Payne, his debts, and his prison terms. “Can it be that though released from detention by one creditor, you have entangled yourself with another?…Lose not a moment in making known your real situation that what can be done for it may be done,” he said in the dead of winter that year.23
Madison wrote his son a sad letter. In it, he told him that he had not told his mother of the most recent of his many unpaid bills that his stepfather had to pay. He asked Payne to come home and stay home so he could keep out of trouble. “Come then, I entreat and conjure you, to the bosom of your parents who are anxious to do everything to save you from tendencies and past errors and provide for comfort and happiness,” he said.24
That effort did little good. Payne later moved to New York and spent several months there. His arrival home was, as usual, further delayed. In New York he spent time with women, gambled (gambling was legal in New York then and the city was home to numerous gambling dens), and drank. The Madisons had no idea where he was getting his money from or how much he had spent, or owed. Dolley continually covered for him in letters to relatives and friends. She wrote her niece that “[Payne] writes in fine health and spirits and says he will yet be detained two or three weeks longer in that city. I flatter myself with the hope of seeing him soon. No, it's impossible for him to prefer Virginia…to the North,” she said. It was one of many letters that told us how well Payne was doing and how much everybody liked him and how his travels were making him a real Renaissance man.25
Of course, Payne was not a Renaissance man at all, just a man who spent more money than he had. He went back to Philadelphia to spend more money and, in the spring of 1829, again wound up in debtors’ prison. Madison, getting ready to attend the Virginia convention that year to write a new state constitution, once again bailed him out. One year later, he had to bail him out of another debtors’ prison in Washington. Paying Payne's bills was a routine for Madison, something he was used to doing. In the winter of 1822, Dolley showed James a heartfelt plea from Payne for money so that he could return to Montpelier. “My good M. came in & I consulted him about sending money to P. He directly drew an order & told me to enclose it to you, that I must beg you to send to him directly if you can, hire a messenger, do so,” she wrote Anna Cutts.26
The Madisons were in so much of a hurry to pay Payne's bills directly
or through intermediaries that they even gave the intermediaries tips ranging up to $20 for a few minutes’ work and paid the intermediaries’ expenses for any travel they incurred in a visit to Payne.
Payne would have gone to jail in New York, too, but John Jacob Astor loaned him $600 to pay off his bills. He gave him another $1,600 to cover his debts on similar binges in Philadelphia and Germantown. Payne took money that Astor gave him for those debts and lost it in a gambling casino. Creditors kept following him for the $1,600 he owed, and Madison had to get one of his tobacco agents in Virginia to pay the bill. Madison then mortgaged more of his land to raise money to cover the debt. His former secretary Edward Coles, who knew Payne well, once loaned Madison $2,000 to pay Payne's debts.
Dolley begged her son to return home in order to stay away from the dens of temptation that she felt were ruining his life. Friends told Payne that his mother was getting sick over his absences. His uncle wrote him that his return would have a “beneficial effect in tranquilizing her nervous agitation…. She has been lying down since this morning complaining of great chilliness and shivering.”27
Despite his mistreatment of her, Dolley never deserted her son. She constantly wrote him loving letters that, between the lines, begged him to calm down and simply be normal. She told him of their retirement and of news in the lives of his cousins. She wrote in one letter that she had no real reason to write but that she just enjoyed “the pleasure of repeating how much I love you and to hear of your happiness.” She would often close her letters to him with “May Heaven bless thee.”28
James and Dolley Madison Page 42
