Isham was born to Mary "Mamie" and Charles Bradford Isham. A drop out of Harvard, a newspaper once said "his frail body was unequal to the strain". Later on, Isham was said to have been a secret operative for the U.S. government inWorld War II.
On August 30, 1919, Isham married Leahalma "Lea" Correa, a New York ‘Society Girl of Sephardic Spanish Descent’, and helped raise her daughter, Frances Mantley. He was a frequent visitor at Hildene. Once when he was young, he was allowed to drive his grandfather Robert Todd Lincoln's 1905 Thomas Automobile, but rolled it in a ditch.
Later in his life, Isham and his wife settled in Dorset, Vermont where they owned a 22-acre (9 ha) farm. A talented amateur musician, he would often play music while his wife wrote children’s stories. On September 1, 1971 at 12:30 am, Isham died at Putnam Memorial Hospital at the age of 79. He left most of his estate to the Red Cross, Salvation Army, andAmerican Cancer Society. He also left a US$440,000 trust fund for his stepdaughter, and sent his grandmother's ‘Chicken Leg Coffee Set’ and ‘White House china’ to theSmithsonian. The rest of his family artifacts were auctioned off.