James Patton Anderson was born on February 16, 1822, in Winchester, Tennessee, one of seven children of Colonel William Preston Anderson, a veteran of the War of 1812, and Margaret L. Adair, also from a prominent military family. Anderson, who was always known within the family as Patton, spent his early years on the family farm. After his father died in 1831, he moved with his mother to his grandfather's home in Kentucky. Five years later, his stepfather, Dr. Joseph Bybee, sent him to Jefferson College in Canonsburg, Pennsylvania. His education was interrupted by financial difficulties and by his stepfather's decision to relocate the family to DeSoto County, Mississippi; however, Anderson eventually resumed his schooling and graduated from Jefferson College in 1840. He read for law at Montrose Law School and passed the Mississippi bar. In 1847, he raised a company of volunteers for the Mexican War, served as a captain, and eventually achieved the rank of lieutenant colonel in command of the 1st Battalion, Mississippi Rifles. Following this service, he served one term in the Mississippi legislature, then accepted an appointment as United States Marshall to the Territory of Washington in 1853. This was also the year of his marriage to Henrietta (Etta) Buford Adair, his eighteen-year-old cousin. The couple departed for the Northwest and settled in Olympia.
Anderson's political career continued and he served as a Democrat in the Thirty-fourth Congress (March 4, 1855-March 3, 1857) after which he was offered but declined an appointment as governor of Washington Territory. Instead, in 1857, he and Etta moved to Florida, where they agreed to take over the management of Casa Bianca plantation near Monticello. This plantation was the property of Etta's aunt, Ellen Adair White Beatty, well known in Florida as the widow of Florida's congressional delegate Joseph M. White. In a complicated legal agreement, Anderson bought Casa Bianca plantation but also agreed to pay a yearly stipend to "Aunt Ellen" as part of the purchase price. This placed him under a heavy financial obligation, one that proved hard to meet over the years.
With the election of Abraham Lincoln in 1860, Anderson joined ardent secessionists and gave his full support to the Confederacy. He entered the Confederate Army as colonel of the 1st Regiment Florida (Infantry), received promotion to brigadier general February 10, 1862 and to major general February 17, 1864. His war-time command started out in Pensacola, took him through all the major campaigns of Tennessee and Georgia, and placed him for awhile in charge of the home front in Florida. He was wounded at the Battle of Jonesboro, Ga., in August 1864 and retired from the field to recuperate. Even after the restoration of the Union, Anderson remained unreconciled to the new order in the South, refusing to countersign his presidential amnesty. His finances were ruined and he took his family to Memphis, Tenn., where he earned his living by editing a publication on agriculture, working in insurance, and serving as a collector of delinquent taxes for Shelby County. He died of complications from war-time wounds on September 20, 1872, and was interred in Elmwood Cemetery, Memphis. Anderson was survived by his widow, Etta, and their children William Preston (b. 1856), Theophilus Beatty (b. 1858), James Patton, Jr. (1860), Elizabeth Cromwell (1863), and Margaret Bybee (1866). The Anderson family returned to Florida in 1883 and settled in Palatka, where Etta became the president of the local J. Patton Anderson Chapter of the United Daughters of the Confederacy. She died in 1917. Margaret (Maggie) Anderson, the last of the Anderson children, served as the family historian and keeper of the family papers until she passed away on May 7, 1965, at the age of 99.