James married Hannah Vandeventer Mitchell on July 6, 1847, in McLean Co., IL. She was the daughter of John Vandeventer and Mary "Polly" Downing. Her first marriage was to Joseph Mitchell, on January 17, 1843, and they has one son, John. James and Hannah lived in Bloomington, McLean County in 1850, Westport, Jackson Co., Missouri in 1860, and Osage, Vernon Co., Missouri in 1870. They had two sons, Robert Emmett Swartz (Dec 1850), and James Thompson Swartz, Jr. (Jul 1855). James, Sr., died March 1876 in Vernon County, Missouri. Hannah Vandeventer-Mitchell-Swartz-Wyckoff was living with her son James, Jr., in 1900, Nebraska City, Otoe Co., Nebraska. It is not known when she died, or where either is buried. James' birth county in Pennsylvania, and parents, are not known at this time (2012).